Rotate Gossip Encryption Keys in Consul

The consul keyring command is used to examine and modify the encryption keys used in Consul’s gossip pools. It is capable of distributing new encryption keys to the agents, retiring old encryption keys, and changing the keys used by the agents to encrypt messages.

In this tutorial, you interact with the Consul keyring and change the gossip encryption key used by your datacenter.

Prerequisites

In order to complete the steps listed in this tutorial, you need a running Consul datacenter with gossip encryption enabled. You can follow Gossip Encryption tutorial to configure gossip encryption on a new datacenter.

Required steps

The steps to rotate the gossip key in use by your datacenter are the following:

1. Check keyring content
2. Generate a new encryption key
3. Add new key to the keyring
4. Verify that the key was installed
5. Promote the new key to primary
6. Remove the old key from the keyring

Check keyring content

List your current keys, and verify that all nodes have the proper key.

$ consul keyring -list

==> Gathering installed encryption keys...
WAN:
 J5+gaLwt5cshalq0HLlGmhgxoqwXO5506MMxp16oiHQ= [3/3]
dc1 (LAN):
 J5+gaLwt5cshalq0HLlGmhgxoqwXO5506MMxp16oiHQ= [11/11]

Generate a new encryption key

You can generate a new gossip key using any method that can creates 16 random bytes encoded in base64.

$ consul keygen | tee encryption.key

FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw=

$ openssl rand -base64 16 | tee encryption.key

22Br/DzyKMBuH5VjhakYEA==

$ dd if=/dev/urandom bs=16 count=1 status=none | base64 | tee encryption.key

DYT6KCxb6ZxhryYRb6WdYw==

Store the new gossip key in an environment variable.

$ export CONSUL_GOSSIP_KEY=$(cat encryption.key)

Add new key to the keyring

Add your newly generated key to the keyring.

$ consul keyring -install ${CONSUL_GOSSIP_KEY}

==> Installing new gossip encryption key...

Verify that the key was installed

Once you have added the key to one of the Consul agents, it will be propagated across the whole datacenter. You do not need to repeat the command on other agents.

You can ensure that the key has been propagated to all agents by verifying the number of agents that recognize the key over the number of total agents in the datacenter.

$ consul keyring -list

==> Gathering installed encryption keys...
WAN:
 J5+gaLwt5cshalq0HLlGmhgxoqwXO5506MMxp16oiHQ= [3/3]
 FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw= [3/3]
dc1 (LAN):
 J5+gaLwt5cshalq0HLlGmhgxoqwXO5506MMxp16oiHQ= [11/11]
 FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw= [11/11]

You must check that the two keys are installed in the datacenter, and are recognized by all 11 agents, as well as by all 3 server agents. Do not proceed to the next step unless all agents have the new key.

Promote the new key to primary

Once all agents have received the key and are able to use it as the primary encryption key, it is possible to promote the new key to primary.

$ consul keyring -use ${CONSUL_GOSSIP_KEY}

==> Changing primary gossip encryption key...

Remove the old key from the keyring

To avoid unused keys remaining in the keyring, we recommended you remove the old primary from the keyring once a new key is installed.

$ consul keyring -remove J5+gaLwt5cshalq0HLlGmhgxoqwXO5506MMxp16oiHQ=

==> Removing gossip encryption key...

Verify that the keyring contains only one key.

$ consul keyring -list

==> Gathering installed encryption keys...
WAN:
 FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw= [3/3]
dc1 (LAN):
 FfRV9j6NXU9LlCI4zLZjjpZdj4Nrqsdm7R8YgzSHzHw= [11/11]

Create a script for rotating the gossip key

Once you have learned the process for changing the gossip encryption key, you can automate it using whatever tool you prefer.

One way is to create a script that does it for you that you can run against one Consul server.

#!/usr/bin/env bash

# Setup Consul address info
export CONSUL_HTTP_ADDR="http://localhost:8500"

# Generate the new key
NEW_KEY=`consul keygen`

# Install the key
consul keyring -install ${NEW_KEY}

# Set as primary
consul keyring -use ${NEW_KEY}

# Retrieve all keys used by Consul
KEYS=`curl -s ${CONSUL_HTTP_ADDR}/v1/operator/keyring`

ALL_KEYS=`echo ${KEYS} | jq -r '.[].Keys| to_entries[].key' | sort | uniq`

for i in `echo ${ALL_KEYS}`; do
  if [ $i != ${NEW_KEY} ] ; then
    consul keyring -remove $i
  fi
done

Next steps

In this tutorial you learned how to rotate the gossip key saved in the keyring for your Consul datacenter.

To learn how to automate gossip key rotation using HashiCorp Vault and consul-template, refer to the Generate gossip keys for Consul using Vault tutorial.