Create a Secure Local Consul Datacenter with Docker Compose

18min
Consul

In this tutorial, you will create a secure local Consul datacenter using Docker Compose and Consul agent configuration files. You will then explore your Consul datacenter with the Consul UI and Consul CLI.

A Consul datacenter contains both clients and servers agents. These agents participate in a LAN gossip pool and RPC forwarding that enables the distributed exchange of information throughout your environment. This information distribution provides the foundation for Consul's service discovery and service mesh capabilities.

Consul security features include the Access Control List (ACL) system, TLS encryption, and gossip encryption. These security features protect Consul communication against eavesdropping, tampering, and spoofing. Check the Consul security model for more information.

While this tutorial uses elements that are not suitable for production environments including example certificates, example gossip encryption keys, global ACL token usage, and no custom ACL policies, it will teach you the core concepts for deploying and interacting with a secure Consul datacenter. Refer to the Consul Reference Architecture for Consul best practices and the Docker Documentation for Docker best practices.

Prerequisites

This tutorial builds on the concepts learned in the Consul getting started collection.

Docker

You will need a local install of Docker running on your machine for this tutorial. You can find the instructions for installing Docker on your specific operating system here. Docker Engine 19.03.0 was tested in this tutorial.

Docker Compose

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Visit the Docker Compose install guide for operating system specific installation instructions. Docker Compose format 3.7 was used in this tutorial.

GitHub repository

Clone the GitHub repository containing the configuration files and resources.

$ git clone https://github.com/hashicorp/learn-consul-docker.git

Change into the directory with the newly cloned repository. This directory contains the complete configuration files.

$ cd learn-consul-docker/datacenter-deploy-secure

Inspect configuration files

You will use the following resources to deploy four containers; three Consul servers and a single Consul client.

Docker Compose YAML

Inside your working directory, inspect the the file named docker-compose.yml.

docker-compose.yml

version: '3.7'

services:
  
  consul-server1:
    image: hashicorp/consul:1.11.2
    container_name: consul-server1
    restart: always
    volumes:
     - ./server1.json:/consul/config/server1.json:ro
     - ./certs/:/consul/config/certs/:ro
    networks:
      - consul
    ports:
      - "8500:8500"
      - "8600:8600/tcp"
      - "8600:8600/udp"
    command: "agent -bootstrap-expect=3"

  consul-server2:
    image: hashicorp/consul:1.11.2
    container_name: consul-server2
    restart: always
    volumes:
     - ./server2.json:/consul/config/server2.json:ro
     - ./certs/:/consul/config/certs/:ro
    networks:
      - consul
    command: "agent -bootstrap-expect=3"

  consul-server3:
    image: hashicorp/consul:1.11.2
    container_name: consul-server3
    restart: always
    volumes:
     - ./server3.json:/consul/config/server3.json:ro
     - ./certs/:/consul/config/certs/:ro
    networks:
      - consul
    command: "agent -bootstrap-expect=3"

  consul-client:
    image: hashicorp/consul:1.11.2
    container_name: consul-client
    restart: always
    volumes:
     - ./client.json:/consul/config/client.json:ro
     - ./certs/:/consul/config/certs/:ro
    networks:
      - consul
    command: "agent"

networks:
  consul:
    driver: bridge

This Docker Compose configuration file instructs Docker to create four Consul containers using the respective configuration files, configure networking, and bootstrap the Consul datacenter with three Consul servers. Three servers in a datacenter is the recommended minimum for achieving a balance between availability and performance. These servers together run the Raft-driven consistent state store for updating catalog, session, prepared query, ACLs, and KV state.

For more information on the Consul Docker image, check Consul's Docker Hub page.

Consul server JSON

Inside your working directory, inspect the three server#.json files.

server1.json

{
  "node_name": "consul-server1",
  "server": true,
  "ui_config": {
    "enabled": true
  },
  "data_dir": "/consul/data",
  "addresses": {
    "http": "0.0.0.0"
  },
  "retry_join": ["consul-server2", "consul-server3"],
  "encrypt": "aPuGh+5UDskRAbkLaXRzFoSOcSM+5vAK+NEYOWHJH7w=",
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "/consul/config/certs/consul-agent-ca.pem",
  "cert_file": "/consul/config/certs/dc1-server-consul-0.pem",
  "key_file": "/consul/config/certs/dc1-server-consul-0-key.pem"
}

Each Consul server configuration file uses the following options:

node_name - Each node in a datacenter must have a unique name. By default, Consul uses the hostname of the machine, but in this tutorial you'll manually set a unique name.
server- Setting this flag to true specifies that you want the agent to start in server mode. By default, Consul sets this value to false.
ui_config - Setting the sub key enabled to true enables the the web UI service from this agent. Introduced in Consul 1.9.0, replaces the ui option.
data-dir - This flag tells Consul agents where they should store their state, which can include sensitive data like ACL tokens for both servers and clients. In production deployments you should be careful about the permissions for this directory.
addresses - This is the address that this agent will listen on for communication from other Consul members. By default, this is 0.0.0.0, meaning Consul will bind to all addresses on the local machine and will advertise the private IPv4 address to the rest of the cluster. If there are multiple private IPv4 addresses available you will have to specify on which address you want Consul to listen on with the -bind option, otherwise Consul will not be able to startup and exit with an error.
retry_join - Address of a Consul server to join upon starting up. Similar to join but allows retrying a join until it is successful. The value can contain one or many IPv4, IPv6, or DNS addresses. In this tutorial, the consul-server# Docker DNS address value is used. Check the Docker DNS docs for more information on Docker's embedded DNS.
encrypt - Specifies the secret key to use for encryption of Consul network traffic. This key must be 32-bytes that are Base64-encoded. The easiest way to create an encryption key is to use consul keygen. All nodes within a cluster must share the same encryption key to communicate.
verify_incoming - If set to true, Consul requires that all incoming connections make use of TLS and that the client provides a certificate signed by a Certificate Authority from the ca_file. This applies to both server RPC and to the HTTPS API.
verify_outgoing - If set to true, Consul requires that all outgoing connections from this agent make use of TLS and that the server provides a certificate that is signed by a Certificate Authority from the ca_file. This applies to clients and servers as both will make outgoing connections.
verify_server_hostname - If set to true, Consul verifies for all outgoing TLS connections that the TLS certificate presented by the servers matches server.<datacenter>.<domain> hostname. This setting is critical to prevent a compromised client from being restarted as a server and having all cluster state including all ACL tokens and Connect CA root keys replicated to it.
ca_file - This provides a file path to a PEM-encoded certificate authority. The certificate authority is used to check the authenticity of client and server connections with the appropriate verify_incoming or verify_outgoing flags.
cert_file - This provides a file path to a PEM-encoded certificate. The certificate is provided to clients or servers to verify the agent's authenticity. It must be provided along with key_file.
key_file - This provides a the file path to a PEM-encoded private key. The key is used with the certificate to verify the agent's authenticity. This must be provided along with cert_file.

Important

Certificates have been pre-generated for this tutorial for the sake of simplicity. Create your own certificates for production environments. Check the TLS encryption tutorial for more information.

Check the Consul configuration documentation to learn more.

Consul client JSON

Inside your working directory, inspect the client.json file.

client.json

{
  "node_name": "consul-client",
  "data_dir": "/consul/data",
  "retry_join": ["consul-server1", "consul-server2", "consul-server3"],
  "encrypt": "aPuGh+5UDskRAbkLaXRzFoSOcSM+5vAK+NEYOWHJH7w=",
  "verify_incoming": true,
  "verify_outgoing": true,
  "verify_server_hostname": true,
  "ca_file": "/consul/config/certs/consul-agent-ca.pem",
  "cert_file": "/consul/config/certs/dc1-server-consul-0.pem",
  "key_file": "/consul/config/certs/dc1-server-consul-0-key.pem"
}

A Consul client configuration file requires fewer settings than a server configuration file.

Check the Consul configuration documentation to learn more.

Create the Consul datacenter

From within your working directory, run the following Docker Compose command to create your secure local Consul datacenter.

$ docker-compose up --detach

Creating network "datacenter-deploy-secure_consul" with driver "bridge"
...
Creating consul-server1 ... done
Creating consul-server2 ... done
Creating consul-server3 ... done
Creating consul-client ... done

Note

Your first run will take the longest as Docker will first pull the Consul image from the Docker Hub repository. Additional runs will not require to download the image again and should only take a few seconds to complete.

View the Consul UI

Navigate to http://localhost:8500/ on your browser to access the Consul UI.

Click on the "Nodes" option in the top navigation bar to go to the nodes page. There you'll find an overview of the entire datacenter including the health status of each node, IP address, number of registered services, and a leader badge indicating which node is hosting the leading Consul server.

Image of Nodes page in Consul UI.

Check the Consul UI tutorial to learn more.

Access Consul containers

You can interact with your containerized Consul datacenter with the Docker CLI.

Docker exec

You can execute Consul CLI commands from your local terminal to your Consul containers using the docker exec command.

$ docker exec consul-client consul members

Node            Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul-server1  172.18.0.4:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-server2  172.18.0.5:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-server3  172.18.0.3:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-client   172.18.0.2:8301  alive   client  1.11.2  2         dc1  default    <default>

Docker exec attach

You can also issue commands inside of your container by opening an interactive shell and using the Consul CLI included in the container.

$ docker exec -it consul-client /bin/sh

/ #

Once inside the consul-client container you can use the consul members command to verify you have access to the Consul datacenter.

$ consul members

Node            Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul-server1  172.18.0.4:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-server2  172.18.0.5:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-server3  172.18.0.3:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-client   172.18.0.2:8301  alive   client  1.11.2  2         dc1  default    <default>

Configure the ACL system

Consul uses ACLs to secure access to the UI, API, CLI, service communications, and agent communications.

This section will guide you through enabling the ACL system, configuring your agents with ACL tokens, and accessing your Consul datacenter with ACL tokens.

Enable ACLs on all Consul agents

The first step for bootstrapping the ACL system is to enable ACLs on the Consul servers in the agent configuration. In this example, you are configuring the default policy of "deny", which means Consul will deny access to resources unless the request uses a token with explicitly granted privileges, and a down policy of "extend-cache", which means that the agents will extend TTLs for cached tokens during an outage.

It is important to ensure the servers are configured properly before enabling ACLs on the clients. This will reduce any duplicate work and troubleshooting if there is a misconfiguration.

Note

Consul will load all files in the configuration directory, for the following examples, you can add the configuration as new files. You can also add them to an existing configuration file, however, you should ensure that the syntax is valid before applying it to the agent. You can use consul validate to check configuration validity for both JSON and HCL files.

Add the following configuration to the configuration directory on all the agents (servers and clients).

consul-acl.hcl

acl {
  enabled        = true
  default_policy = "deny"
  down_policy    = "extend-cache"
  enable_token_persistence = true
}

You can copy the file inside the docker container using the docker cp command.

$ docker cp consul-acl.json consul-server1:/consul/config/consul-acl.json

The agents will need to be restarted to load the new configuration. Take special care to restart the servers one at a time, restarting the leader as last node, and ensure each server has joined and is operating correctly before restarting another.

You can find the leader either from the UI using the leader badge shown nearby the server node acting as leader ur using the consul operator command.

$ docker exec consul-client consul operator raft list-peers

Node            ID                                    Address          State     Voter  RaftProtocol
consul-server2  d55b6515-25a9-1ef6-01cc-562a959105d0  172.18.0.5:8300  leader    true   3
consul-server1  b3355c40-1745-f9fa-63c1-33b9edd5ced7  172.18.0.4:8300  follower  true   3
consul-server3  2028137c-eda7-99ef-37eb-75dc875271db  172.18.0.3:8300  follower  true   3

You can use the State column value to identify the leader. In this example the leader is the node consul-server-2.

Once identified the leader proceed with a restart of the nodes. You can restart Consul on the containers using the docker restart command to restart the container running the process.

$ docker restart consul-server1

If ACLs are enabled correctly, the leader's logs will contain the following warning and info messages.

[INFO]  agent.server: New leader elected: payload=consul-server3
...
[INFO]  agent.server: initializing acls
[INFO]  agent.server: Created ACL 'global-management' policy
[INFO]  agent.server: Created ACL anonymous token from configuration
[INFO]  agent.leader: started routine: routine="legacy ACL token upgrade"
[INFO]  agent.leader: started routine: routine="acl token reaping"
...
[WARN]  agent: Coordinate update blocked by ACLs: accessorID=00000000-0000-0000-0000-000000000002

Note

Now that you have enabled ACLs, you will need a token to complete any operation. You can't do anything else to the datacenter until you bootstrap the ACL system and generate the bootstrap token. For simplicity, you will use the global management token created during the bootstrap for the remainder of the tutorial.

Create the bootstrap token

Once ACLs have been enabled, you can bootstrap the ACL system to generate the first token, named bootstrap token. The bootstrap token is a global management token with unrestricted privileges.

First, open an interactive shell to any one of your Consul server containers.

$ docker exec -it consul-server1 /bin/sh

/ #

From the interactive shell, create the bootstrap token.

$ consul acl bootstrap

AccessorID:       db6eade6-d0b9-9e5c-d8b0-b46a3b7eec9c
SecretID:         b619d009-c109-0c7d-7a27-c07205f6187a
Description:      Bootstrap Token (Global Management)
Local:            false
Create Time:      2022-02-09 16:34:22.935628658 +0000 UTC
Policies:
   00000000-0000-0000-0000-000000000001 - global-management

Note

The SecretID is the ACL token used to authenticate API and CLI requests.

On the server where the bootstrap command was issued, the logs should contain the following log message.

[INFO] consul.acl: ACL bootstrap completed
[DEBUG] http: Request PUT /v1/acl/bootstrap (2.347965ms) from=127.0.0.1:40566

Since ACLs have been enabled, you will now need to use a token to complete any additional operations. For example, even checking the member list will require a token.

$ consul members -token "b619d009-c109-0c7d-7a27-c07205f6187a"

Node            Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul-server1  172.18.0.4:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-server2  172.18.0.5:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-server3  172.18.0.3:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-client   172.18.0.2:8301  alive   client  1.11.2  2         dc1  default    <default>

An alternative to using the token on the command line with the -token flag is to set the token as an environment variable.

$ export CONSUL_HTTP_TOKEN=b619d009-c109-0c7d-7a27-c07205f6187a

$ consul members

Node            Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul-server1  172.18.0.4:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-server2  172.18.0.5:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-server3  172.18.0.3:8301  alive   server  1.11.2  2         dc1  default    <all>
consul-client   172.18.0.2:8301  alive   client  1.11.2  2         dc1  default    <default>

When authentication is enabled, a Consul token should be provided to API requests using the X-Consul-Token header.

$ curl \
    --header "X-Consul-Token: b619d009-c109-0c7d-7a27-c07205f6187a" \
    http://127.0.0.1:8500/v1/agent/members

An alternative method is to use curl with the Bearer scheme.

$ curl \
    --header "Authorization: Bearer b619d009-c109-0c7d-7a27-c07205f6187a" \
    http://127.0.0.1:8500/v1/agent/members

The output for the command is similar to the one below:

[
  {
    "Name": "consul-server1",
    "Addr": "172.18.0.4",
    "Port": 8301,
    "Tags": {
      "acls": "1",
      "build": "1.11.2:37c7d06b",
      "dc": "dc1",
      "expect": "3",
      "ft_fs": "1",
      "ft_si": "1",
      "id": "b3355c40-1745-f9fa-63c1-33b9edd5ced7",
      "port": "8300",
      "raft_vsn": "3",
      "role": "consul",
      "segment": "",
      "use_tls": "1",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2",
      "wan_join_port": "8302"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 5,
    "DelegateCur": 4
  },
  {
    "Name": "consul-server2",
    "Addr": "172.18.0.5",
    "Port": 8301,
    "Tags": {
      "acls": "1",
      "build": "1.11.2:37c7d06b",
      "dc": "dc1",
      "expect": "3",
      "ft_fs": "1",
      "ft_si": "1",
      "id": "d55b6515-25a9-1ef6-01cc-562a959105d0",
      "port": "8300",
      "raft_vsn": "3",
      "role": "consul",
      "segment": "",
      "use_tls": "1",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2",
      "wan_join_port": "8302"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 5,
    "DelegateCur": 4
  },
  {
    "Name": "consul-client",
    "Addr": "172.18.0.2",
    "Port": 8301,
    "Tags": {
      "build": "1.11.2:37c7d06b",
      "dc": "dc1",
      "id": "a3f87fce-9d09-3970-bfaf-c0963c980101",
      "role": "node",
      "segment": "",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 5,
    "DelegateCur": 4
  },
  {
    "Name": "consul-server3",
    "Addr": "172.18.0.3",
    "Port": 8301,
    "Tags": {
      "acls": "1",
      "build": "1.11.2:37c7d06b",
      "dc": "dc1",
      "expect": "3",
      "ft_fs": "1",
      "ft_si": "1",
      "id": "2028137c-eda7-99ef-37eb-75dc875271db",
      "port": "8300",
      "raft_vsn": "3",
      "role": "consul",
      "segment": "",
      "use_tls": "1",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2",
      "wan_join_port": "8302"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 5,
    "DelegateCur": 4
  }
]

Note

The bootstrap token can only be created once, bootstrapping will be disabled after the bootstrap token is created. Once the ACL system is bootstrapped, ACL tokens can be managed through the ACL API.

Add the agent token to all Consul agents

For this tutorial, all agents (servers and clients) will share the same token for the sake of simplicity. Using the bootstrap token for all agents is NOT a secure practice. In production it is important you generate individual tokens based on more restrictive policies. Check the Access Control System tutorial to learn more about production ACL practices.

It is important to ensure the servers are configured properly before enabling ACLs on the clients. This will reduce any duplicate work and troubleshooting if there is a misconfiguration.

Assign the bootstrap token to all of the Consul agents (servers and clients) with the set-agent-token subcommand. Start with the servers and ensure they are working correctly before applying the client tokens. Alternatively, you can set the token in the agent configuration file.

Using the interactive shell, assign the token to your agent.

$ consul acl set-agent-token agent "b619d009-c109-0c7d-7a27-c07205f6187a"

ACL token "agent" set successfully

You will need to complete this process on every agent. Additionally, you will need to set the CONSUL_HTTP_TOKEN on every agent to the bootstrap token or use the token API option.

Your environment is now secured with gossip encryption, TLS encryption, and ACLs.

Additional ACL configuration

This tutorial only covers the basics of ACLs in a Consul datacenter. In your production environment you will need to create more fine grained policies. To learn more about fine-tuning your ACL security, check the production ACL tutorial.

Clean up your environment

To clean up your environment, execute the following command.

$ docker-compose down --rmi all

Stopping consul-server1 ... done
Stopping consul-client  ... done
Stopping consul-server3 ... done
Stopping consul-server2 ... done
Removing consul-server1 ... done
Removing consul-client  ... done
Removing consul-server3 ... done
Removing consul-server2 ... done
Removing network datacenter-deploy-secure_consul
Removing image hashicorp/consul:1.11.2

Next steps

In this tutorial, you learned to deploy and configure a secure local containerized Consul datacenter using Docker Compose. You learned how to interact with your containers using docker exec, to use CLI and API endpoints to bootstrap the ACL system and assign tokens to the Consul agents. Finally, you learned how to clean up your environment.

You can continue learning how to deploy a Consul datacenter in production by completing the Deployment guide. The collection includes securing the datacenter with Access Control Lists, encryption, DNS configuration, and datacenter federation.

You can also extend your Consul skills by exploring the following tutorials:

For additional reference documentation on the official Docker image for Consul, refer to the following websites: