Create a Secure Local Consul Datacenter with Docker Compose
In this tutorial, you will create a secure local Consul datacenter using Docker Compose and Consul agent configuration files. You will then explore your Consul datacenter with the Consul UI and Consul CLI.
A Consul datacenter contains both clients and servers agents. These agents participate in a LAN gossip pool and RPC forwarding that enables the distributed exchange of information throughout your environment. This information distribution provides the foundation for Consul's service discovery and service mesh capabilities.
Consul security features include the Access Control List (ACL) system, TLS encryption, and gossip encryption. These security features protect Consul communication against eavesdropping, tampering, and spoofing. Check the Consul security model for more information.
While this tutorial uses elements that are not suitable for production environments including example certificates, example gossip encryption keys, global ACL token usage, and no custom ACL policies, it will teach you the core concepts for deploying and interacting with a secure Consul datacenter. Refer to the Consul Reference Architecture for Consul best practices and the Docker Documentation for Docker best practices.
Prerequisites
This tutorial builds on the concepts learned in the Consul getting started collection.
Docker
You will need a local install of Docker running on your machine for this tutorial. You can find the instructions for installing Docker on your specific operating system here. Docker Engine 19.03.0 was tested in this tutorial.
Docker Compose
Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Visit the Docker Compose install guide for operating system specific installation instructions. Docker Compose format 3.7 was used in this tutorial.
GitHub repository
Clone the GitHub repository containing the configuration files and resources.
Change into the directory with the newly cloned repository. This directory contains the complete configuration files.
Inspect configuration files
You will use the following resources to deploy four containers; three Consul servers and a single Consul client.
Docker Compose YAML
Inside your working directory, inspect the the file named docker-compose.yml
.
This Docker Compose configuration file instructs Docker to create four Consul containers using the respective configuration files, configure networking, and bootstrap the Consul datacenter with three Consul servers. Three servers in a datacenter is the recommended minimum for achieving a balance between availability and performance. These servers together run the Raft-driven consistent state store for updating catalog, session, prepared query, ACLs, and KV state.
For more information on the Consul Docker image, check Consul's Docker Hub page.
Consul server JSON
Inside your working directory, inspect the three server#.json
files.
Each Consul server configuration file uses the following options:
node_name
- Each node in a datacenter must have a unique name. By default, Consul uses the hostname of the machine, but in this tutorial you'll manually set a unique name.server
- Setting this flag totrue
specifies that you want the agent to start in server mode. By default, Consul sets this value tofalse
.ui_config
- Setting the sub keyenabled
totrue
enables the the web UI service from this agent. Introduced in Consul 1.9.0, replaces theui
option.data-dir
- This flag tells Consul agents where they should store their state, which can include sensitive data like ACL tokens for both servers and clients. In production deployments you should be careful about the permissions for this directory.addresses
- This is the address that this agent will listen on for communication from other Consul members. By default, this is0.0.0.0
, meaning Consul will bind to all addresses on the local machine and will advertise the private IPv4 address to the rest of the cluster. If there are multiple private IPv4 addresses available you will have to specify on which address you want Consul to listen on with the-bind
option, otherwise Consul will not be able to startup and exit with an error.retry_join
- Address of a Consul server to join upon starting up. Similar tojoin
but allows retrying a join until it is successful. The value can contain one or many IPv4, IPv6, or DNS addresses. In this tutorial, theconsul-server#
Docker DNS address value is used. Check the Docker DNS docs for more information on Docker's embedded DNS.encrypt
- Specifies the secret key to use for encryption of Consul network traffic. This key must be 32-bytes that are Base64-encoded. The easiest way to create an encryption key is to useconsul keygen
. All nodes within a cluster must share the same encryption key to communicate.verify_incoming
- If set totrue
, Consul requires that all incoming connections make use of TLS and that the client provides a certificate signed by a Certificate Authority from theca_file
. This applies to both server RPC and to the HTTPS API.verify_outgoing
- If set totrue
, Consul requires that all outgoing connections from this agent make use of TLS and that the server provides a certificate that is signed by a Certificate Authority from theca_file
. This applies to clients and servers as both will make outgoing connections.verify_server_hostname
- If set totrue
, Consul verifies for all outgoing TLS connections that the TLS certificate presented by the servers matchesserver.<datacenter>.<domain>
hostname. This setting is critical to prevent a compromised client from being restarted as a server and having all cluster state including all ACL tokens and Connect CA root keys replicated to it.ca_file
- This provides a file path to a PEM-encoded certificate authority. The certificate authority is used to check the authenticity of client and server connections with the appropriateverify_incoming
orverify_outgoing
flags.cert_file
- This provides a file path to a PEM-encoded certificate. The certificate is provided to clients or servers to verify the agent's authenticity. It must be provided along withkey_file
.key_file
- This provides a the file path to a PEM-encoded private key. The key is used with the certificate to verify the agent's authenticity. This must be provided along withcert_file
.
Important
Certificates have been pre-generated for this tutorial for the sake of simplicity. Create your own certificates for production environments. Check the TLS encryption tutorial for more information.
Check the Consul configuration documentation to learn more.
Consul client JSON
Inside your working directory, inspect the client.json
file.
A Consul client configuration file requires fewer settings than a server configuration file.
Check the Consul configuration documentation to learn more.
Create the Consul datacenter
From within your working directory, run the following Docker Compose command to create your secure local Consul datacenter.
Note
Your first run will take the longest as Docker will first pull the Consul image from the Docker Hub repository. Additional runs will not require to download the image again and should only take a few seconds to complete.
View the Consul UI
Navigate to http://localhost:8500/ on your browser to access the Consul UI.
Click on the "Nodes" option in the top navigation bar to go to the nodes page. There you'll find an overview of the entire datacenter including the health status of each node, IP address, number of registered services, and a leader badge indicating which node is hosting the leading Consul server.
Check the Consul UI tutorial to learn more.
Access Consul containers
You can interact with your containerized Consul datacenter with the Docker CLI.
Docker exec
You can execute Consul CLI commands from your local terminal to your Consul
containers using the docker exec
command.
Docker exec attach
You can also issue commands inside of your container by opening an interactive shell and using the Consul CLI included in the container.
Once inside the consul-client
container you can use the consul members
command to verify you have access to the Consul datacenter.
Configure the ACL system
Consul uses ACLs to secure access to the UI, API, CLI, service communications, and agent communications.
This section will guide you through enabling the ACL system, configuring your agents with ACL tokens, and accessing your Consul datacenter with ACL tokens.
Enable ACLs on all Consul agents
The first step for bootstrapping the ACL system is to enable ACLs on the Consul servers in the agent configuration. In this example, you are configuring the default policy of "deny", which means Consul will deny access to resources unless the request uses a token with explicitly granted privileges, and a down policy of "extend-cache", which means that the agents will extend TTLs for cached tokens during an outage.
It is important to ensure the servers are configured properly before enabling ACLs on the clients. This will reduce any duplicate work and troubleshooting if there is a misconfiguration.
Note
Consul will load all files in the configuration directory, for the
following examples, you can add the configuration as new files. You can also add
them to an existing configuration file, however, you should ensure that the
syntax is valid before applying it to the agent. You can use
consul validate
to check
configuration validity for both JSON and HCL files.
Add the following configuration to the configuration directory on all the agents (servers and clients).
You can copy the file inside the docker container using the docker cp
command.
The agents will need to be restarted to load the new configuration. Take special care to restart the servers one at a time, restarting the leader as last node, and ensure each server has joined and is operating correctly before restarting another.
You can find the leader either from the UI using the leader badge shown nearby
the server node acting as leader ur using the consul operator
command.
You can use the State
column value to identify the leader. In this example the
leader is the node consul-server-2
.
Once identified the leader proceed with a restart of the nodes. You can restart
Consul on the containers using the docker restart
command to restart the
container running the process.
If ACLs are enabled correctly, the leader's logs will contain the following warning and info messages.
Note
Now that you have enabled ACLs, you will need a token to complete any operation. You can't do anything else to the datacenter until you bootstrap the ACL system and generate the bootstrap token. For simplicity, you will use the global management token created during the bootstrap for the remainder of the tutorial.
Create the bootstrap token
Once ACLs have been enabled, you can bootstrap the ACL system to generate the first token, named bootstrap token. The bootstrap token is a global management token with unrestricted privileges.
First, open an interactive shell to any one of your Consul server containers.
From the interactive shell, create the bootstrap token.
Note
The SecretID
is the ACL token used to authenticate API and CLI requests.
On the server where the bootstrap
command was issued, the logs should contain
the following log message.
Since ACLs have been enabled, you will now need to use a token to complete any additional operations. For example, even checking the member list will require a token.
An alternative to using the token on the command line with the -token
flag is to
set the token as an environment variable.
Note
The bootstrap token can only be created once, bootstrapping will be disabled after the bootstrap token is created. Once the ACL system is bootstrapped, ACL tokens can be managed through the ACL API.
Add the agent token to all Consul agents
For this tutorial, all agents (servers and clients) will share the same token for the sake of simplicity. Using the bootstrap token for all agents is NOT a secure practice. In production it is important you generate individual tokens based on more restrictive policies. Check the Access Control System tutorial to learn more about production ACL practices.
It is important to ensure the servers are configured properly before enabling ACLs on the clients. This will reduce any duplicate work and troubleshooting if there is a misconfiguration.
Assign the bootstrap token to all of the Consul agents (servers and clients)
with the set-agent-token
subcommand.
Start with the servers and ensure they are working correctly before applying the
client tokens. Alternatively, you can set the token in the agent configuration
file.
Using the interactive shell, assign the token to your agent.
You will need to complete this process on every agent. Additionally, you will
need to set the CONSUL_HTTP_TOKEN
on every agent to the bootstrap token or use
the token API option.
Your environment is now secured with gossip encryption, TLS encryption, and ACLs.
Additional ACL configuration
This tutorial only covers the basics of ACLs in a Consul datacenter. In your production environment you will need to create more fine grained policies. To learn more about fine-tuning your ACL security, check the production ACL tutorial.
Clean up your environment
To clean up your environment, execute the following command.
Next steps
In this tutorial, you learned to deploy and configure a secure local
containerized Consul datacenter using Docker Compose. You learned how to
interact with your containers using docker exec
, to use CLI and API endpoints
to bootstrap the ACL system and assign tokens to the Consul agents. Finally, you
learned how to clean up your environment.
You can continue learning how to deploy a Consul datacenter in production by completing the Deployment guide. The collection includes securing the datacenter with Access Control Lists, encryption, DNS configuration, and datacenter federation.
You can also extend your Consul skills by exploring the following tutorials:
- ACL bootstrapping guide.
- ACL production guide
- Running Consul on Docker
- Running Consul on Kubernetes
- Service Discovery
- Service Mesh
For additional reference documentation on the official Docker image for Consul, refer to the following websites: