Deploy Consul server

16min
Consul
Interactive

Consul is a service networking solution that enables you to manage secure network connectivity between services and across on-prem and multi-cloud environments and runtimes. Consul offers service discovery, service mesh, traffic management, and automated updates to network infrastructure device. Check out the What is Consul? page to learn more.

In this tutorial, you will configure, deploy, and bootstrap a Consul server on a virtual machine (VM). After deploying Consul, you will interact with Consul using the UI, CLI, and API.

In the following tutorials, you will deploy a demo application and configure it to use Consul's service discovery and service mesh capabilities. During the process, you will learn how to leverage Consul to securely connect your services running on any environment.

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of this tutorial, you have an instance of HashiCups running on four VMs and one empty VM that you will use to deploy Consul server.

Architectural diagram 00

By the end of this tutorial, you will have deployed a Consul server agent running on the extra virtual machine.

Architectural diagram 01

Prerequisites

This tutorial uses an interactive lab to guide you through how to deploy Consul server on a dedicated VM. This lab environment includes all required binaries and sample configurations. We highly recommend using the interactive lab environment to complete this tutorial.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Verify Consul binary

Verify that the VM you want to deploy the Consul server on has the Consul binary.

$ consul version
Consul v1.12.4
Revision 94542765
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Generate Consul server configuration

In order to securely configure Consul by ensuring all communications between the Consul server and clients are inaccessible to unintended agents, you need to provide Consul with:

A gossip encryption key.
A root certificate authority (CA) certificate from a private CA. Consul will use this CA certificate to sign all certificates in the Consul datacenter.
A certificate key-pair for each server you intend to deploy signed by the above CA.

In addition, you want to enable ACLs to ensure every request to your Consul datacenter is authorized.

This tutorial and interactive lab environment uses scripts in the tutorial's GitHub repository to generate these secrets and the Consul configuration files. You will use these scripts to generate your Consul server configuration.

The interactive lab environment includes these scripts. In Consul T1, list the files in your current directory.

$ tree
.
|-- consul_conf.env
|-- generate_consul_server_config.sh
`-- generate_consul_server_tokens.sh

0 directories, 3 files

Clone the example repository in the virtual machine you want to deploy the Consul server on.

$ git clone https://github.com/hashicorp-education/learn-consul-get-started-vms.git

Change into the directory with the newly cloned repository.

$ cd learn-consul-get-started-vms

The script relies on default parameters to generate the configuration files. Set the following default values. Ensure you have permission to write in the specified paths.

Tip

We suggest using an unprivileged user to run Consul for security reasons.

$ export DATACENTER="dc1" \
export DOMAIN="consul" \
export CONSUL_DATA_DIR="/etc/consul/data" \
export CONSUL_CONFIG_DIR="/etc/consul/config" \

Generate all necessary files to configure and run the Consul server agent.

$ ./generate_consul_server_config.sh
Clean existing configuration
Generate Consul folders
Generate agent configuration - agent-server-secure.hcl
Generate server configuration - agent-server-specific.hcl
Generate gossip encryption key configuration - agent-gossip-encryption.hcl
Create CA for Consul datacenter
==> Saved consul-agent-ca.pem
==> Saved consul-agent-ca-key.pem
Create server Certificate and key pair
==> WARNING: Server Certificates grants authority to become a
    server and access all state in the cluster including root keys
    and all ACL tokens. Do not distribute them to production hosts
    that are not server nodes. Store them as securely as CA keys.
==> Using consul-agent-ca.pem and consul-agent-ca-key.pem
==> Saved dc1-server-consul-0.pem
==> Saved dc1-server-consul-0-key.pem
Generate TLS configuration - agent-server-tls.hcl
Generate ACL configuration - agent-server-acl.hcl
Validate configuration
## ...
Configuration is valid!

When the script completes, list the generated files. If you did not change the defaults, you will find it in /etc/consul/config.

$ tree ${CONSUL_CONFIG_DIR}
/etc/consul/config
|-- agent-gossip-encryption.hcl
|-- agent-server-acl.hcl
|-- agent-server-secure.hcl
|-- agent-server-specific.hcl
|-- agent-server-tls.hcl
|-- consul-agent-ca-key.pem
|-- consul-agent-ca.pem
|-- dc1-server-consul-0-key.pem
`-- dc1-server-consul-0.pem

0 directories, 9 files

The scripts generated multiple configuration files to separate the configuration so it is easier to read and tune them for your environment. The following are the generated files and a description of what they do:

The agent-gossip-encryption.hcl file configures gossip encryption.
The agent-server-acl.hcl file configures the ACL system.
The agent-server-secure.hcl file configures a secure server agent. This is an example configuration, and should not be used in production environments.
The agent-server-specific.hcl file contains specific server parameters.
The agent-server-tls.hcl file configures specific TLS encryption.
The *.pem files are certificate key-pair Consul uses to enforce mTLS for datacenter communications.

Visit the agent configuration documentation to interpret the files or to tune them when applying them to your scenario.

Test configuration

Verify the configuration generated is valid.

Tip

Despite the INFO messages, the Consul configuration files are valid.

$ consul validate ${CONSUL_CONFIG_DIR}
## ...
Configuration is valid!

Start Consul server

Now that you have you created the Consul configuration files, you can start the Consul server.

$ consul agent -node=consul -config-dir=${CONSUL_CONFIG_DIR}
==> Starting Consul agent...
           Version: '1.12.4'
           Node ID: '7c0a4f6d-dabf-181b-4cce-dec99289ea29'
         Node name: 'consul'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: true)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: 443, gRPC: 8502, DNS: 53)
      Cluster Addr: 172.20.0.3 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true, Auto-Encrypt-TLS: true

==> Log data will now stream in as it occurs:
## ...
[INFO]  agent: Started DNS server: address=127.0.0.1:8600 network=udp
[INFO]  agent: Started DNS server: address=127.0.0.1:8600 network=tcp
[INFO]  agent: Starting server: address=127.0.0.1:8500 network=tcp protocol=http
[INFO]  agent: Starting server: address=[::]:8443 network=tcp protocol=https
[INFO]  agent: Started gRPC server: address=127.0.0.1:8502 network=tcp
[INFO]  agent: started state syncer
[INFO]  agent: Consul agent running!
## ...
[WARN]  agent: Node info update blocked by ACLs: node=7c0a4f6d-dabf-181b-4cce-dec99289ea29 accessorID=00000000-0000-0000-0000-000000000002
## ...

The output shows the Consul server successfully starting, but some checks are blocked by ACLs. This is because the datacenter is configured to have ACL enabled by default, denying any request that does not present a valid token.

You must bootstrap the ACL system to finish setting up your Consul server.

Configure Consul CLI to interact with Consul server

In order to interact with the Consul server, you need to setup your terminal.

Tip

Ensure ${DATACENTER} ${DOMAIN} and ${CONSUL_CONFIG_DIR} are still defined as environment variables. Refer to the Generate Consul server configuration to redefine these environment variables.

In Consul T2, run the following commands to configure the Consul CLI to interact with the Consul server.

$ export CONSUL_HTTP_ADDR="https://consul:8443" \
export CONSUL_HTTP_SSL=true \
export CONSUL_CACERT="${CONSUL_CONFIG_DIR}/consul-agent-ca.pem" \
export CONSUL_TLS_SERVER_NAME="server.${DATACENTER}.${DOMAIN}" \
export CONSUL_FQDN_ADDR="consul"

In another terminal, run the following commands to configure the Consul CLI to interact with the Consul server. Update CONSUL_ADDRESS to the virtual machine's IP address or DNS name.

$ export CONSUL_HTTP_ADDR="https://CONSUL_ADDRESS:8443" \
export CONSUL_HTTP_SSL=true \
export CONSUL_CACERT="${CONSUL_CONFIG_DIR}/consul-agent-ca.pem" \
export CONSUL_TLS_SERVER_NAME="server.${DATACENTER}.${DOMAIN}" \
export CONSUL_FQDN_ADDR="consul"

Bootstrap ACLs

Verify that the Consul CLI can reach your Consul server. The output informs you that while the Consul CLI can reach your Consul server, Consul's ACLs are blocking the request.

$ consul info
Error querying agent: Unexpected response code: 403 (Permission denied: token with AccessorID '00000000-0000-0000-0000-000000000002' lacks permission 'agent:read' on "consul")

Bootstrap the Consul ACL system and save the output in a file named acl-token-bootstrap.json.

$ consul acl bootstrap --format json > ./acl-token-bootstrap.json

The command generates a management token with full permissions over your datacenter. The output will look similar to the following. The management token is the value associated with the SecretID key.

acl-token-bootstrap.json

{
  "CreateIndex": 24,
  "ModifyIndex": 24,
  "AccessorID": "a09ee518-f4bf-f1a0-7bdd-2124ef1157af",
  "SecretID": "4becaa33-8fa3-a4c4-6ac5-fed2e20ee3a3",
  "Description": "Bootstrap Token (Global Management)",
  "Policies": [
    {
      "ID": "00000000-0000-0000-0000-000000000001",
      "Name": "global-management"
    }
  ],
  "Local": false,
  "CreateTime": "2022-08-23T14:17:57.686753287Z",
  "Hash": "X2AgaFhnQGRhSSF/h0m6qpX1wj/HJWbyXcxkEM/5GrY="
}

Extract the management token from the file and set it to the CONSUL_HTTP_TOKEN environment variable.

$ export CONSUL_HTTP_TOKEN=`cat ./acl-token-bootstrap.json | jq -r ".SecretID"`

After you set the management token, the Consul server logs (Consul T1) will show a message to indicate that you have successfully initialized the ACL system.

[WARN]  agent.server.acl: failed to remove bootstrap file: error="remove /etc/consul/data/acl-bootstrap-reset: no such file or directory"
[INFO]  agent.server.acl: ACL bootstrap completed

Now that you have bootstrap the ACL system, you can interact with the Consul server.

$ consul info
agent:
  check_monitors = 0
  check_ttls = 0
  checks = 0
  services = 0
build:
  prerelease =
  revision = 8c237209
  version = 1.13.0
  version_metadata =
consul:
  acl = enabled
  bootstrap = true
  known_datacenters = 1
  leader = true
  leader_addr = 172.20.0.3:8300
  server = true
## ...

Create server tokens

In the Consul server logs, notice that it still contains warnings related to ACL permissions. This is because the server tries to update the node information but the ACL system blocks the requests.

[WARN]  agent: Coordinate update blocked by ACLs: accessorID=00000000-0000-0000-0000-000000000002
[WARN]  agent: Node info update blocked by ACLs: node=c4d03d9d-911f-07a6-2c72-b2bfa924c44b accessorID=00000000-0000-0000-0000-000000000002

In order to complete configuring the Consul server, you need to create the tokens for the server agents and assign them to the server.

The generate_consul_sever_tokens.sh script automates the process of creating policies and tokens for your Consul server. This script generates ACL policies for Consul DNS service and for the server agent and applies them to the Consul server.

$ ./generate_consul_server_tokens.sh 
Create ACL policies and tokens
Setup ACL tokens for Server
ACL token "agent" set successfully
ACL token "default" set successfully

You will create ACL tokens for the Consul DNS service and your server agent. For each token, you need to define the policy file, create the Consul policy, create the ACLs token for the policy, and assign the token to the server agent.

Define the policy files

First, define the DNS policy file.

$ tee ./acl-policy-dns.hcl > /dev/null << EOF
## dns-request-policy.hcl
node_prefix "" {
  policy = "read"
}
service_prefix "" {
  policy = "read"
}
# Required if you use prepared queries
query_prefix "" {
  policy = "read"
}
EOF

Then, define the server policy file.

$ tee ./acl-policy-server-node.hcl > /dev/null << EOF
## consul-server-one-policy.hcl
node_prefix "consul" {
  policy = "write"
}
EOF

Create the policy files

First, create the DNS policy using the previously created policy definition.

$ consul acl policy create -name 'acl-policy-dns' -description 'Policy for DNS endpoints' -rules @./acl-policy-dns.hcl

Then, create the server policy using the previously created policy definition.

$ consul acl policy create -name 'acl-policy-server-node' -description 'Policy for Server nodes' -rules @./acl-policy-server-node.hcl

Create ACL tokens for each policy

First, create the DNS token.

$ consul acl token create -description 'DNS - Default token' -policy-name acl-policy-dns --format json > ./acl-token-dns.json

You can find the ACL token specific to DNS in the generated file.

acl-token-dns.json

{
  "CreateIndex": 44,
  "ModifyIndex": 44,
  "AccessorID": "270eeab9-0d0e-a0c7-8dca-4cde065dc2ea",
  "SecretID": "5e785a4d-035f-4834-82fc-e71a61e7046f",
  "Description": "DNS - Default token",
  "Policies": [
    {
      "ID": "3f6d39c8-844d-0412-cb06-c8f6c0cb4a3e",
      "Name": "acl-policy-dns"
    }
  ],
  "Local": false,
  "CreateTime": "2022-08-23T14:34:43.223378819Z",
  "Hash": "iqlU4q6wel2SC7PtAwzvgb8mYjyU8NnmT9C+kS1hsNM="
}

Then, create the server node token.

$ consul acl token create -description "server agent token" -policy-name acl-policy-server-node  --format json > ./server-acl-token.json

You can find the ACL token specific to your server node in the generated file.

acl-token-dns.json

{
  "CreateIndex": 47,
  "ModifyIndex": 47,
  "AccessorID": "0a6f1b1b-fc38-1979-4b44-c7cee91966ed",
  "SecretID": "47dc9322-78a2-51d1-c1ac-bcc16959b635",
  "Description": "server agent token",
  "Policies": [
    {
      "ID": "d01854bc-6298-6bce-76e1-0c08bebd9dc9",
      "Name": "acl-policy-server-node"
    }
  ],
  "Local": false,
  "CreateTime": "2022-08-23T14:36:29.932621394Z",
  "Hash": "H3wfyn0nqViLmerl32uM3uRCAcY3JkEb8VX22NRnvBs="
}

Assign tokens to the server agent

After you created the tokens, you will now assign them to your Consul agent.

First, define two environment variables containing the tokens.

$ export DNS_TOKEN=`cat ./acl-token-dns.json | jq -r ".SecretID"` \
export SERVER_TOKEN=`cat ./server-acl-token.json | jq -r ".SecretID"`

Then, assign the DNS token to the server.

$ consul acl set-agent-token default ${DNS_TOKEN}
ACL token "default" set successfully

Next, assign the server token to the server.


$ consul acl set-agent-token agent ${SERVER_TOKEN}
ACL token "agent" set successfully

After you create the server tokens, your Consul logs will show the updated ACL tokens.

## ...
[INFO]  agent: Updated agent's ACL token: token=agent
## ...
[INFO]  agent: Updated agent's ACL token: token=default
## ...

Interact with Consul server

Now that you have completed configuring and deploying your Consul server, you will interact with it. Consul provides different ways to retrieve information about the datacenter — select the tab(s) for your preferred method.

Use the Consul CLI to retrieve members in your Consul datacenter.

$ consul members
Node    Address          Status  Type    Build   Protocol  DC   Partition  Segment
consul  10.2.31.35:8301  alive   server  1.12.4  2         dc1  default    <all>ux

Check the Consul CLI commands reference for the full list of available commands.

Query the Consul API to retrieve members in your Consul datacenter.

$  curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --connect-to server.${DATACENTER}.${DOMAIN}:8443:consul:8443 \
  --cacert ${CONSUL_CONFIG_DIR}/consul-agent-ca.pem \
  https://server.${DATACENTER}.${DOMAIN}:8443/v1/agent/members | jq

This will respond with something similar to the following:

[
  {
    "Name": "consul",
    "Addr": "10.2.31.35",
    "Port": 8301,
    "Tags": {
      "acls": "1",
      "bootstrap": "1",
      "build": "1.12.4:94542765",
      "dc": "dc1",
      "ft_fs": "1",
      "ft_si": "1",
      "id": "5b015410-0be5-d469-b5a7-ec609484c035",
      "port": "8300",
      "raft_vsn": "3",
      "role": "consul",
      "segment": "",
      "use_tls": "1",
      "vsn": "2",
      "vsn_max": "3",
      "vsn_min": "2",
      "wan_join_port": "8302"
    },
    "Status": 1,
    "ProtocolMin": 1,
    "ProtocolMax": 5,
    "ProtocolCur": 2,
    "DelegateMin": 2,
    "DelegateMax": 5,
    "DelegateCur": 4
  }
]

Check the Consul API reference for the full list of available endpoints.

Click on the Consul UI tab.

You will find a page that reminds you that you are not authorized to get information from your Consul datacenter unless you successfully authenticate using a token.

403 Unauthorized

To authenticate, click on Login and enter the management token you created while bootstrapping the Consul server.

Once authenticated, Consul will redirected to the Services page where you will find a healthy and available consul service.

Service Page

Interact with Consul KV

Consul includes a key/value (KV) store that you can use to manage your service's configuration. Even though you can use the KV store using the CLI, API, and UI, this tutorial only covers the CLI and API methods. Select the tab(s) for your preferred method.

Create a key named db_port with a value of 5432.

$ consul kv put consul/configuration/db_port 5432
Success! Data written to: consul/configuration/db_port

Then, retrieve the value.

$ consul kv get consul/configuration/db_port  
5432

Create a key named db_port with a value of 5432.

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --connect-to server.${DATACENTER}.${DOMAIN}:8443:consul:8443 \
  --cacert ${CONSUL_CONFIG_DIR}/consul-agent-ca.pem \
  --request PUT \
  --data '1' \
  https://server.${DATACENTER}.${DOMAIN}:8443/v1/kv/consul/configuration/db_port

This will respond with something similar to the following:

true

Then, retrieve the value.

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --connect-to server.${DATACENTER}.${DOMAIN}:8443:consul:8443 \
  --cacert ${CONSUL_CONFIG_DIR}/consul-agent-ca.pem \
  https://server.${DATACENTER}.${DOMAIN}:8443/v1/kv/consul/configuration/db_port | jq

This will respond with something similar to the following:

[
  {
    "LockIndex": 0,
    "Key": "consul/configuration/server_number",
    "Flags": 0,
    "Value": "MQ==",
    "CreateIndex": 129,
    "ModifyIndex": 129
  }
]

Notice the response returns the base64 encoded value.

To retrieve the raw value, extract the value and then base64 decode it.

$ curl --silent \
  --header "X-Consul-Token: $CONSUL_HTTP_TOKEN" \
  --connect-to server.${DATACENTER}.${DOMAIN}:8443:consul:8443 \
  --cacert ${CONSUL_CONFIG_DIR}/consul-agent-ca.pem \
  https://server.${DATACENTER}.${DOMAIN}:8443/v1/kv/consul/configuration/db_port | jq -r ".[].Value" | base64 --decode

This will respond with something similar to the following:

Interact with Consul DNS

Consul also provides you with a fully featured DNS server that you can use to resolve the IPs for your services. By default, Consul DNS service is configured to listen on port 8600.

$ dig @127.0.0.1 -p 8600 consul.service.consul

; <<>> DiG 9.16.27-Debian <<>> @127.0.0.1 -p 8600 consul.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 48200
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;consul.service.consul.        IN    A

;; ANSWER SECTION:
consul.service.consul.    0    IN    A    172.20.0.3

;; Query time: 0 msec
;; SERVER: 127.0.0.1#8600(127.0.0.1)
;; WHEN: Tue Aug 23 15:45:53 UTC 2022
;; MSG SIZE  rcvd: 66

Next steps

In this tutorial, you deployed a Consul server on a VM. After deploying Consul, you interacted with Consul using the CLI, UI, and API.

Even when deployed without Consul clients, you can:

use Consul's KV store as a centralized configuration management tool. You can use this and consul-template to configure your services automatically.
use Consul server as a DNS server> You can use this to register and resolve external services in your network.

In the next tutorial, you will deploy Consul clients on the VMs hosting your application. Then, you will register the services running on each server and set up health checks for each services. This enables service discovery using Consul's distributed health check system and DNS.

For more information about the topics covered in this tutorial, refer to the following resources:

Back to Collection