Create a Local Consul Datacenter

In this tutorial, you'll create your first datacenter with multiple members.

When a new Consul agent starts, it doesn't know about other agents; it is essentially a datacenter with one member. To add a new agent to an existing datacenter you give it the IP address of any other agent in the datacenter (either a client or a server), which causes the new agent to join the datacenter. Once the agent is a member of the new datacenter, it automatically learns about the other agents via gossip.

In this tutorial, you will join two agents together to create a two-member Consul datacenter.

Prerequisites

Consul is a distributed application that is designed to have one agent per machine. To run two agents on the same computer you will need to install VirtualBox, and Vagrant, which will run virtual machines to simulate a distributed environment.

Set up the environment

Make a directory to store Vagrant's configuration for this tutorial.

$ mkdir consul-getting-started-join

Create a new file in the directory called Vagrantfile and paste the code below into it. This file will instruct Vagrant to create two virtual machines on your computer with the Consul binary preinstalled.

# -*- mode: ruby -*-
# vi: set ft=ruby :

$script = <<SCRIPT
echo "Installing dependencies ..."
sudo apt-get update
sudo apt-get install -y unzip curl jq dnsutils
echo "Determining Consul version to install ..."
CHECKPOINT_URL="https://checkpoint-api.hashicorp.com/v1/check"
if [ -z "$CONSUL_DEMO_VERSION" ]; then
    CONSUL_DEMO_VERSION=$(curl -s "${CHECKPOINT_URL}"/consul | jq .current_version | tr -d '"')
fi
echo "Fetching Consul version ${CONSUL_DEMO_VERSION} ..."
cd /tmp/
curl -s https://releases.hashicorp.com/consul/${CONSUL_DEMO_VERSION}/consul_${CONSUL_DEMO_VERSION}_linux_amd64.zip -o consul.zip
echo "Installing Consul version ${CONSUL_DEMO_VERSION} ..."
unzip consul.zip
sudo chmod +x consul
sudo mv consul /usr/bin/consul
sudo mkdir /etc/consul.d
sudo chmod a+w /etc/consul.d
SCRIPT

# Specify a Consul version
CONSUL_DEMO_VERSION = ENV['CONSUL_DEMO_VERSION']

# Specify a custom Vagrant box for the demo
DEMO_BOX_NAME = ENV['DEMO_BOX_NAME'] || "debian/stretch64"

# Vagrantfile API/syntax version.
# NB: Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.box = DEMO_BOX_NAME

  config.vm.provision "shell",
                          inline: $script,
                          env: {'CONSUL_DEMO_VERSION' => CONSUL_DEMO_VERSION}

  config.vm.define "n1" do |n1|
      n1.vm.hostname = "n1"
      n1.vm.network "private_network", ip: "172.20.20.10"
  end

  config.vm.define "n2" do |n2|
      n2.vm.hostname = "n2"
      n2.vm.network "private_network", ip: "172.20.20.11"
  end
end

Boot your two virtual machines. This may take a moment to download everything needed for the environment to spin up.

$ vagrant up

Bringing machine 'n1' up with 'virtualbox' provider...
Bringing machine 'n2' up with 'virtualbox' provider...
...

Start the agents

In previous tutorials, you used a single agent in development mode to test Consul's functionality. However, you should never run development agents in production. In this tutorial, you will configure your first Consul agent to run in server mode instead, via the following command line flags. (In production you would provide these settings to consul in a configuration file.)

• server- Providing this flag specifies that you want the agent to start in server mode.

• -bootstrap-expect - This tells the Consul server how many servers the datacenter should have in total. All the servers will wait for this number to join before bootstrapping the replicated log, which keeps data consistent across all the servers. Because you are setting up a one-server datacenter, you'll set this value to 1.

• -node - Each node in a datacenter must have a unique name. By default, Consul uses the hostname of the machine, but you'll manually override it, and set it to agent-one.

• -bind - This is the address that this agent will listen on for communication from other Consul members. It must be accessible by all other nodes in the datacenter. If you don't set a bind address Consul will try to listen on all IPv4 interfaces and will fail to start if it finds multiple private IPs. Since production servers often have multiple interfaces, you should always provide a bind address. In this case it is 172.20.20.10, which you specified as the address of the first VM in your Vagrantfile.

• data-dir - This flag tells Consul agents where they should store their state, which can include sensitive data like ACL tokens for both servers and clients. In production deployments you should be careful about the permissions for this directory.

• config-dir - This flag tells consul where to look for its configuration. You will set it to a standard location: /etc/consul.d.

Start Consul server

Once the environment is up, ssh into the first node, n1, to begin the configuration of your datacenter.

$ vagrant ssh n1

Linux n1 4.9.0-12-amd64 #1 SMP Debian 4.9.210-1 (2020-01-20) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

vagrant@n1:~$ 

Start your first Consul agent by running the following command. Consul will start up in the foreground of your terminal window.

vagrant@n1:~$ consul agent \
  -server \
  -bootstrap-expect=1 \
  -node=agent-one \
  -bind=172.20.20.10 \
  -data-dir=/tmp/consul \
  -config-dir=/etc/consul.d

...

Start Consul client

Open a new terminal window and move into consul-getting-started-join directory. Then ssh into your second virtual machine.

$ vagrant ssh n2

Now start up your second Consul agent in client mode. You'll set the bind address to the IP address of the second VM (172.20.20.11, specified in the Vagrantfile) and the name to agent-two. Don't include the -server flag and the agent will start in client mode. Consul will run in the foreground of your terminal.

vagrant@n2:~$ consul agent \
  -node=agent-two \
  -bind=172.20.20.11 \
  -enable-script-checks=true \
  -data-dir=/tmp/consul \
  -config-dir=/etc/consul.d

...

Check datacenter membership

Now you have two Consul agents running: one server and one client. The two agents still don't know about each other and each comprise their own single-node datacenters.

Verify this by ssh-ing into each VM and checking each agent's membership information. You'll need to open a new terminal window and change into the consul-getting-started-join directory.

Check the membership of agent-two.

$ vagrant ssh n2

Linux n2 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
You have new mail.
Last login: Fri Aug  2 23:42:33 2019 from 10.0.2.2

vagrant@n2:~$

vagrant@n2:~$ consul members

Node       Address            Status  Type    Build  Protocol  DC   Segment
agent-two  172.20.20.11:8301  alive   client  1.9.3  2         dc1  <default>

Open a new terminal window and change directories into consul-getting-started-join.

Check the membership of agent-one.

$ vagrant ssh n1

Linux n1 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u2 (2019-05-13) x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
You have new mail.
Last login: Fri Aug  2 20:37:46 2019 from 10.0.2.2

vagrant@n1:~$ 

vagrant@n1:~$ consul members

Node       Address            Status  Type    Build  Protocol  DC   Segment
agent-one  172.20.20.10:8301  alive   server  1.9.3  2         dc1  <all>

Join the agents

You're now ready to create your multi-agent datacenter. Open the terminal window where you are ssh-ed into the second VM, and run the consul join command on the Consul client, giving it the bind address of the Consul server.

vagrant@n2:~$ consul join 172.20.20.10

Successfully joined cluster by contacting 1 nodes.

In the same window, run consul members again and you should get both agents listed in the output.

vagrant@n2:~$ consul members

Node       Address            Status  Type    Build  Protocol  DC   Segment
agent-one  172.20.20.10:8301  alive   server  1.9.3  2         dc1  <all>
agent-two  172.20.20.11:8301  alive   client  1.9.3  2         dc1  <default>

Switch to the terminal window where your Consul server is running on the first VM, and you'll notice some log output indicating that agent two joined it.

Now switch to the terminal where your client is running on the second VM. You'll notice that the client had been throwing warnings and errors indicating that no servers were available. When the client learned about the server, it stopped throwing errors and synced its node information.

...
[WARN]  agent.router.manager: No servers available
[ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
[WARN]  agent.router.manager: No servers available
[ERROR] agent.anti_entropy: failed to sync remote state: error="No known Consul servers"
[INFO]  agent: (LAN) joining: lan_addresses=[172.20.20.10]
[INFO]  agent.client.serf.lan: serf: EventMemberJoin: agent-one 172.20.20.10
[INFO]  agent: (LAN) joined: number_of_nodes=1
[INFO]  agent.client: adding server: server="agent-one (Addr: tcp/172.20.20.10:8300) (DC: dc1)"
[INFO]  agent: Synced node info
...

Consul clients can not function without a server. All datacenters must have at least one agent running in server mode for Consul to function correctly.

In datacenters with more than one server, more than half of the servers must be in communication with each other at all times for the datacenter to function correctly. This is called maintaining quorum. You can learn more about the quorum requirements of servers in the architecture documentation.

Switch to the window where you are ssh-ed onto the second VM and run consul members on the client. The client will also list both agents as members.

vagrant@n2:~$ consul members

Node       Address            Status  Type    Build  Protocol  DC   Segment
agent-one  172.20.20.10:8301  alive   server  1.9.3  2         dc1  <all>
agent-two  172.20.20.11:8301  alive   client  1.9.3  2         dc1  <default>

Notes on auto-join

In production, new Consul agents should automatically join the datacenter without human intervention. You can configure Consul to automatically discover new agents in AWS, Google Cloud or Azure by adding the relevant cloud auto join object to your Consul configuration file. This will allow a new node to join the datacenter without any hard-coded configuration.

Alternatively, you can provide hard-coded addresses of known Consul agents to new agents using the -retry-join flag.

Query a node

You can query Consul agents using the DNS interface or HTTP API.

For the DNS API, the structure of the names is NAME.node.consul or NAME.node.DATACENTER.consul. If the datacenter is omitted, Consul will only search the local datacenter.

From the terminal window where you are ssh-ed into agent-one, query the DNS interface for the address of agent-two.

vagrant@n1:~$ dig @127.0.0.1 -p 8600 agent-two.node.consul

...

;; QUESTION SECTION:
;agent-two.node.consul. IN  A

;; ANSWER SECTION:
agent-two.node.consul.  0 IN    A   172.20.20.11

...

The ability to look up nodes in addition to services is useful for system administration, in addition to service discovery. For example, knowing the address of the node to SSH into is as easy as making the node a part of the Consul datacenter and querying it.

Stop the agents

Stop both of your agents gracefully by either typing Ctrl-c in the terminal windows where they are running, or issuing the consul leave command.

Clean up the environment

Vagrant will automatically stop and power down the virtual machines it created, remove their hard disks from your machine, and free up all of the disk space and RAM they consume.

It will not get rid of the directory you created or the Vagrant file it contains, so if you would like to re-run this tutorial, all you need to do is issue vagrant up again from inside the consul-getting-started-join directory.

Clean up your virtual environment by running the following command from within the consul-getting-started-join directory.

$ vagrant destroy

    n2: Are you sure you want to destroy the 'n2' VM? [y/N] y
==> n2: Forcing shutdown of VM...
==> n2: Destroying VM and associated drives...
    n1: Are you sure you want to destroy the 'n1' VM? [y/N] y
==> n1: Forcing shutdown of VM...
==> n1: Destroying VM and associated drives...

Next steps

In this tutorial, you set up a multi-agent Consul datacenter by joining two Consul agents, a server and a client. Get more hands-on experience with Consul's features with the following tutorials:

• Get Started with Consul Service Mesh
• Consul on Kubernetes
• Consul and Nomad