HashiConf Our community conference is taking place in San Francisco and online October 10-12. Register now
Consul
Consul on VMs

Securely connect your services with Consul service mesh

  • 13min

  • Consul
  • Interactive

In the previous tutorial, you deployed Consul client agents and registered services to your Consul catalog.

In this tutorial, you will introduce zero trust security in your network by implementing Consul service mesh. This will enable secure service-to-service communication and allow you to leverage Consul's full suite of features.

In order to do this, you will edit the service definitions on your Consul clients, launch Envoy sidecar proxies, and create service intentions to allow traffic across your services in your network.

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you have a Consul datacenter with one server and four clients running on VMs. The services connect directly to each other using the VMs address and access every service in the network.

Architectural diagram 02

By the end of this tutorial, you will have a fully deployed Consul service mesh with Envoy sidecar proxies running alongside each service. The services will be configured so they cannot be reachable, unless expliclty allowed through Consul service intentions.

Architectural diagram 03

Prerequisites

This tutorial uses an interactive lab to guide you through how to setting up service mesh on your VM workloads. This lab environment includes all required binaries and sample configurations. We highly recommend using the interactive lab environment to complete this tutorial.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Verify Consul binary

Verify that the VM you want to deploy the Consul server on has the Consul binary.

$ consul version
Consul v1.12.4
Revision 94542765
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

Verify Envoy installation on the VMs

In the interactive lab environment, click on each tab associated with the client nodes (NGINX, Frontend, API, and Database) and verify Envoy is installed.

$ envoy --version
/opt/bin/envoy  version: 0512f18b764828497febd0f6dcecc1861003d614/1.21.5/Clean/RELEASE/BoringSSL

Configure environment

This tutorial and interactive lab environment uses scripts in the tutorial's GitHub repository to generate the Consul configuration files for your client agents.

The interactive lab environment includes these scripts and the pre-configured Consul server. In Operator, list the files in your current directory.

$ tree
.
|-- assets
|   |-- acl-policy-dns.hcl
|   |-- acl-policy-server-node.hcl
|   |-- acl-token-bootstrap.json
|   |-- acl-token-dns.json
|   |-- agent-client-hashicups-api-acl-tokens.hcl
|   |-- agent-client-hashicups-db-acl-tokens.hcl
|   |-- agent-client-hashicups-frontend-acl-tokens.hcl
|   |-- agent-client-hashicups-nginx-acl-tokens.hcl
|   |-- agent-client-secure.hcl
|   |-- agent-gossip-encryption.hcl
|   |-- agent-server-acl.hcl
|   |-- agent-server-secure.hcl
|   |-- agent-server-specific.hcl
|   |-- agent-server-tls.hcl
|   |-- consul-agent-ca-key.pem
|   |-- consul-agent-ca.pem
|   |-- dc1-server-consul-0-key.pem
|   |-- dc1-server-consul-0.pem
|   |-- server-acl-token.json
|   |-- svc-hashicups-api.hcl
|   |-- svc-hashicups-db.hcl
|   |-- svc-hashicups-frontend.hcl
|   `-- svc-hashicups-nginx.hcl
|-- consul_conf.env
|-- filebrowser.db
|-- generate_consul_client_config_mesh.sh
|-- logs
|   |-- filebrowser.log
|   `-- provision.log
`-- start_scenario.sh

2 directories, 29 files

In addition, the Consul client agents must have following Consul resources that are compatible with your Consul server:

  • The root CA certificate that you generated during the Consul server deployment.
  • The gossip encryption key that you used to encrypt gossip communication.

Define these files' path. The script will include resources in the auto-generated client configuration files

$ export CA_CERT="/home/app/assets/consul-agent-ca.pem"; \
export GOSSIP_CONFIG="/home/app/assets/agent-gossip-encryption.hcl"

Finally, the script uses Consul CLI to interact with Consul server and generate tokens for the Consul client agents. In order to interact with the Consul server, you need to setup your terminal.

Run the following commands to configure the Consul CLI to interact with the Consul server.

$ export CONSUL_HTTP_ADDR="https://consul:8443" \
export CONSUL_HTTP_SSL=true \
export CONSUL_CACERT="$CA_CERT" \
export CONSUL_TLS_SERVER_NAME="server.${DATACENTER}.${DOMAIN}" \
export CONSUL_FQDN_ADDR="consul"

You also need to provide an ACL token to authenticate to your Consul server. In the previous tutorial, you generated an ACL token and stored it in a file.

acl-token-bootstrap.json
{
  "CreateIndex": 24,
  "ModifyIndex": 24,
  "AccessorID": "a09ee518-f4bf-f1a0-7bdd-2124ef1157af",
  "SecretID": "4becaa33-8fa3-a4c4-6ac5-fed2e20ee3a3",
  "Description": "Bootstrap Token (Global Management)",
  "Policies": [
    {
      "ID": "00000000-0000-0000-0000-000000000001",
      "Name": "global-management"
    }
  ],
  "Local": false,
  "CreateTime": "2022-08-23T14:17:57.686753287Z",
  "Hash": "X2AgaFhnQGRhSSF/h0m6qpX1wj/HJWbyXcxkEM/5GrY="
}

Retrieve this value and set it to an environment variable named CONSUL_HTTP_TOKEN.

$ export CONSUL_HTTP_TOKEN=`cat /home/app/assets/acl-token-bootstrap.json | jq -r ".SecretID"`

Verify your Consul CLI can interact with your Consul server.

$ consul members
Node    Address           Status  Type    Build   Protocol  DC   Partition  Segment
consul              10.2.24.216:8301  alive   server  1.12.4  2         dc1  default    <all>
hashicups-api       10.2.22.84:8301   alive   client  1.12.4  2         dc1  default    <default>
hashicups-db        10.2.14.224:8301  alive   client  1.12.4  2         dc1  default    <default>
hashicups-frontend  10.2.5.209:8301   alive   client  1.12.4  2         dc1  default    <default>
hashicups-nginx     10.2.0.35:8301    alive   client  1.12.4  2         dc1  default    <default>

Generate Consul client configuration

Once you have set up your environment by defining the defaults for your script and configuring the Consul CLI, generate the Consul clients configuration files.

$ ./generate_consul_client_config_mesh.sh
Create service db configuration
Create service api configuration
Create service frontend configuration
Create service nginx configuration
Create intention configuration files

The script creates new service definitions for the services to add them to the service mesh. In addition, it creates several global configuration files for the Consul service mesh control plane.

$ tree service_mesh_configs/
service_mesh_configs/
service_mesh_configs/
|-- global
|   |-- intention-api.hcl
|   |-- intention-api.json
|   |-- intention-db.hcl
|   |-- intention-db.json
|   |-- intention-frontend.hcl
|   `-- intention-frontend.json
|-- hashicups-api
|   `-- svc-hashicups-api.hcl
|-- hashicups-db
|   `-- svc-hashicups-db.hcl
|-- hashicups-frontend
|   `-- svc-hashicups-frontend.hcl
`-- hashicups-nginx
    `-- svc-hashicups-nginx.hcl

5 directories, 10 files

The new service definition includes an extra definition for the service upstreams. An upstream is a service that the defined service needs to communicate with.

The following is the new service definition for the NGINX service. Notice the new upstream definition — the NGINX service needs to connect to the frontend and API services. Consul will instruct the sidecar proxies to bind the upstream services to the relevant ports.

service_mesh_configs/hashicups-nginx/svc-hashicups-nginx.hcl
## svc-hashicups-nginx.hcl
service {
  name = "hashicups-nginx"
  id = "hashicups-nginx-1"
  tags = ["v1"]
  port = 80
  
  connect {
    sidecar_service {
      proxy {
        upstreams = [
          {
            destination_name = "hashicups-frontend"
            local_bind_port = 3000
          },
          {
            destination_name = "hashicups-api"
            local_bind_port = 8081
          }
        ]
      }
    }
  }

  check {
    id =  "check-hashicups-nginx",
    name = "Product hashicups-nginx status check",
    service_id = "hashicups-nginx-1",
    tcp  = "hashicups-nginx:80",
    interval = "1s",
    timeout = "1s"
  }
}

Review and create intentions

The initial Consul configuration denied all service connections by default. We recommend this setting in production environments to follow the "least-privilege" principle, by restricting all network access unless explicitly defined.

Intentions let you allow and restrict access between services. Intentions are destination-orientated — this means you create the intentions for the destination, then define which services can access it.

The following intentions are required for HashiCups:

Tip

Notice these descriptions show traffic going from the destination to the source service.

  1. The db service needs to be reached by the api service.
  2. The api service needs to be reached by the nginx services.
  3. The frontend service needs to be reached by the nginx service.

The script already generated these configuration files and saved them in the service_mesh_configs/global directory. Create the service intentions.

Use config write to create the following intentions.

Create the intentions for the db service.

$ consul config write service_mesh_configs/global/intention-db.hcl  
Config entry written: service-intentions/db

Create the intentions for the api service.

$ consul config write service_mesh_configs/global/intention-api.hcl 
Config entry written: service-intentions/api

Create the intentions for the frontend service.

$ consul config write service_mesh_configs/global/intention-frontend.hcl 
Config entry written: service-intentions/frontend

Apply new Consul client configuration

Before you can apply the new client configuration, you must copy them into each Consul client VMs.

Tip

In the interactive lab environment, the HashiCups application nodes have a running SSH server. As a result, you can use ssh and scp commands to perform the following operations. If the nodes in your personal environment does not have an SSH server, you may need to use a different approach to create the configuration directories and copy the files.

$ scp service_mesh_configs/hashicups-db/* hashicups-db:/etc/consul/config/; \
scp service_mesh_configs/hashicups-api/* hashicups-api:/etc/consul/config/; \
scp service_mesh_configs/hashicups-frontend/* hashicups-frontend:/etc/consul/config/; \
scp service_mesh_configs/hashicups-nginx/* hashicups-nginx:/etc/consul/config/

Start sidecar proxies on client VMs

Once you copied the configuration files on the different VMs, login on each Consul client VMs and restart the Consul agent.

Start sidecar proxy for Database

Log into the virtual machine that hosts the database and reload the Consul configuration.

Tip

For the interactive lab environment, select the tab that corresponds with the service — in this case, Database.

$ consul reload
Configuration reload triggered

Then, configure terminal to use Consul client token.

$ export CONSUL_CONFIG_DIR="/etc/consul/config"; \
export CONSUL_HTTP_TOKEN=`cat ${CONSUL_CONFIG_DIR}/agent-client-*-acl-tokens.hcl | grep agent | awk '{print $3}'| sed 's/"//g'`

Finally, start the Envoy sidecar proxy for the service. Notice this command configures the sidecar proxy specifically for the database service (-sidecar-for hashicups-db-1). If you host multiple services on a VM, you must start an Envoy sidecar proxy for every service.

$ /usr/local/bin/consul connect envoy -envoy-binary /usr/local/bin/envoy -sidecar-for hashicups-db-1
## ...
[info][upstream] [source/common/upstream/cluster_manager_impl.cc:207] cm init: all clusters initialized
[info][main] [source/server/server.cc:849] all clusters initialized. initializing init manager
[info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'public_listener:0.0.0.0:21000'
[info][config] [source/server/listener_manager_impl.cc:784] all dependencies initialized. starting workers
## ...

### Start sidecar proxy for API

Log into the virtual machine that hosts the API and reload the Consul configuration.

$ consul reload
Configuration reload triggered

Then, configure terminal to use Consul client token.

$ export CONSUL_CONFIG_DIR="/etc/consul/config"; \
export CONSUL_HTTP_TOKEN=`cat ${CONSUL_CONFIG_DIR}/agent-client-*-acl-tokens.hcl | grep agent | awk '{print $3}'| sed 's/"//g'`

Finally, start the Envoy sidecar proxy for the service.

$ /usr/local/bin/consul connect envoy -envoy-binary /usr/local/bin/envoy -sidecar-for hashicups-api-1
## ...
[info][upstream] [source/common/upstream/cluster_manager_impl.cc:207] cm init: all clusters initialized
[info][main] [source/server/server.cc:849] all clusters initialized. initializing init manager
[info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'public_listener:0.0.0.0:21000'
[info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'db:127.0.0.1:5432'
[info][config] [source/server/listener_manager_impl.cc:784] all dependencies initialized. starting workers
## ...

Start sidecar proxy for Frontend

Log into the virtual machine that hosts the frontend and reload the Consul configuration.

$ consul reload
Configuration reload triggered

Then configure terminal to use Consul client token.

$ export CONSUL_CONFIG_DIR="/etc/consul/config"; \
export CONSUL_HTTP_TOKEN=`cat ${CONSUL_CONFIG_DIR}/agent-client-*-acl-tokens.hcl | grep agent | awk '{print $3}'| sed 's/"//g'`

Finally, start the Envoy sidecar proxy for the service.

$ /usr/local/bin/consul connect envoy -envoy-binary /usr/local/bin/envoy -sidecar-for hashicups-frontend-1
## ...
[info][upstream] [source/common/upstream/cluster_manager_impl.cc:207] cm init: all clusters initialized
[info][main] [source/server/server.cc:849] all clusters initialized. initializing init manager
[info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'api:127.0.0.1:8081'
[info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'public_listener:0.0.0.0:21000'
[info][config] [source/server/listener_manager_impl.cc:784] all dependencies initialized. starting workers
## ...

Start sidecar proxy for NGINX

Log into the virtual machine that hosts NGINX and reload the Consul configuration.

$ consul reload
Configuration reload triggered

Then, configure terminal to use Consul client token.

$ export CONSUL_CONFIG_DIR="/etc/consul/config"; \
export CONSUL_HTTP_TOKEN=`cat ${CONSUL_CONFIG_DIR}/agent-client-*-acl-tokens.hcl | grep agent | awk '{print $3}'| sed 's/"//g'`

Finally, start the Envoy sidecar proxy for the service.

$ /usr/local/bin/consul connect envoy -envoy-binary /usr/local/bin/envoy -sidecar-for hashicups-nginx-1
## ...
[info][upstream] [source/common/upstream/cluster_manager_impl.cc:182] cm init: initializing secondary clusters
[info][upstream] [source/common/upstream/cluster_manager_impl.cc:207] cm init: all clusters initialized
[info][main] [source/server/server.cc:849] all clusters initialized. initializing init manager
[info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'public_listener:0.0.0.0:21000'
[info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'frontend:127.0.0.1:3000'
[info][upstream] [source/server/lds_api.cc:77] lds: add/update listener 'api:127.0.0.1:8081'
[info][config] [source/server/listener_manager_impl.cc:784] all dependencies initialized. starting workers
## ...

Restart services to listen on localhost

Navigate to the Operator terminal and restart the services to listen only on localhost. This ensures that the services cannot communicate with each other directly.

$ ssh app@hashicups-db "bash -c '/start_database.sh local'"; \
ssh app@hashicups-api "bash -c '/start_api.sh local'"; \
ssh app@hashicups-frontend "bash -c '/start_frontend.sh local'"; \
ssh app@hashicups-nginx "bash -c '/start_nginx.sh local'"

This tutorial still configures the NGINX service to listen on the VM's IP so you can still access it remotely. For production, we recommend using an ingress gateway to manage access to the service.

Click on the Consul UI tab.

Confirm that HashiCups still works despite being configured to localhost. The Envoy sidecar proxies route each service's local traffic to the relevant upstream.

HashiCups UI

Next steps

In this tutorial, you learned how to migrate your Consul services to Consul service mesh by updating each service's definitions, starting Envoy sidecar proxies for each service, and updating the services' dependencies to bind to localhost.

In the process, you integrated Zero Trust Security in your network and learned how to define explicitly define service-to-service permissions using intentions.

In the next tutorial, you will learn how to monitor the services in your Consul service mesh using the Grafana suite.

For more information about the topics covered in this tutorial, refer to the following resources:

Previous
Next