Observe Consul service mesh traffic

21min
|
Consul
Interactive

Workflow

In the previous tutorial, you secured external access to your application's service mesh with Consul API gateway.

In this tutorial, you will configure and use Consul to observe traffic within your service mesh. Observing service traffic helps you understand how services interact with each other and effectively debug service traffic issues.

Consul supports the Grafana monitoring suite so that the metrics collection configuration can automatically scale when you deploy new services. That way, you do not need to manually add new scraping endpoints for every new service you deploy.

In this tutorial, you will:

Create configuration files for the Grafana agent tool.
Start the Grafana agent on all Consul nodes.
Review metrics visualizations in the Grafana Dashboard.
Review metrics visualizations in the Consul UI.

Note

Because this tutorial is part of the Get Started on VMs tutorial collection, the following workflow was designed for education and demonstration. It uses scripts to generate agent configurations and requires you to execute commands manually on different nodes. If you are setting up a production environment you should codify and automate the installation and deployment process according to your infrastructure and networking needs. Refer to the VM production patterns tutorial collection for Consul production deployment considerations and best practices.

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you have a fully deployed Consul service mesh with Envoy sidecar proxies running alongside each service.

Architectural diagram 03

By the end of this tutorial, you will have metrics collection running on your Consul nodes and will be able to visualize metrics in both the Consul UI and a Grafana dashboard.

Architectural diagram 05

Prerequisites

If you completed the previous tutorial, the infrastructure is already in place with all prerequisites.

For this tutorial, you will need:

An AWS account configured for use with Terraform
aws-cli >= 2.0
terraform >= 1.0
git >= 2.0

Clone GitHub repository

Clone the GitHub repository containing the configuration files and resources.

$ git clone https://github.com/hashicorp-education/learn-consul-get-started-vms

Change into the directory that contains the complete configuration files for this tutorial.

$ cd learn-consul-get-started-vms/self-managed/infrastructure/aws

Create infrastructure

With these Terraform configuration files, you are ready to deploy your infrastructure.

Issue the terraform init command from your working directory to download the necessary providers and initialize the backend.

$ terraform init
Initializing the backend...
Initializing provider plugins...
...
Terraform has been successfully initialized!
...

Then, deploy the resources. Enter yes to confirm the run.

$ terraform apply -var-file=../../ops/conf/GS_04_service_mesh_access.tfvars
## ...
Do you want to perform these actions?
 Terraform will perform the actions described above.
 Only 'yes' will be accepted to approve.
 Enter a value: yes
## ...
Apply complete! Resources: 49 added, 0 changed, 0 destroyed.

The Terraform deployment could take up to 15 minutes to complete. Feel free to explore the next sections of this tutorial while waiting for the environment to complete initialization.

After it completes deployment, Terraform returns a list of outputs you can use to interact with the newly created environment.

Outputs:

connection_string = "ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`"
ip_bastion = "34.222.160.35"
retry_join = <sensitive>
ui_consul = "https://44.244.31.105:8443"
ui_grafana = "http://34.222.160.35:3000/d/hashicups/hashicups"
ui_hashicups = "http://34.220.146.55"
ui_hashicups_API_GW = "https://54.201.39.96:8443"

The Terraform output provides useful information, including the bastion host IP address. The following is a brief description of the Terraform outputs:

The connection_string provides the command to SSH connect to the bastion host.
The ip_bastion provides the IP address of the bastion host. You will use the bastion host to run the rest of the commands in this tutorial.
The retry_join output lists Consul's retry_join configuration parameter. The tutorials will use this to generate Consul server and client configuration.
The ui_consul output lists the Consul UI address.
The ui_grafana output lists the Grafana UI address. You will use this address in this tutorial to monitor your Consul datacenter.
The ui_hashicups output lists the HashiCups UI address. You can use it to verify the HashiCups demo application is running properly.
The ui_hashicups_API_GW output lists the API gateway address that permits access to HashiCups UI address.

Terraform output provides a series of useful information, including bastion host IP address.

$ ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`

Verify Grafana agent binary

Check each of the client nodes to verify the Grafana agent is installed.

NGINX : hashicups-nginx-0
Frontend: hashicups-frontend-0
API: hashicups-api-0
Database: hashicups-db-0
API gateway: gateway-api-0

For example, to check Grafana agent installation on the Database VM, log into the Database VM from the bastion host.

$ ssh -i certs/id_rsa hashicups-db-0

Verify Grafana agent binary is installed.

$ grafana-agent --version 
agent, version v0.44.2 (branch: HEAD, revision: 132c9ac06)
  build user:       root@buildkitsandbox
  build date:       2025-01-29T16:28:02Z
  go version:       go1.22.11
  platform:         linux/amd64
  tags:             netgo,builtinassets,promtail_journal_enabled

Repeat the steps for all VMs you want to monitor.

Configure environment

The tutorial creates all the files in a destination folder. Export the path where you wish to create the configuration files for the scenario.

$ export OUTPUT_FOLDER=/home/admin/assets/scenario/conf/

Make sure the folder exists.

$ mkdir -p ${OUTPUT_FOLDER}

Source the env-scenario.env file to set the variables in the terminal session.

$ source assets/scenario/env-scenario.env

Configure the Consul CLI to interact with the Consul server.

$ export CONSUL_HTTP_ADDR="https://consul-server-0:8443" \
  export CONSUL_HTTP_SSL=true \
  export CONSUL_CACERT="${OUTPUT_FOLDER}secrets/consul-agent-ca.pem" \
  export CONSUL_TLS_SERVER_NAME="server.${CONSUL_DATACENTER}.${CONSUL_DOMAIN}" \
  export CONSUL_HTTP_TOKEN=`cat ${OUTPUT_FOLDER}secrets/acl-token-bootstrap.json | jq -r ".SecretID"`

The lab contains two environment files that will help you configure the terminal and generate the required configuration files.

$ ls -1 ~/assets/scenario/env*.env 
~/assets/scenario/env-consul.env
~/assets/scenario/env-scenario.env

Import the two files in your environment.

$ source ~/assets/scenario/env-scenario.env; \
source ~/assets/scenario/env-consul.env

Verify your Consul CLI can interact with your Consul server.

$ consul members 
Node                  Address        Status  Type    Build   Protocol  DC   Partition  Segment
consul-server-0       10.0.2.8:8301  alive   server  1.20.2  2         dc1  default    <all>
gateway-api-0         10.0.2.7:8301  alive   client  1.20.2  2         dc1  default    <default>
hashicups-api-0       10.0.2.6:8301  alive   client  1.20.2  2         dc1  default    <default>
hashicups-db-0        10.0.2.9:8301  alive   client  1.20.2  2         dc1  default    <default>
hashicups-frontend-0  10.0.2.5:8301  alive   client  1.20.2  2         dc1  default    <default>
hashicups-nginx-0     10.0.2.4:8301  alive   client  1.20.2  2         dc1  default    <default>

Configure Grafana agent

You can configure the Grafana agent to collect several types of data from your VM. In this tutorial, you will configure these data types:

metrics, to define a collection of Prometheus-compatible scrape configs that Grafana Mimir will store.
logs, to configure how the Agent collects logs and sends them to a Grafana Loki push API endpoint.

Generate configuration for Grafana Agent

This tutorial and interactive lab environment use scripts in this tutorial's GitHub repository to generate the Consul configuration files for your client agents.

The script requires a few parameters to work correctly:

OUTPUT_FOLDER to place the generated files.
PROMETHEUS_URI to push metrics to.
LOKI_URI to push logs to.

$ export OUTPUT_FOLDER=/home/admin/assets/scenario/conf/; \
export PROMETHEUS_URI=`getent hosts mimir | awk '{print $1}'`; \
export LOKI_URI=`getent hosts loki | awk '{print $1}'`

With these variables configured, generate the Grafana agent configuration.

$ ~/ops/scenarios/00_base_scenario_files/supporting_scripts/generate_consul_monitoring_config.sh 
[generate_consul_monitoring_config.sh] - - Generate configuration for Grafana agent
+ --------------------
| Parameter Check
+ --------------------
[WARN] Script is running with the following values:
[WARN] ----------
[WARN] PROMETHEUS_URI = 172.18.0.4
[WARN] LOKI_URI = 172.18.0.5
[WARN] ----------
[WARN] Generated configuration will be placed under:
[WARN] OUTPUT_FOLDER = ~/assets/scenario/conf/
[WARN] ----------
Parameter Check

+ --------------------
| Generate Grafana Agent configuration
+ --------------------
Generating configuration for consul-server-0
Generating configuration for hashicups-db-0
Generating configuration for hashicups-api-0
Generating configuration for hashicups-frontend-0
Generating configuration for hashicups-nginx-0
Generating configuration for gateway-api-0

Review Grafana agent configuration files for the agents that will run on your client VMs.

$ tree ${OUTPUT_FOLDER}monitoring 
~/assets/scenario/conf/monitoring
|-- grafana-agent-consul-server-0.yaml
|-- grafana-agent-gateway-api-0.yaml
|-- grafana-agent-hashicups-api-0.yaml
|-- grafana-agent-hashicups-db-0.yaml
|-- grafana-agent-hashicups-frontend-0.yaml
`-- grafana-agent-hashicups-nginx-0.yaml

1 directory, 6 files

Copy configuration to client VMs

Copy configuration to consul-server-0.

$ rsync -av \
    -e "ssh -i ~/certs/id_rsa" \
    ${OUTPUT_FOLDER}monitoring/grafana-agent-consul-server-0.yaml \
    consul-server-0:grafana-agent.yaml

The output is similar to the following:

sending incremental file list
grafana-agent-consul-server-0.yaml

sent 821 bytes  received 35 bytes  1,712.00 bytes/sec
total size is 687  speedup is 0.80

Copy configuration to hashicups-db-0.

$ rsync -av \
    -e "ssh -i ~/certs/id_rsa" \
    ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-db-0.yaml \
    hashicups-db-0:grafana-agent.yaml

The output is similar to the following:

sending incremental file list
grafana-agent-hashicups-db-0.yaml

sent 954 bytes  received 35 bytes  1,978.00 bytes/sec
total size is 821  speedup is 0.83

Copy configuration to hashicups-api-0.

$ rsync -av \
    -e "ssh -i ~/certs/id_rsa" \
    ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-api-0.yaml \
    hashicups-api-0:grafana-agent.yaml

The output is similar to the following:

sending incremental file list
grafana-agent-hashicups-api-0.yaml

sent 957 bytes  received 35 bytes  1,984.00 bytes/sec
total size is 823  speedup is 0.83

Copy configuration to hashicups-frontend-0.

$ rsync -av \
    -e "ssh -i ~/certs/id_rsa" \
    ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-frontend-0.yaml \
    hashicups-frontend-0:grafana-agent.yaml

The output is similar to the following:

sending incremental file list
grafana-agent-hashicups-frontend-0.yaml

sent 972 bytes  received 35 bytes  2,014.00 bytes/sec
total size is 833  speedup is 0.83

Copy configuration to hashicups-nginx-0.

$ rsync -av \
    -e "ssh -i ~/certs/id_rsa" \
    ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-nginx-0.yaml \
    hashicups-nginx-0:grafana-agent.yaml

The output is similar to the following:

sending incremental file list
grafana-agent-hashicups-nginx-0.yaml

sent 963 bytes  received 35 bytes  1,996.00 bytes/sec
total size is 827  speedup is 0.83

Copy configuration to gateway-api-0.

$ rsync -av \
    -e "ssh -i ~/certs/id_rsa" \
    ${OUTPUT_FOLDER}monitoring/grafana-agent-gateway-api-0.yaml \
    gateway-api-0:grafana-agent.yaml

The output is similar to the following:

sending incremental file list
grafana-agent-gateway-api-0.yaml

sent 951 bytes  received 35 bytes  1,972.00 bytes/sec
total size is 819  speedup is 0.83

Start Grafana agent on VMs

Log into each client VM and start the Grafana agent.

Start Grafana agent on the Consul server

Log into consul-server-0 from the bastion host.

$ ssh -i certs/id_rsa consul-server-0
##..
admin@consul-server-0:~

Start the Grafana agent.

$ grafana-agent -config.file ~/grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Exit the SSH session to return to the bastion host.

$ exit
logout
Connection to consul-server-0 closed.
admin@bastion:~$

Start Grafana Agent for Database

Log into hashicups-db-0 from the bastion host.

$ ssh -i certs/id_rsa hashicups-db-0
##..
admin@hashicups-db-0:~

Start the Grafana agent.

$ grafana-agent -config.file ~/grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Exit the SSH session to return to the bastion host.

$ exit
logout
Connection to hashicups-db-0 closed.
admin@bastion:~$

Start Grafana agent for API

Log into hashicups-api-0 from the bastion host.

$ ssh -i certs/id_rsa hashicups-api-0
##..
admin@hashicups-api-0:~

Start the Grafana agent.

$ grafana-agent -config.file ~/grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Exit the SSH session to return to the bastion host.

$ exit
logout
Connection to hashicups-api-0 closed.
admin@bastion:~$

Start Grafana agent for Frontend

Log into hashicups-frontend-0 from the bastion host.

$ ssh -i certs/id_rsa hashicups-frontend-0
##..
admin@hashicups-frontend-0:~

Start the Grafana agent.

$ grafana-agent -config.file ~/grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Exit the SSH session to return to the bastion host.

$ exit
logout
Connection to hashicups-frontend-0 closed.
admin@bastion:~$

Start Grafana agent for NGINX

Log into hashicups-nginx-0 from the bastion host.

$ ssh -i certs/id_rsa hashicups-nginx-0
##..
admin@hashicups-nginx-0:~

Start the Grafana agent.

$ grafana-agent -config.file ~/grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Exit the SSH session to return to the bastion host.

$ exit
logout
Connection to hashicups-nginx-0 closed.
admin@bastion:~$

Start Grafana agent for API gateway

Log into gateway-api-0 from the bastion host.

$ ssh -i certs/id_rsa gateway-api-0
##..
admin@gateway-api-0:~

Start the Grafana agent.

$ grafana-agent -config.file ~/grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Exit the SSH session to return to the bastion host.

$ exit
logout
Connection to gateway-api-0 closed.
admin@bastion:~$

Visualization security

The metrics proxy endpoint is internal and intended only for UI use. However by enabling it anyone with network access to the agent's API port may use it to access metrics from the backend. With ACLs enabled, the proxy endpoint requires a valid token with read access to all nodes and services.

For this tutorial you will use the bootstrap token to visualize the metrics.

Retrieve it from the environment variable before ending the session.

$ echo $CONSUL_HTTP_TOKEN

Note

The environment variables defined for the session will not be available anymore if you exit the ssh session to return to your local machine. If you need to setup your environment again after exiting the session follow the steps listed in Configure environment.

Visualize metrics in Grafana

Grafana agent instances on each client VM are now sending metrics to Grafana. Open the Grafana UI to view the metrics.

This scenario includes Grafana dashboards.

Retrieve the Grafana UI address from Terraform.

$ terraform output -raw ui_grafana

Open the address in a browser.

The HashiCups dashboard shows an overview of the services deployed in the service mesh.

HashiCups dashboard in Grafana UI

Topology visualization in Consul UI

Consul provides configuration entries that you may use to get a summary of traffic across services, as well as some metrics to get a basic overview of service health.

In order to visualize metrics it is necessary to generate some traffic for your application.

Open the HashiCups UI address. You can obtain it with the terraform output command.

$ terraform output ui_hashicups_API_GW

Complete a few purchases in the application. Then open the Consul UI and log in using the bootstrap token.

$ terraform output ui_consul

You can retrieve the bootstrap token from the environment variable.

$ echo $CONSUL_HTTP_TOKEN

In your Consul dashboard, click Services. Then select hashicups-api to find the service topology page.

Consul UI with topology visualization for hashicups-api service

Click the Open Dashboard link. The link under the service box opens the dashboard with the service selected for more specific information.

HashiCups dashboard in Grafana UI with hashicups-api service selected

Next steps

Congratulations! You completed the final tutorial in this collection.

Destroy the infrastructure

Make sure you destroy the infrastructure you created for this tutorial.

From the ./self-managed/infrastruture/aws folder of the repository, use terraform to destroy the infrastructure.

$ terraform destroy --auto-approve

In this tutorial, you learned how to use Grafana to monitor your Consul service mesh and the services deployed in it.

You now have a distributed system to monitor your Consul service mesh. You can include the Grafana agent and metrics configurations in your VM image. Then you can have your metrics automatically gathered when you add new services to the mesh without the need to edit your monitoring suite configuration.

For more information about the topics covered in this tutorial, refer to the following resources:

Access services in your service mesh

Next Collection

Connect services

Tutorial scenario

Prerequisites

Login into the bastion host VM

Verify Grafana agent binary

Configure environment

Configure Grafana agent

Generate configuration for Grafana Agent

Copy configuration to client VMs

Start Grafana agent on VMs

Start Grafana agent on the Consul server

Start Grafana Agent for Database

Start Grafana agent for API

Start Grafana agent for Frontend

Start Grafana agent for NGINX

Start Grafana agent for API gateway

Visualization security

Visualize metrics in Grafana

Topology visualization in Consul UI

Next steps

Destroy the infrastructure