Observe Consul service mesh traffic

In the previous tutorial, you learned how to configure and deploy your services using Consul service mesh solution. By using Consul service mesh, you enabled zero trust security in your network by having all service-to-service traffic encrypted and managed by Envoy sidecar proxies.

In this tutorial, you will configure and use Consul to observe traffic within your service mesh. This enables you to quickly understand how services interact with each other and effectively debug your services' traffic.

Using the Grafana suite, you can have your monitoring configuration embedded in your service VM and have the metrics collection configuration automatically scale when you deploy new services without the need to manually add new scraping endpoints for each new service you want to deploy.

In this tutorial, you will:

• Create configuration files for Grafana agent tool
• Start Grafana agent on all Consul nodes
• Visualize metrics in Grafana
• Visualize available metrics in Consul UI

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you have a fully deployed Consul service mesh with Envoy sidecar proxies running alongside each service.

By the end of this tutorial, you will have enabled metrics collection on your Consul nodes and will be able to visualize metrics both in Consul UI and in a Grafana dashboard.

Prerequisites

$ git clone https://github.com/hashicorp-education/learn-consul-get-started-vms

$ git clone git@github.com:hashicorp-education/learn-consul-get-started-vms

$ cd learn-consul-get-started-vms/self-managed/infrastructure/aws

$ terraform init
Initializing the backend...
Initializing provider plugins...
...
Terraform has been successfully initialized!
...

$ terraform apply -var-file=./conf/03_service_mesh.tfvars
## ...
Do you want to perform these actions?
 Terraform will perform the actions described above.
 Only 'yes' will be accepted to approve.
 Enter a value: yes
## ...
Apply complete! Resources: 49 added, 0 changed, 0 destroyed.

Outputs:

connection_string = "ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`"
ip_bastion = "<redacted-output>"
remote_ops = "export BASTION_HOST=<redacted-output>"
retry_join = "provider=aws tag_key=ConsulJoinTag tag_value=auto-join-hcoc"
ui_consul = "https://<redacted-output>:8443"
ui_grafana = "http://<redacted-output>:3000/d/hashicups/hashicups"
ui_hashicups = "http://<redacted-output>"

Login into the bastion host VM

Terraform output provides a series of useful information, including bastion host IP address.

Login to the bastion host using ssh.

$ ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`

Verify Grafana agent binary

Check on each of the Consul nodes (Consul server, NGINX, Frontend, API, and Database) to verify Grafana agent is installed.

$ grafana-agent --version
agent, version v0.33.2 (branch: HEAD, revision: 93a15c09)
  build user:       root@808a765754ec
  build date:       2023-05-11T20:58:39Z
  go version:       go1.20.3
  platform:         linux/amd64
  tags:             netgo,builtinassets,promtail_journal_enabled

Configure Grafana Agent

You can configure Grafana Agent to collect several kinds of data from your VM. In this tutorial, you will use configurations for:

• metrics block, to define a collection of Prometheus-compatible scrape configs to be written in Mimir.
• logs block, to configure how the Agent collects logs and sends them to a Loki push API endpoint.

Generate configuration for Grafana Agent

This tutorial and interactive lab environment uses scripts in the tutorial's GitHub repository to generate the Consul configuration files for your client agents.

The Bastion Host includes the script in the ops/scenarios/99_supporting_scripts folder.

$ tree ./ops/scenarios/99_supporting_scripts/
./ops/scenarios/99_supporting_scripts/
|-- generate_consul_client_config.sh
|-- generate_consul_monitoring_config.sh
|-- generate_consul_server_config.sh
|-- generate_consul_server_tokens.sh
|-- generate_consul_service_config.sh
`-- generate_consul_service_intentions.sh

1 directory, 6 files

The script requires a few parameters to work correctly:

• an OUTPUT_FOLDER to place the files generated
• a PROMETHEUS_URI to push metrics to. In this scenario we configured Grafana Mimir for this task listening on the bastion host.
• a LOKI_URI to push logs to. In this scenario we configured Grafana Loki for this task listening on the bastion host.

$ export OUTPUT_FOLDER="./assets/scenario/conf/"; \
  export PROMETHEUS_URI=`getent hosts mimir | awk '{print $1}'`; \
  export LOKI_URI=`getent hosts loki | awk '{print $1}'`

With these values configured, generate the configuration.

$ bash ops/scenarios/99_supporting_scripts/generate_consul_monitoring_config.sh
 -- Parameter Check
 -- Generating Grafana Agent configuration

The script creates the Grafana Agent configuration for all agents.

$ tree ${OUTPUT_FOLDER}monitoring
./assets/scenario/conf/monitoring
├── grafana-agent-consul-server-0.yaml
├── grafana-agent-gateway-api.yaml
├── grafana-agent-hashicups-api.yaml
├── grafana-agent-hashicups-db.yaml
├── grafana-agent-hashicups-frontend.yaml
└── grafana-agent-hashicups-nginx.yaml

0 directories, 6 files

Copy configuration on client VMs

After the script generates the configuration files, you will copy these files in each client node.

$ scp -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-consul-server-0.yaml consul-server-0:grafana-agent.yaml; \
  scp -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-db.yaml hashicups-db:grafana-agent.yaml; \
  scp -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-api.yaml hashicups-api:grafana-agent.yaml; \
  scp -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-frontend.yaml hashicups-frontend:grafana-agent.yaml; \
  scp -i certs/id_rsa ${OUTPUT_FOLDER}monitoring/grafana-agent-hashicups-nginx.yaml hashicups-nginx:grafana-agent.yaml

Start Grafana Agent on VMs

Once you copied the configuration files on the different VMs, login on each Consul client VMs and start the Grafana Agent.

Start Grafana Agent for Consul server

Login to the Consul server VM.

$ ssh -i certs/id_rsa consul-server-0

Start the Grafana Agent.

$ grafana-agent -config.file grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Once the Grafana agent is started, exit the ssh session to return to the bastion host.

$ exit

Start Grafana Agent for Database

Login to the Database VM from the bastion host.

$ ssh -i certs/id_rsa hashicups-db

Start the Grafana Agent.

$ grafana-agent -config.file grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Once the Grafana agent is started, exit the ssh session to return to the bastion host.

$ exit

Start Grafana Agent for API

Login to the API VM from the bastion host.

$ ssh -i certs/id_rsa hashicups-api

Start the Grafana Agent.

$ grafana-agent -config.file grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Once the Grafana agent is started, exit the ssh session to return to the bastion host.

$ exit

Start Grafana Agent for Frontend

Login to the Frontend VM from the bastion host.

$ ssh -i certs/id_rsa hashicups-frontend

Start the Grafana Agent.

$ grafana-agent -config.file grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Once the Grafana agent is started, exit the ssh session to return to the bastion host.

$ exit

Start Grafana Agent for NGINX

Login to the NGINX VM from the bastion host.

$ ssh -i certs/id_rsa hashicups-nginx

Start the Grafana Agent.

$ grafana-agent -config.file grafana-agent.yaml > /tmp/grafana-agent.log 2>&1 &

Once the Grafana agent is started, exit the ssh session to return to the bastion host.

$ exit

Visualize metrics in Grafana

Once started all the Grafana Agents, the metrics will be available to Grafana. Open the Grafana UI to view the metrics.

The scenario includes some predefined Grafana dashboards.

Retrieve the Grafana UI address from Terraform.

$ terraform output -raw ui_grafana

Open the address in a browser.

The HashiCups dashboard shows an overview of the services deployed in the service mesh.

Topology visualization in Consul UI

Consul provides configuration entries that can be used to get a summary of traffic across services as well as some metrics to get a basic overview of service health.

In order to visualize metrics it is necessary to generate some traffic for your application. In the lab, select the tab HashiCups tab and perform a few transactions in the application.

After completing a few purchases select the Consul UI tab and login using the bootstrap token.

In your Consul dashboard, select Services then hashicups-api to find the topology page.

Click on the Open Dashboard link. The link under the service box will open the dashboard with the specific service selected to get more specific information directly.

Destroy the infrastructure

Once completed the steps for the tutorial, you should clean the infrastructure you created.

From the ./self-managed/infrastruture/aws folder of the repository, use terraform to destroy the infrastructure.

$ terraform destroy --auto-approve

Next steps

In this tutorial, you learned how to monitor your Consul service mesh and the services deployed in it using the Grafana suite.

You now have a distributed system to monitor your Consul service mesh. Using the Grafana Agent lets you embed the metrics export into your VMs' golden images and have your metrics automatically gathered when you add new services to the mesh without the need to edit your monitoring suite configuration.

For more information about the topics covered in this tutorial, refer to the following resources:

• Service Mesh Observability Overview
• Service Mesh Observability: UI Visualization

In the next tutorial you will learn how to link your Consul datacenter to the HCP management plane service to manage and monitor it directly from the HCP web interface.