Monitor your Consul service mesh

9min
Consul
Interactive

In the previous tutorial, you learned how to configure and deploy your services using Consul service mesh solution.

By using Consul service mesh, you enabled zero trust security in your network by having all service-to-service traffic encrypted and managed by Envoy sidecar proxies.

In this tutorial, you will learn how to monitor your service fleet using the Grafana suite.

Specifically, you will learn how collect and export the following metrics:

Consul agent telemetry metrics
Envoy sidecar proxy telemetry metrics.
Logs for Consul agents, service applications, sidecar proxies and Grafana Agent.

This tutorial uses the Grafana Agent tool to collect metrics. The Grafana Agent tool is a unified tools that is able to collect different metrics using a single configuration and is compatible with all the products of the Grafana Stack.

Using the Grafana Agent tool you can have your monitoring configuration embedded in your service VM and have the metrics collection configuration automatically scale when you deploy new services without the need to manually add new scraping endpoints for each new service you want to deploy.

Tutorial scenario

This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.

At the beginning of the tutorial, you have a fully deployed Consul service mesh with Envoy sidecar proxies running alongside each service.

Architectural diagram 03

By the end of this tutorial, you will have enabled metrics collection on your Consul nodes and will be able to visualize metrics both in Consul UI and in a Grafana dashboard.

Prerequisites

This tutorial uses an interactive lab to guide you through how to enable metrics collection for your Consul service mesh. This lab environment includes all required binaries and sample configurations. We highly recommend using the interactive lab environment to complete this tutorial.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

You will need the following to complete this tutorial:

A Consul service mesh with serviced deployed
Telemetry and topology visualization configured at startup for Consul agents.
A running Grafana Loki instance, to collect logs
A running Grafana Mimir instance, to collect metrics
A running Grafana instance to show dashboards using the Loki and Mimir instances as datasources.
Grafana Agent binary installed on all Consul agent nodes. You need to install the agent binary on each VM you want to monitor.

Configure Consul agents

For telemetry metrics to be exposed you need to configure all Consul agents you want to monitor to expose metrics over the /v1/agent/metrics API endpoint.

Add the following to your Consul configuration on all agents you want to monitor.

telemetry {
  prometheus_retention_time = "60s"
  disable_hostname = true
}

To configure Consul's UI to fetch metrics there are two required configuration settings. These need to be set on each Consul Agent that is responsible for serving the UI. If there are multiple clients with the UI enabled in a datacenter for redundancy these configurations must be added to all of them.

ui_config {
  enabled = true
  dashboard_url_templates {
    service = "http://grafana:3000/d/hashicups/hashicups?orgId=1&var-service={{Service.Name}}"
  }
  metrics_provider = "prometheus"
  metrics_proxy {
    base_url = "http://mimir:9009/prometheus"
    path_allowlist = ["/api/v1/query_range", "/api/v1/query", "/prometheus/api/v1/query_range", "/prometheus/api/v1/query"]
  }
}

Info: the scenario used in the configuration has the Grafana instance accessible at grafana hostname and the Mimir instance accessible at mimir. Change the configuration to reflect your scenario hostnames or IP addresses.

Pre-existing environments: the configuration in this section should be applied to your Consul agents at startup. In case you are implementing monitoring on an existing datacenter you will have to restart your Consul agents after changing the configuration.

Configure Grafana Agent

You can configure Grafana Agent to collect several kinds of data from your VM. In this tutorial, you will use configurations for:

metrics block, to define a collection of Prometheus-compatible scrape configs to be written in Mimir.
logs block, to configure how the Agent collects logs and sends them to a Loki push API endpoint.

Generate configuration for Grafana Agent

This tutorial and interactive lab environment uses scripts in the tutorial's GitHub repository to generate the Consul configuration files for your client agents.

The interactive lab environment includes these scripts and the pre-configured Consul server. In Operator, list the files in your current directory.

$ tree
.
|-- assets
|   |-- acl-policy-dns.hcl
|   |-- acl-policy-server-node.hcl
|   |-- acl-token-bootstrap.json
|   |-- acl-token-dns.json
|   |-- agent-client-hashicups-api-acl-tokens.hcl
|   |-- agent-client-hashicups-db-acl-tokens.hcl
|   |-- agent-client-hashicups-frontend-acl-tokens.hcl
|   |-- agent-client-hashicups-nginx-acl-tokens.hcl
|   |-- agent-client-secure.hcl
|   |-- agent-gossip-encryption.hcl
|   |-- agent-linux-amd64
|   |-- agent-server-acl.hcl
|   |-- agent-server-secure.hcl
|   |-- agent-server-specific.hcl
|   |-- agent-server-telemetry.hcl
|   |-- agent-server-tls.hcl
|   |-- consul-agent-ca-key.pem
|   |-- consul-agent-ca.pem
|   |-- dc1-server-consul-0-key.pem
|   |-- dc1-server-consul-0.pem
|   |-- global
|   |   |-- intention-api.hcl
|   |   |-- intention-api.json
|   |   |-- intention-db.hcl
|   |   |-- intention-db.json
|   |   |-- intention-frontend.hcl
|   |   `-- intention-frontend.json
|   |-- server-acl-token.json
|   |-- svc-hashicups-api.hcl
|   |-- svc-hashicups-db.hcl
|   |-- svc-hashicups-frontend.hcl
|   `-- svc-hashicups-nginx.hcl
|-- consul_conf.env
|-- filebrowser.db
|-- generate_consul_monitoring_config.sh
|-- logs
|   |-- filebrowser.log
|   `-- provision.log
`-- start_scenario.sh

3 directories, 37 files

Clone the example repository.

$ git clone https://github.com/hashicorp-education/learn-consul-get-started-vms.git

Change into the directory with the newly cloned repository.

$ cd learn-consul-get-started-vms

Use the provided script to generate the configuration for Grafana Agent.

$ ./generate_consul_monitoring_config.sh
Generating Grafana Agent configuration
/grafana_agent_configs ~
~

The script creates the Grafana Agent configuration for all agents.

$ tree grafana_agent_configs/
grafana_agent_configs/
|-- grafana-agent-consul-server.yaml
|-- grafana-agent-hashicups-api.yaml
|-- grafana-agent-hashicups-db.yaml
|-- grafana-agent-hashicups-frontend.yaml
`-- grafana-agent-hashicups-nginx.yaml

0 directories, 5 files

Copy configuration on client VMs

After the script generates the configuration files, you will copy these files in each client node.

Tip: In the interactive lab environment, the HashiCups application nodes have a running SSH server. As a result, you can use ssh and scp commands to perform the following operations. If the nodes in your personal environment does not have an SSH server, you may need to use a different approach to create the configuration directories and copy the files.

$ scp grafana_agent_configs/grafana-agent-consul-server.yaml consul:/home/app/grafana-agent.yaml; \
scp grafana_agent_configs/grafana-agent-hashicups-db.yaml hashicups-db:/home/app/grafana-agent.yaml; \
scp grafana_agent_configs/grafana-agent-hashicups-api.yaml hashicups-api:/home/app/grafana-agent.yaml; \
scp grafana_agent_configs/grafana-agent-hashicups-frontend.yaml hashicups-frontend:/home/app/grafana-agent.yaml; \
scp grafana_agent_configs/grafana-agent-hashicups-nginx.yaml hashicups-nginx:/home/app/grafana-agent.yaml

Start Grafana Agent on VMs

Once you copied the configuration files on the different VMs, login on each Consul client VMs and start the Grafana Agent.

Note: For production environments, consider using systemd to start the Grafana Agent as a daemon service.

Start Grafana Agent for Consul

Log into the virtual machine that hosts the Consul server and start the Grafana Agent.

Tip: For the interactive lab environment, select the tab that corresponds with the service — in this case, Consul.

$ /home/app/grafana-agent -config.file /home/app/grafana-agent.yaml

Start Grafana Agent for Database

Log into the virtual machine that hosts the database and start the Grafana Agent.

$ /home/app/grafana-agent -config.file /home/app/grafana-agent.yaml

Start Grafana Agent for API

Log into the virtual machine that hosts the API and start the Grafana Agent.

$ /home/app/grafana-agent -config.file /home/app/grafana-agent.yaml

Start Grafana Agent for Frontend

Log into the virtual machine that hosts the frontend and start the Grafana Agent.

$ /home/app/grafana-agent -config.file /home/app/grafana-agent.yaml

Start Grafana Agent for NGINX

Log into the virtual machine that hosts the NGINX and start the Grafana Agent.

$ /home/app/grafana-agent -config.file /home/app/grafana-agent.yaml

Visualize metrics in Grafana

Once started all the Grafana Agents, the metrics will be available to Grafana. Open the Grafana UI to view the metrics.

The interactive lab environment includes some predefined Grafana dashboards.

Navigate to the Grafana UI tab to find the Grafana dashboard. Click on the Browse icon (four boxes) in the left sidebar to view all available dashboards.

The repository includes some example Grafana dashboards you can adapt to your environment.

$ cd config/grafana/dashboards

$ tree
.
├── consul-server-monitor.json
├── hashicups.json
└── logging.json

0 directories, 3 files

Check Grafana documentation to learn how to import a dashboard from a JSON schema.

The HashiCups dashboard shows an overview of the services deployed in the service mesh.

HashiCups dashboard in Grafana UI

Topology visualization in Consul UI

Consul provides configuration entries that can be used to get a summary of traffic across services as well as some metrics to get a basic overview of service health.

In order to visualize metrics it is necessary to generate some traffic for your application. In the lab, select the tab HashiCups tab and perform a few transactions in the application.

After completing a few purchases select the Consul UI tab and login using the bootstrap token.

In your Consul dashboard, select Services then hashicups-api to find the topology page.

Consul UI with topology visualization for hashicups-api service

Note: to visualize the metrics in Consul UI, login into Consul using a valid ACL token.

By clicking on the Open Dashboard link you will get redirected to the Grafana dashboard you configured in your Consul ui_config section of the configuration.

The link under the service box will open the dashboard with the specific service selected, to get more specific information directly.

HashiCups dashboard in Grafana UI with hashicups-api service selected

Next Steps

In this tutorial you learned how to monitor your Consul service mesh and the services deployed in it using the Grafana suite.

You now have a distributed system to monitor your Consul service mesh. Using the Grafana Agent permits you to embed the metrics export into your VMs' golden images and have your metrics automatically gathered when you add new services to the mesh without the need to edit your monitoring suite configuration.

For more information about the topics covered in this tutorial, refer to the following resources:

Next Collection