Observe Consul service mesh traffic
In the previous tutorial, you learned how to configure and deploy your services using Consul service mesh solution.
By using Consul service mesh, you enabled zero trust security in your network by having all service-to-service traffic encrypted and managed by Envoy sidecar proxies.
In this tutorial, you will learn how to monitor your service fleet using the Grafana suite.
Specifically, you will learn how collect and export the following metrics:
- Consul agent telemetry metrics
- Envoy sidecar proxy telemetry metrics.
- Logs for Consul agents, service applications, sidecar proxies and Grafana Agent.
This tutorial uses the Grafana Agent tool to collect metrics. The Grafana Agent tool is a unified tools that is able to collect different metrics using a single configuration and is compatible with all the products of the Grafana Stack.
Using the Grafana Agent tool you can have your monitoring configuration embedded in your service VM and have the metrics collection configuration automatically scale when you deploy new services without the need to manually add new scraping endpoints for each new service you want to deploy.
Tutorial scenario
This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.
At the beginning of the tutorial, you have a fully deployed Consul service mesh with Envoy sidecar proxies running alongside each service.
By the end of this tutorial, you will have enabled metrics collection on your Consul nodes and will be able to visualize metrics both in Consul UI and in a Grafana dashboard.
Prerequisites
This tutorial uses an interactive lab to guide you through how to enable metrics collection for your Consul service mesh. This lab environment includes all required binaries and sample configurations. We highly recommend using the interactive lab environment to complete this tutorial.
Launch Terminal
This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.
Configure Grafana Agent
You can configure Grafana Agent to collect several kinds of data from your VM. In this tutorial, you will use configurations for:
metrics
block, to define a collection of Prometheus-compatible scrape configs to be written in Mimir.logs
block, to configure how the Agent collects logs and sends them to a Loki push API endpoint.
Generate configuration for Grafana Agent
This tutorial and interactive lab environment uses scripts in the tutorial's GitHub repository to generate the Consul configuration files for your client agents.
The interactive lab environment includes these scripts and the pre-configured Consul server. In Operator, list the files in your current directory.
Use the provided script to generate the configuration for Grafana Agent.
The script creates the Grafana Agent configuration for all agents.
Copy configuration on client VMs
After the script generates the configuration files, you will copy these files in each client node.
Tip
In the interactive lab environment, the HashiCups application nodes
have a running SSH server. As a result, you can use ssh
and scp
commands to
perform the following operations. If the nodes in your personal environment does
not have an SSH server, you may need to use a different approach to create the
configuration directories and copy the files.
Start Grafana Agent on VMs
Once you copied the configuration files on the different VMs, login on each Consul client VMs and start the Grafana Agent.
Note
For production environments, consider using systemd
to start the
Grafana Agent as a daemon service.
Start Grafana Agent for Consul
Log into the virtual machine that hosts the Consul server and start the Grafana Agent.
Tip
For the interactive lab environment, select the tab that corresponds with the service — in this case, Consul.
Start Grafana Agent for Database
Log into the virtual machine that hosts the database and start the Grafana Agent.
Start Grafana Agent for API
Log into the virtual machine that hosts the API and start the Grafana Agent.
Start Grafana Agent for Frontend
Log into the virtual machine that hosts the frontend and start the Grafana Agent.
Start Grafana Agent for NGINX
Log into the virtual machine that hosts the NGINX and start the Grafana Agent.
Visualize metrics in Grafana
Once started all the Grafana Agents, the metrics will be available to Grafana. Open the Grafana UI to view the metrics.
The interactive lab environment includes some predefined Grafana dashboards.
Navigate to the Grafana UI tab to find the Grafana dashboard. Click on the Browse icon (four boxes) in the left sidebar to view all available dashboards.
The HashiCups dashboard shows an overview of the services deployed in the service mesh.
Topology visualization in Consul UI
Consul provides configuration entries that can be used to get a summary of traffic across services as well as some metrics to get a basic overview of service health.
In order to visualize metrics it is necessary to generate some traffic for your application. In the lab, select the tab HashiCups tab and perform a few transactions in the application.
After completing a few purchases select the Consul UI tab and login using the bootstrap token.
In your Consul dashboard, select Services then hashicups-api to find the topology page.
Note
to visualize the metrics in Consul UI, login into Consul using a valid ACL token.
By clicking on the Open Dashboard link you will get redirected to the Grafana
dashboard you configured in your Consul ui_config
section of the configuration.
The link under the service box will open the dashboard with the specific service selected, to get more specific information directly.
Next Steps
In this tutorial you learned how to monitor your Consul service mesh and the services deployed in it using the Grafana suite.
You now have a distributed system to monitor your Consul service mesh. Using the Grafana Agent permits you to embed the metrics export into your VMs' golden images and have your metrics automatically gathered when you add new services to the mesh without the need to edit your monitoring suite configuration.
For more information about the topics covered in this tutorial, refer to the following resources: