Observe Consul Service Mesh Traffic with Prometheus
In this tutorial, you will configure Consul service mesh to install a pre-configured instance of Prometheus suitable for development and evaluation purposes.
Consul service mesh on Kubernetes provides a deep integration with Prometheus, and even includes a built-in starter experience for installing Prometheus to demo or for dev environments. The following service mesh resource metrics can be configured for immediate collection using the official Consul Helm chart.
- Envoy proxies
- Consul server and client agents
- Registered services
- Consul gateways
Prerequisites
To try the features in this tutorial, you will need a Kubernetes cluster with Consul service mesh installed. In case you do not have one, and are interested in testing the use cases in this tutorial, you can follow the Consul Service Mesh on Minikube or the Consul Service Mesh on Kind tutorial to deploy a local Kubernetes dev cluster.
Configure observability
To install Consul service mesh on Kubernetes using the official Helm chart or the Consul K8S ClI, you must ensure you have enabled Consul service mesh features by setting the connectInject.enabled
flag to true
in the values file you supply to the ConsulHelm chart installation. When you enable the Consul service mesh, Consul injects a consul-sidecar
and envoy-sidecar
container into in each service deployment you add to the mesh.
The consul-sidecar
container is capable of coordinating Layer 7 (L7) metrics collection (HTTP status codes, request latency, transaction volume, etc.) for your service mesh applications. The metrics integration also supports collecting Consul agent metrics and gateway metrics. With a small amount of configuration, this data can be exported to Prometheus and visualized in the Consul UI. We've provided a sample configuration in a repository for you.
Note
If you followed the instructions in the Deploy Consul Service Mesh on Kubernetes tutorial, you will need to upgrade your mesh with the configuration discussed in this section.
Clone the GitHub repository that contains the files you'll use with this tutorial.
Next, clone the GitHub repository that contains the files you'll use with this tutorial.
Change directories into sample code for this tutorial located in the repository you just cloned.
Checkout the tagged version verified for this tutorial.
Deploy Consul
You can deploy a complete Consul datacenter using the official Consul Helm chart or the Consul K8S CLI. Feel free to review the Consul Kubernetes installation documentation to learn more about these installation options.
Review the custom values file
Below is the minimal configuration file we've provided in your cloned repository. This configuration will enable both the Consul service mesh and its observability features.
Notice that there are several metrics settings included throughout the configuration. In this tutorial, we'll discuss these settings and how they affect the observability configuration of your service mesh.
For a complete list of all possible settings, refer to the official documentation.
Install Consul in your cluster
You can now deploy a complete Consul datacenter in your Kubernetes cluster using the official Consul Helm chart or the Consul K8S CLI.
Check that Consul is running in your Kubernetes cluster using kubectl
. Consul
setup is complete when all pods have a status of Running
, as illustrated in the
following output.
Global settings
Consul's metrics features can either be enabled globally or per component. By
default, metrics features are disabled globally, and both the Consul UI and
service mesh are configured to inherit their enablement from the global
setting.
By setting global.metrics.enabled
to true
as shown in this example YAML, you
are turning on metrics collection for Envoy proxies, registered services, and
Consul gateways globally. The configuration above also sets global.metrics.enableAgentMetrics
to true
, which configures Consul to publish and collect metrics for all Consul
server and client agents.
Note
Consul Prometheus does not currently support TLS configurations. This
means that if global.tls.enabled
and global.tls.httpsOnly
are set to true
,
and Consul agent metrics are enabled, the installation will error out during Helm
template rendering. You can work around this by setting the Helm chart value
enableAgentMetrics
to false
.
The global.metrics.enableGatewayMetrics
, setting which is not shown above, is
turned on by default when metrics are enabled globally. You can override this
setting manually if you do not wish to collect metrics about your Consul gateways.
Enable the Consul UI
To support visualizing metrics using the Consul UI, you must ensure that the Consul UI is enabled. We recommend you enable the Consul UI whether you plan to enable metrics or not.
Service mesh configuration
Consul service mesh is configured using settings contained within the connectInject
stanza. Set the connectInject.enabled
setting to true
to enable Consul service
mesh.
By setting connectInject.default
to true
, as shown in this sample YAML, sidecars
will be automatically injected with your service deployment by default. Also, note
that a metrics stanza has been added for you, and that the defaultEnableMerging
entry has been set to true
. This ensures that the consul-sidecar
that gets
injected for each deployment is configured to merge Envoy metrics with your
registered service metrics into a single metrics endpoint that Prometheus will
then scrape.
Enable Prometheus
To install a demo instance of Prometheus to your mesh using the Prometheus
Helm chart, set the prometheus.enabled
entry to true
. With this feature enabled,
you can test Consul with Prometheus natively integrated beginning on Day 0. However,
this feature is not intended for production environments. You should be aware that
with this configuration, upgrading to future versions of the Consul Helm chart may
result in Prometheus being upgraded as well.
View in Consul UI
To view metrics in the UI, you will need to have services in the mesh
deployed to your datacenter. We have provided a demo application in the repo that
you downloaded earlier that you can use with this tutorial to test Consul's metrics
features. You can use kubectl
to deploy the demo application.
Ensure that the demo application is healthy with kubectl get pods
to verify that all pods have a status of Running
before proceeding to the next
step.
Once all application pods are running, start the traffic simulation job.
You can now view service metrics in the Consul UI. In a new terminal session, issue the following command to expose the Consul UI to your development host.
Open localhost:8500
in a new browser tab, and navigate
to the "Services" screen. Select a service. You should observe that
a chart with some basic metrics is embedded in the service tile.
Hover over the four tile elements to get tooltip descriptions of the different metrics. You can also hover the timeline chart embedded in the tile, and review additional metrics for any point in time during the last 15 minutes.
Next steps
In this tutorial, you were introduced to some observability features of Consul service mesh on Kubernetes, and were instructed on how to configure a demo installation of Prometheus suitable for development or evaluation. For more information on how to create a production grade observability pipeline for your Consul service mesh using Prometheus and Grafana in Kubernetes, refer to our Layer 7 Observability with Consul Service Mesh, Prometheus, Grafana, and Kubernetes. tutorial.
In addition to configuring metrics defaults globally via the Helm chart, you can
also provide, and even override, configuration on a per-pod basis using annotations.
Annotations are inspected by the connect-inject
handler, and if relevant, passed
on to the consul-sidecar
. Refer to the official documentation
for a complete discussion on how annotations will affect your metrics configuration.
In the tutorial, Manage Traffic with Consul Service Mesh, you will be introduced to the comprehensive traffic management features that Consul provides at the proxy, service, and WAN levels.