Observe Consul Service Mesh Traffic with Prometheus
In this tutorial, you will configure Consul service mesh to install a pre-configured instance of Prometheus suitable for development and evaluation purposes.
Consul service mesh on Kubernetes provides a deep integration with Prometheus, and even includes a built-in starter experience for installing Prometheus to demo or for dev environments. The following service mesh resource metrics can be configured for immediate collection using the official Consul Helm chart.
- Envoy proxies
- Consul server and client agents
- Registered services
- Consul gateways
To try the features in this tutorial, you will need a Kubernetes cluster with Consul service mesh installed. In case you do not have one, and are interested in testing the use cases in this tutorial, you can follow the Consul Service Mesh on Minikube or the Consul Service Mesh on Kind tutorial to deploy a local Kubernetes dev cluster.
To install Consul service mesh on Kubernetes using the official Helm chart or the Consul K8S ClI, you must ensure you have enabled Consul service mesh features by setting the
connectInject.enabled flag to
true in the values file you supply to the ConsulHelm chart installation. When you enable the Consul service mesh, Consul injects a
envoy-sidecar container into in each service deployment you add to the mesh.
consul-sidecar container is capable of coordinating Layer 7 (L7) metrics collection (HTTP status codes, request latency, transaction volume, etc.) for your service mesh applications. The metrics integration also supports collecting Consul agent metrics and gateway metrics. With a small amount of configuration, this data can be exported to Prometheus and visualized in the Consul UI. We've provided a sample configuration in a repository for you.
If you followed the instructions in the Deploy Consul Service Mesh on Kubernetes tutorial, you will need to upgrade your mesh with the configuration discussed in this section.
Clone the GitHub repository that contains the files you'll use with this tutorial.
Next, clone the GitHub repository that contains the files you'll use with this tutorial.
$ git clone https://github.com/hashicorp/learn-consul-kubernetes.git
Change directories into sample code for this tutorial located in the repository you just cloned.
$ cd learn-consul-kubernetes/service-mesh/observability
Checkout the tagged version verified for this tutorial.
git checkout tags/v0.0.8
You can deploy a complete Consul datacenter using the official Consul Helm chart or the Consul K8S CLI. Feel free to review the Consul Kubernetes installation documentation to learn more about these installation options.
Review the custom values file
Below is the minimal configuration file we've provided in your cloned repository. This configuration will enable both the Consul service mesh and its observability features.
global: name: consul datacenter: dc1 metrics: # Configures the Helm chart’s components to expose Prometheus metrics for the Consul service mesh. enabled: true # Publishes consul agent metrics. Only applicable if global.metrics.enabled is true. enableAgentMetrics: true server: replicas: 1 ui: enabled: true connectInject: enabled: true default: true # Configures metrics for Consul service mesh services. All values are overridable # via annotations on a per-pod basis. metrics: # Configures the Consul sidecar to run a merged metrics server # to combine and serve both Envoy and Application metrics. defaultEnableMerging: true controller: enabled: true # prometheus configures a Prometheus installation. prometheus: # When true, the Helm chart will install a demo Prometheus server # instance alongside Consul. enabled: true
Notice that there are several metrics settings included throughout the configuration. In this tutorial, we'll discuss these settings and how they affect the observability configuration of your service mesh.
For a complete list of all possible settings, refer to the official documentation.
Install Consul in your cluster
You can now deploy a complete Consul datacenter in your Kubernetes cluster using the official Consul Helm chart or the Consul K8S CLI.
$ helm repo add hashicorp https://helm.releases.hashicorp.com "hashicorp" has been added to your repositories
$ helm install --values config.yaml consul hashicorp/consul --create-namespace --namespace consul --version "0.43.0"
You can review the official Helm chart values to learn more about the default settings.
Check that Consul is running in your Kubernetes cluster using
setup is complete when all pods have a status of
Running, as illustrated in the
$ kubectl get pods --namespace consul NAME READY STATUS RESTARTS AGE consul-client-dfrvz 1/1 Running 0 5m8s consul-connect-injector-b4c5d6748-2v2dl 1/1 Running 0 5m8s consul-connect-injector-b4c5d6748-mxhwv 1/1 Running 0 5m8s consul-controller-559465fd96-dp7ck 1/1 Running 0 5m8s consul-server-0 1/1 Running 0 5m8s consul-webhook-cert-manager-7cf6df6c4f-brgwt 1/1 Running 0 5m8s prometheus-server-5cbddcc44b-kk698 2/2 Running 0 5m8s
Consul's metrics features can either be enabled globally or per component. By
default, metrics features are disabled globally, and both the Consul UI and
service mesh are configured to inherit their enablement from the
global: name: consul datacenter: dc1 metrics: # Configures the Helm chart’s components to expose Prometheus metrics for the Consul service mesh. enabled: true # Publishes consul agent metrics. Only applicable if global.metrics.enabled is true. enableAgentMetrics: true
true as shown in this example YAML, you
are turning on metrics collection for Envoy proxies, registered services, and
Consul gateways globally. The configuration above also sets
true, which configures Consul to publish and collect metrics for all Consul
server and client agents.
Consul Prometheus does not currently support TLS configurations. This
means that if
global.tls.httpsOnly are set to
and Consul agent metrics are enabled, the installation will error out during Helm
template rendering. You can work around this by setting the Helm chart value
global.metrics.enableGatewayMetrics, setting which is not shown above, is
turned on by default when metrics are enabled globally. You can override this
setting manually if you do not wish to collect metrics about your Consul gateways.
Enable the Consul UI
To support visualizing metrics using the Consul UI, you must ensure that the Consul UI is enabled. We recommend you enable the Consul UI whether you plan to enable metrics or not.
ui: enabled: true
Service mesh configuration
Consul service mesh is configured using settings contained within the
stanza. Set the
connectInject.enabled setting to
true to enable Consul service
connectInject: enabled: true default: true # Configures metrics for Consul service mesh services. All values are overridable # via annotations on a per-pod basis. metrics: # Configures the Consul sidecar to run a merged metrics server # to combine and serve both Envoy and Application metrics. defaultEnableMerging: true
true, as shown in this sample YAML, sidecars
will be automatically injected with your service deployment by default. Also, note
that a metrics stanza has been added for you, and that the
entry has been set to
true. This ensures that the
consul-sidecar that gets
injected for each deployment is configured to merge Envoy metrics with your
registered service metrics into a single metrics endpoint that Prometheus will
To install a demo instance of Prometheus to your mesh using the Prometheus
Helm chart, set the
prometheus.enabled entry to
true. With this feature enabled,
you can test Consul with Prometheus natively integrated beginning on Day 0. However,
this feature is not intended for production environments. You should be aware that
with this configuration, upgrading to future versions of the Consul Helm chart may
result in Prometheus being upgraded as well.
# prometheus configures a Prometheus installation. prometheus: # When true, the Helm chart will install a demo Prometheus server # instance alongside Consul. enabled: true
View in Consul UI
To view metrics in the UI, you will need to have services in the mesh
deployed to your datacenter. We have provided a demo application in the repo that
you downloaded earlier that you can use with this tutorial to test Consul's metrics
features. You can use
kubectl to deploy the demo application.
$ kubectl apply -f hashicups service/frontend-service created configmap/nginx-configmap created deployment.apps/frontend created service/products-api-service created serviceaccount/products-api created configmap/db-configmap created deployment.apps/products-api created service/postgres created deployment.apps/postgres created service/public-api-service created deployment.apps/public-api created
Ensure that the demo application is healthy with
kubectl get pods
to verify that all pods have a status of
Running before proceeding to the next
$ kubectl get pods --namespace default NAME READY STATUS RESTARTS AGE frontend-6f548678bb-l7nkr 2/2 Running 0 2m9s postgres-7847fc4696-kzvsc 3/3 Running 0 2m9s product-api-59757677b8-n9r5q 2/2 Running 0 2m9s public-api-7bb69fcfd6-t76rb 2/2 Running 0 2m10s
Once all application pods are running, start the traffic simulation job.
$ kubectl apply -f traffic.yaml configmap/k6-configmap created deployment.apps/traffic created
You can now view service metrics in the Consul UI. In a new terminal session, issue the following command to expose the Consul UI to your development host.
$ kubectl port-forward consul-server-0 --namespace consul 8500:8500 Forwarding from 127.0.0.1:8500 -> 8500 Forwarding from [::1]:8500 -> 8500
localhost:8500 in a new browser tab, and navigate
to the "Services" screen. Select a service. You should observe that
a chart with some basic metrics is embedded in the service tile.
Hover over the four tile elements to get tooltip descriptions of the different metrics. You can also hover the timeline chart embedded in the tile, and review additional metrics for any point in time during the last 15 minutes.
In this tutorial, you were introduced to some observability features of Consul service mesh on Kubernetes, and were instructed on how to configure a demo installation of Prometheus suitable for development or evaluation. For more information on how to create a production grade observability pipeline for your Consul service mesh using Prometheus and Grafana in Kubernetes, refer to our Layer 7 Observability with Consul Service Mesh, Prometheus, Grafana, and Kubernetes. tutorial.
In addition to configuring metrics defaults globally via the Helm chart, you can
also provide, and even override, configuration on a per-pod basis using annotations.
Annotations are inspected by the
connect-inject handler, and if relevant, passed
on to the
consul-sidecar. Refer to the official documentation
for a complete discussion on how annotations will affect your metrics configuration.
In the tutorial, Manage Traffic with Consul Service Mesh, you will be introduced to the comprehensive traffic management features that Consul provides at the proxy, service, and WAN levels.