Observe Consul Service Mesh Traffic with Prometheus

In this tutorial, you will configure Consul service mesh to install a pre-configured instance of Prometheus suitable for development and evaluation purposes.

Consul service mesh on Kubernetes provides a deep integration with Prometheus, and even includes a built-in starter experience for installing Prometheus to demo or for dev environments. The following service mesh resource metrics can be configured for immediate collection using the official Consul Helm chart.

• Envoy proxies
• Consul server and client agents
• Registered services
• Consul gateways

Prerequisites

To try the features in this tutorial, you will need a Kubernetes cluster with Consul service mesh installed. In case you do not have one, and are interested in testing the use cases in this tutorial, you can follow the Consul Service Mesh on Minikube or the Consul Service Mesh on Kind tutorial to deploy a local Kubernetes dev cluster.

Configure observability

To install Consul service mesh on Kubernetes using the official Helm chart or the Consul K8S ClI, you must ensure you have enabled Consul service mesh features by setting the connectInject.enabled flag to true in the values file you supply to the ConsulHelm chart installation. When you enable the Consul service mesh, Consul injects a consul-sidecar and envoy-sidecar container into in each service deployment you add to the mesh.

The consul-sidecar container is capable of coordinating Layer 7 (L7) metrics collection (HTTP status codes, request latency, transaction volume, etc.) for your service mesh applications. The metrics integration also supports collecting Consul agent metrics and gateway metrics. With a small amount of configuration, this data can be exported to Prometheus and visualized in the Consul UI. We've provided a sample configuration in a repository for you.

Clone the GitHub repository that contains the files you'll use with this tutorial.

Next, clone the GitHub repository that contains the files you'll use with this tutorial.

$ git clone https://github.com/hashicorp/learn-consul-kubernetes.git

$ git clone git@github.com:hashicorp/learn-consul-kubernetes.git

Change directories into sample code for this tutorial located in the repository you just cloned.

$ cd learn-consul-kubernetes/service-mesh/observability

Checkout the tagged version verified for this tutorial.

git checkout tags/v0.0.8

Deploy Consul

You can deploy a complete Consul datacenter using the official Consul Helm chart or the Consul K8S CLI. Feel free to review the Consul Kubernetes installation documentation to learn more about these installation options.

Review the custom values file

Below is the minimal configuration file we've provided in your cloned repository. This configuration will enable both the Consul service mesh and its observability features.

global:
  name: consul
  datacenter: dc1
  metrics:
    # Configures the Helm chart’s components to expose Prometheus metrics for the Consul service mesh.
    enabled: true
    # Publishes consul agent metrics. Only applicable if  global.metrics.enabled is true.
    enableAgentMetrics: true
server:
  replicas: 1
ui:
  enabled: true
connectInject:
  enabled: true
  default: true
  # Configures metrics for Consul service mesh services. All values are overridable
  # via annotations on a per-pod basis.
  metrics:
    # Configures the Consul sidecar to run a merged metrics server
    # to combine and serve both Envoy and Application metrics.
    defaultEnableMerging: true
controller:
  enabled: true
# prometheus configures a Prometheus installation.
prometheus:
  # When true, the Helm chart will install a demo Prometheus server
  # instance alongside Consul.
  enabled: true

Notice that there are several metrics settings included throughout the configuration. In this tutorial, we'll discuss these settings and how they affect the observability configuration of your service mesh.

For a complete list of all possible settings, refer to the official documentation.

Install Consul in your cluster

You can now deploy a complete Consul datacenter in your Kubernetes cluster using the official Consul Helm chart or the Consul K8S CLI.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

$ helm install --values config.yaml consul hashicorp/consul --create-namespace --namespace consul --version "0.43.0"

$ brew tap hashicorp/tap

$ brew install hashicorp/tap/consul-k8s

$ consul-k8s install -config-file=config.yaml -set global.image=hashicorp/consul:1.12.0

Check that Consul is running in your Kubernetes cluster using kubectl. Consul setup is complete when all pods have a status of Running, as illustrated in the following output.

$ kubectl get pods --namespace consul
NAME                                           READY   STATUS    RESTARTS   AGE
consul-client-dfrvz                            1/1     Running   0          5m8s
consul-connect-injector-b4c5d6748-2v2dl        1/1     Running   0          5m8s
consul-connect-injector-b4c5d6748-mxhwv        1/1     Running   0          5m8s
consul-controller-559465fd96-dp7ck             1/1     Running   0          5m8s
consul-server-0                                1/1     Running   0          5m8s
consul-webhook-cert-manager-7cf6df6c4f-brgwt   1/1     Running   0          5m8s
prometheus-server-5cbddcc44b-kk698             2/2     Running   0          5m8s

Global settings

Consul's metrics features can either be enabled globally or per component. By default, metrics features are disabled globally, and both the Consul UI and service mesh are configured to inherit their enablement from the global setting.

global:
  name: consul
  datacenter: dc1
  metrics:
    # Configures the Helm chart’s components to expose Prometheus metrics for the Consul service mesh.
    enabled: true
    # Publishes consul agent metrics. Only applicable if  global.metrics.enabled is true.
    enableAgentMetrics: true

By setting global.metrics.enabled to true as shown in this example YAML, you are turning on metrics collection for Envoy proxies, registered services, and Consul gateways globally. The configuration above also sets global.metrics.enableAgentMetrics to true, which configures Consul to publish and collect metrics for all Consul server and client agents.

The global.metrics.enableGatewayMetrics, setting which is not shown above, is turned on by default when metrics are enabled globally. You can override this setting manually if you do not wish to collect metrics about your Consul gateways.

Enable the Consul UI

To support visualizing metrics using the Consul UI, you must ensure that the Consul UI is enabled. We recommend you enable the Consul UI whether you plan to enable metrics or not.

ui:
  enabled: true

Service mesh configuration

Consul service mesh is configured using settings contained within the connectInject stanza. Set the connectInject.enabled setting to true to enable Consul service mesh.

connectInject:
  enabled: true
  default: true
  # Configures metrics for Consul service mesh services. All values are overridable
  # via annotations on a per-pod basis.
  metrics:
    # Configures the Consul sidecar to run a merged metrics server
    # to combine and serve both Envoy and Application metrics.
    defaultEnableMerging: true

By setting connectInject.default to true, as shown in this sample YAML, sidecars will be automatically injected with your service deployment by default. Also, note that a metrics stanza has been added for you, and that the defaultEnableMerging entry has been set to true. This ensures that the consul-sidecar that gets injected for each deployment is configured to merge Envoy metrics with your registered service metrics into a single metrics endpoint that Prometheus will then scrape.

Enable Prometheus

To install a demo instance of Prometheus to your mesh using the Prometheus Helm chart, set the prometheus.enabled entry to true. With this feature enabled, you can test Consul with Prometheus natively integrated beginning on Day 0. However, this feature is not intended for production environments. You should be aware that with this configuration, upgrading to future versions of the Consul Helm chart may result in Prometheus being upgraded as well.

# prometheus configures a Prometheus installation.
prometheus:
  # When true, the Helm chart will install a demo Prometheus server
  # instance alongside Consul.
  enabled: true

View in Consul UI

To view metrics in the UI, you will need to have services in the mesh deployed to your datacenter. We have provided a demo application in the repo that you downloaded earlier that you can use with this tutorial to test Consul's metrics features. You can use kubectl to deploy the demo application.

$ kubectl apply -f hashicups
service/frontend-service created
configmap/nginx-configmap created
deployment.apps/frontend created
service/products-api-service created
serviceaccount/products-api created
configmap/db-configmap created
deployment.apps/products-api created
service/postgres created
deployment.apps/postgres created
service/public-api-service created
deployment.apps/public-api created

Ensure that the demo application is healthy with kubectl get pods to verify that all pods have a status of Running before proceeding to the next step.

$ kubectl get pods --namespace default
NAME                           READY   STATUS    RESTARTS   AGE
frontend-6f548678bb-l7nkr      2/2     Running   0          2m9s
postgres-7847fc4696-kzvsc      3/3     Running   0          2m9s
product-api-59757677b8-n9r5q   2/2     Running   0          2m9s
public-api-7bb69fcfd6-t76rb    2/2     Running   0          2m10s

Once all application pods are running, start the traffic simulation job.

$ kubectl apply -f traffic.yaml
configmap/k6-configmap created
deployment.apps/traffic created

You can now view service metrics in the Consul UI. In a new terminal session, issue the following command to expose the Consul UI to your development host.

$ kubectl port-forward consul-server-0 --namespace consul 8500:8500
Forwarding from 127.0.0.1:8500 -> 8500
Forwarding from [::1]:8500 -> 8500

Open localhost:8500 in a new browser tab, and navigate to the "Services" screen. Select a service. You should observe that a chart with some basic metrics is embedded in the service tile.

Hover over the four tile elements to get tooltip descriptions of the different metrics. You can also hover the timeline chart embedded in the tile, and review additional metrics for any point in time during the last 15 minutes.

Next steps

In this tutorial, you were introduced to some observability features of Consul service mesh on Kubernetes, and were instructed on how to configure a demo installation of Prometheus suitable for development or evaluation. For more information on how to create a production grade observability pipeline for your Consul service mesh using Prometheus and Grafana in Kubernetes, refer to our Layer 7 Observability with Consul Service Mesh, Prometheus, Grafana, and Kubernetes. tutorial.

In addition to configuring metrics defaults globally via the Helm chart, you can also provide, and even override, configuration on a per-pod basis using annotations. Annotations are inspected by the connect-inject handler, and if relevant, passed on to the consul-sidecar. Refer to the official documentation for a complete discussion on how annotations will affect your metrics configuration.

In the tutorial, Manage Traffic with Consul Service Mesh, you will be introduced to the comprehensive traffic management features that Consul provides at the proxy, service, and WAN levels.