HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream

Observability setup

This page covers install-time telemetry and log wiring. Day-2 dashboard interpretation, alerting, rollout processes, and troubleshooting belong in the Administration Guide.

Monitoring

Monitoring Consul is crucial to ensure the health, performance, and reliability of your service discovery and configuration management infrastructure. Monitoring helps you identify issues, troubleshoot problems, and proactively manage your Consul cluster.

Metrics

Consul can be configured to send telemetry data to a remote monitoring system. This allows you to monitor the health of agents over time, spot trends, and plan for future needs. You will need a monitoring agent and console for this.

Consul supports the following telemetry agents:

  • prometheus
  • Circonus
  • DataDog (via dogstatsd)
  • StatsD (via statsd, statsite, telegraf, etc.)

If you are using StatsD, you will also need a compatible database and server, such as Grafana, Chronograf, or Prometheus.

Agent Telemetry can be enabled in the agent configuration file.

telemetry {
  dogstatsd_addr = "localhost:8125"
  disable_hostname = true
}

Consul on Kubernetes integrates with Prometheus and Grafana to provide metrics for Consul service mesh. The metrics available are:

  • Mesh service metrics
  • Mesh sidecar proxy metrics
  • Consul agent metrics
  • Ingress, terminating, and mesh gateway metrics

Prometheus annotations are used to instruct Prometheus to scrape metrics from Pods. Prometheus annotations only support scraping from one endpoint on a Pod, so Consul on Kubernetes supports metrics merging whereby service metrics and sidecar proxy metrics are merged into one endpoint. If there are no service metrics, it also supports just scraping the sidecar proxy metrics. Metrics for services in the mesh can be configured with the Helm values nested under connectInject.metrics(opens in new tab).

connectInject:
  metrics:
    defaultEnabled: true # by default, this inherits from the value     global.metrics.enabled
    defaultEnableMerging: true

The Prometheus annotations specify which endpoint to scrape the metrics from. The annotations point to a listener on 0.0.0.0:20200 on the Envoy sidecar. You can configure the listener and the corresponding Prometheus annotations using the following Helm values. Alternatively, you can specify the consul.hashicorp.com/prometheus-scrape-port and consul.hashicorp.com/prometheus-scrape-path Consul annotations to override them on a per-Pod basis:

connectInject:
  metrics:
    defaultPrometheusScrapePort: 20200
    defaultPrometheusScrapePath: "/metrics"

Consul agent metrics

Metrics from the Consul server Pods can be scraped with Prometheus by setting the field global.metrics.enableAgentMetrics to true. Additionally, one can configure the metrics retention time on the agents by configuring the field global.metrics.agentMetricsRetentionTime which expects a duration and defaults to "1m". This value must be greater than "0m" for the Consul servers to emit metrics at all. As the Prometheus deployment currently does not support scraping TLS endpoints, agent metrics are currently unsupported when TLS is enabled.

global:
  metrics:
    enabled: true
    enableAgentMetrics: true
    agentMetricsRetentionTime: "1m"

Gateway metrics

Metrics from the Consul ingress, terminating, and mesh gateways can be scraped with Prometheus by setting the field global.metrics.enableGatewayMetrics to true. The gateways emit standard Envoy proxy metrics. To ensure that the metrics are not exposed to the public internet, as mesh and ingress gateways can have public IPs, their metrics endpoints are exposed on the Pod IP of the respective gateway instance, rather than on all interfaces on 0.0.0.0.

global:
  metrics:
    enabled: true
    enableGatewayMetrics: true

Further, we need to authorize the Consul telemetry collector service to get metrics from other service proxies. We create service intentions that authorize proxies to push metrics to the collector.

Logs

Consul logs to standard output which can be redirected in your startup/init/systemd unit file or to any file you choose. That said, there is also a -syslog(opens in new tab) command line flag and a corresponding enable_syslog(opens in new tab) configuration file option that will enable Consul to log to syslog as well.

Additionally, you can specify the location of where you want the logs saved, the number of bytes that should be written to a log before it needs to be rotated, the duration a log should be written to before it needs to be rotated, and the maximum number of older log file archives to keep using the -log-file(opens in new tab), -log-rotate-bytes(opens in new tab), -log-rotate-duration(opens in new tab) and -log-rotate-max-files(opens in new tab) command line flags respectively.

You can also connect to any running Consul instance with consul monitor(opens in new tab) to view operational logging at any level from err to trace.

Edit this page on GitHub