Agent Telemetry

The Consul agent collects various runtime metrics about the performance of different libraries and subsystems. These metrics are aggregated on a ten second (10s) interval and are retained for a configurable amount of time. An interval is the period of time between instances of data being collected and aggregated.

When telemetry is being streamed to an external metrics store, the interval is defined to be that store's flush interval.

You can use this telemetry information for to debug and observe Consul's operations.

To retrieve telemetry data, send a signal to the Consul process:

• On Unix, send USR1
• On Windows, sendBREAK.

When Consul receives the signal, it dumps the current telemetry information to the agent's stderr. The output resembles the following example:

## ...
[G] 'consul.raft.applied_index': 291.000
[G] 'consul.state.connect_instances.dc1.ingress-gateway': 0.000
[G] 'consul.state.config_entries.dc1.file-system-certificate': 0.000
[G] 'consul.version.1.21.5.': 1.000
[G] 'consul.memberlist.node.instances.lan.left': 0.000
[G] 'consul.session_ttl.active': 0.000
[G] 'consul.runtime.total_gc_pause_ns': 12378333.000
[G] 'consul.autopilot.failure_tolerance': 0.000
[G] 'consul.state.config_entries.dc1.mesh': 0.000
[G] 'consul.runtime.num_goroutines': 335.000
[G] 'consul.raft.commitNumLogs': 1.000
[G] 'consul.state.connect_instances.dc1.terminating-gateway': 0.000
[G] 'consul.state.connect_instances.dc1.connect-proxy': 4.000
[G] 'consul.state.kv_entries.dc1': 0.000
[G] 'consul.state.config_entries.dc1.service-router': 0.000
[G] 'consul.state.config_entries.dc1.service-resolver': 0.000
[G] 'consul.state.config_entries.dc1.service-splitter': 0.000
[G] 'consul.runtime.free_count': 1809554.000
[G] 'consul.runtime.heap_objects': 150825.000
[G] 'consul.raft.leader.dispatchNumLogs': 1.000
[G] 'consul.server.isLeader': 1.000
[G] 'consul.state.services.dc1': 10.000
[G] 'consul.state.connect_instances.dc1.api-gateway': 1.000
[G] 'consul.state.config_entries.dc1.service-intentions': 4.000
[G] 'consul.state.config_entries.dc1.terminating-gateway': 0.000
[G] 'consul.runtime.sys_bytes': 67721480.000
## ...

Telemetry information is also retrievable with the metrics API endpoint, with output available in JSON format and Prometheus format.

By default, all metric names of gauge type use the hostname of the Consul agent as a prefix. For example, consul.<hostname>.server.isLeader. To disable prefixing the hostname, set telemetry.disable_hostname=true in the agent configuration.

You can also send Consul telemetry to a statsite or statsd server where it can be aggregated and flushed to Graphite or any other metrics store. Configure telemetry parameters in the agent configuration to use this option.

A Grafana dashboard, maintained by the Consul team, is also available. The dashboard helps display these metrics for easy visualization and can be used as a starting point to create your own dashboards.

The following documentation and tutorials provide additional examples of the telemetry configuration process:

• the Monitor Consul datacenter health with Telegraf documentation
• the Monitor Consul server health and performance with metrics and logs tutorial
• the Observe Consul service mesh traffic VMs tutorial
• the Observe Consul service mesh traffic Kubernetes tutorial

Key Metrics

These are some of the most important metrics that can help you understand the health of your cluster at a glance. For a full list of metrics emitted by Consul, refer to the Telemetry reference documentation.

• Transaction timing metrics, to monitor Consul write performances.
• Leadership changes metrics, to monitor Consul datacenter stability.
• Certificate Authority expiration metrics, to keep track of CA expiration and to plan certificate rotation.
• Autopilot metrics, to monitor the overall health of your cluster.
• Memory usage metrics, to monitor Consul resource usage.
• Garbage collection metrics, to monitor for time spent in garbage collection.
• Network activity metrics, to measure the load created from a Consul agent.
• Raft thread saturation metrics, to measure how much capacity a Consul server has to accept additional write load.
• Raft replication capacity issues metrics, to monitor the health and capacity of raft replication on servers
• License expiration metrics,
  Enterprise
  to monitor license validity.
• WAL logstore performance metrics, to monitor WAL backend performances.
• Bolt DB performance metrics, to monitor Bolt DB performances.

Transaction timing

Why they're important: Taken together, these metrics indicate how long it takes to complete write operations in various parts of the Consul cluster. Generally these should all be fairly consistent and no more than a few milliseconds. Sudden changes in any of the timing values could be due to unexpected load on the Consul servers, or due to problems on the servers themselves.

What to look for: Deviations (in any of these metrics) of more than 50% from baseline over the previous hour.

Leadership changes

Why they're important: Normally, your Consul cluster should have a stable leader. If there are frequent elections or leadership changes, it would likely indicate network issues between the Consul servers, or that the Consul servers themselves are unable to keep up with the load.

What to look for: For a healthy cluster, you're looking for a lastContact lower than 200ms, leader > 0 and candidate == 0. Deviations from this might indicate flapping leadership.

Certificate Authority expiration

Why they're important: Consul service mesh requires a CA to sign all certificates used to identify and connect the services. The mesh network ceases to work if they expire and become invalid. The root CA is particularly important to monitor as Consul does not automatically rotate it. The TLS certificate metric monitors the certificate that the server's agent uses to connect with the other agents in the cluster.

What to look for: The Root CA should be monitored for an approaching expiration, to indicate it is time for you to rotate the root CA either manually or with external automation. Consul should rotate the signing (intermediate) certificate automatically, but we recommend monitoring the rotation. When the certificate does not rotate, check the server agent logs for messages related to the CA system. The agent TLS certificate's rotation handling varies based on the configuration.

Autopilot

Why it's important: Autopilot can expose the overall health of your cluster with a simple boolean.

What to look for: Alert if healthy is 0. Some other indicators of an unhealthy cluster would be:

• High consul.raft.commitTime. This can help reflect the speed of state store changes being performed by the agent. If this number is rising, the server may be experiencing an issue due to degraded resources on the host.
• Leadership change metrics - Check for deviation from the recommended values. This can indicate failed leadership elections or flapping nodes.

Memory usage

Why they're important: Consul keeps all of its data in memory. If Consul consumes all available memory, it will crash.

What to look for: If consul.runtime.sys_bytes exceeds 90% of total available system memory.

Garbage collection

Why it's important: GC pause is a "stop-the-world" event, meaning that all runtime threads are blocked until GC completes. Normally these pauses last only a few nanoseconds. But if memory usage is high, the Go runtime may GC so frequently that it starts to slow down Consul.

What to look for: Warning if total_gc_pause_ns exceeds 2 seconds/minute, critical if it exceeds 5 seconds/minute.

Network activity - RPC count

Why they're important: These measurements indicate the current load created from a Consul agent, including when the load becomes high enough to be rate limited. A high RPC count, especially from consul.client.rpc.exceeded meaning that the requests are being rate-limited, could imply a misconfigured Consul agent.

What to look for: Sudden large changes to the consul.client.rpc metrics (greater than 50% deviation from baseline). consul.client.rpc.exceeded or consul.client.rpc.failed count > 0, as it implies that an agent is being rate-limited or fails to make an RPC request to a Consul server.

Raft thread saturation

Why they're important: These measurements are a useful proxy for how much capacity a Consul server has to accept additional write load. High saturation of the Raft goroutines can lead to elevated latency in the rest of the system and cause cluster instability.

What to look for: Generally, a server's steady-state saturation should be less than 50%.

Raft replication capacity issues

Why they're important: These metrics allow operators to monitor the health and capacity of raft replication on servers. When Consul is handling large amounts of data and high write throughput it is possible for the cluster to get into the following state:

• Write throughput is at 500 commits per second or higher, and constant
• The leader is writing out a large snapshot every minute or so
• The snapshot is large enough that it takes considerable time to restore from disk on a restart or from the leader if a follower gets behind
• Disk IO available allows the leader to write a snapshot faster than it can be restored from disk on a follower

Under these conditions, a follower after a restart may be unable to catch up on replication and become a voter again since it takes longer to restore from disk or the leader than the leader takes to write a new snapshot and truncate its logs. Servers retain raft_trailing_logs (default 10240) log entries even if their snapshot was more recent. On a leader processing 500 commits/second, that is only about 20 seconds worth of logs. Assuming the leader is able to write out a snapshot and truncate the logs in less than 20 seconds, there will only be 20 seconds worth of "recent" logs available on the leader right after the leader has taken a snapshot and never more than about 80 seconds worth assuming it is taking a snapshot and truncating logs every 60 seconds.

In this state, followers must be able to restore a snapshot into memory and resume replication in under 80 seconds otherwise they will never be able to rejoin the cluster until write rates reduce. If they take more than 20 seconds then there will be a chance that they are unlucky with timing when they restart and have to download a snapshot again from the servers one or more times. If they take 50 seconds or more then they will likely fail to catch up more often than they succeed and will remain non-voters for some time until they happen to complete the restore just before the leader truncates its logs.

In the worst case, the follower will be left continually downloading snapshots from the leader which are always too old to use by the time they are restored. This can put additional strain on the leader transferring large snapshots repeatedly as well as reduce the fault tolerance and serving capacity of the cluster.

Since Consul 1.5.3 raft_trailing_logs has been configurable. Increasing it allows the leader to retain more logs and give followers more time to restore and catch up. The tradeoff is potentially slower appends which eventually might affect write throughput and latency negatively so setting it arbitrarily high is not recommended.

Since Consul 1.10.0 raft_trailing_logs is now reloadable with consul reload or SIGHUP allowing operators to increase this without the leader restarting or loosing leadership allowing the cluster to be recovered gracefully.

Monitoring these metrics can help avoid or diagnose this state.

What to look for:

consul.raft.leader.oldestLogAge should resemble a saw-tooth wave that increases linearly with time until the leader takes a snapshot, when it jumps down as the oldest logs are truncated. The lowest point on that line should remain comfortably higher (2x or more) than the time it takes to restore a snapshot.

There are two ways a snapshot can be restored on a follower: from disk on startup or from the leader during an installSnapshot RPC. The leader only sends an installSnapshot RPC if the follower is new and has no state, or if its state is too old for it to catch up with the leaders logs.

consul.raft.fsm.lastRestoreDuration shows the time it took to restore from either source the last time it happened. Most of the time this duration begins when the server was started. It is a gauge that will always show the last restore duration (in Consul v1.10.0 and later), however long ago that was.

consul.raft.rpc.installSnapshot is the timing information from the leader's perspective when it installs a new snapshot on a follower. It includes the time spent transferring the data as well as the follower restoring it. Since these events are typically infrequent, you may need to graph the last value observed, for example using max_over_time with a large range in Prometheus. While the restore part will also be reflected in lastRestoreDuration, it can be useful to observe this too since the logs need to be able to cover this entire operation including the snapshot delivery to ensure followers can always catch up safely.

Graphing consul.raft.leader.oldestLogAge on the same axes as the other two metrics here can help see at a glance if restore times are creeping dangerously close to the limit of what the leader is retaining at the current write rate.

Note that if servers don't restart often, then the snapshot could have grown significantly since the last restore happened so last restore times might not reflect what would happen if an agent restarts now.

License expiration

Enterprise

Why they're important: This measurement indicates how many hours are left before the Consul Enterprise license expires. When the license expires, some Consul Enterprise features stop functioning. For example, after license expiration it is no longer possible to create or modify resources in non-default namespaces, nor to manage namespace definitions themselves even though reads of namespaced resources still work.

What to look for: This metric should be monitored to ensure that the license does not expire to prevent degradation of functionality.

WAL logstore performance

The WAL LogStore replaced BoltDB as the backend used in previous versions of Consul. Refer to Persistent data backend architecture for more information.

Requirements:

• Consul 1.20.0+

Why they're important: The consul.raft.wal.last_segment_age_seconds is a direct indicator on how quickly writes are filling the disk.

What to look for: The primary thing to look for are increases in the consul.raft.wal.last_segment_age_seconds times. Combining this information with the values of consul.raft.commitTime, consul.raft.rpc.installSnapshot, and consul.raft.leader.dispatchLog should provide you with an overview of the logstore performances.

Bolt DB performance

Requirements:

• Consul 1.11.0+

Why they're important: The consul.raft.boltdb.storeLogs metric is a direct indicator of disk write performance of a Consul server. If there are issues with the disk or performance degradations related to Bolt DB, these metrics will show the issue and potentially the cause as well.

What to look for: The primary thing to look for are increases in the consul.raft.boltdb.storeLogs times. Its value will directly govern an upper limit to the throughput of write operations within Consul.

In Consul, each write operation will turn into a single Raft log to be committed. Raft will process these logs and store them within Bolt DB in batches. Each call to store logs within Bolt DB is measured to record how long it took as well as how many logs were contained in the batch. Writing logs in this fashion is serialized so that a subsequent log storage operation can only be started after the previous one completed. The maximum number of log storage operations that can be performed each second is represented with the consul.raft.boltdb.writeCapacity metric. When log storage operations are becoming slower you may not see an immediate decrease in write capacity due to increased batch sizes of the each operation. However, the max batch size allowed is 64 logs. Therefore if the logsPerBatch metric is near 64 and the storeLogs metric is seeing increased time to write each batch to disk, then it is likely that increased write latencies and other errors may occur.

There can be a number of potential issues that can cause this. Often times it could be performance of the underlying disks that is the issue. Other times it may be caused by Bolt DB behavior. Bolt DB keeps track of free space within the raft.db file. When needing to allocate data it will use existing free space first before further expanding the file. By default, Bolt DB will write a data structure containing metadata about free pages within the DB to disk for every log storage operation. Therefore if the free space within the database grows excessively large, such as after a large spike in writes beyond the normal steady state and a subsequent slow down in the write rate, then Bolt DB could end up writing a large amount of extra data to disk for each log storage operation. This has the potential to drastically increase disk write throughput, potentially beyond what the underlying disks can keep up with. To detect this situation you can look at the consul.raft.boltdb.freelistBytes metric. This metric is a count of the extra bytes that are being written for each log storage operation beyond the log data itself. While not a clear indicator of an actual issue, this metric can be used to diagnose why the consul.raft.boltdb.storeLogs metric is high.

If Bolt DB log storage performance becomes an issue and is caused by free list management then setting raft_logstore.boltdb.no_freelist_sync to true in the server's configuration may help to reduce disk IO and log storage operation times. Disabling free list syncing will however increase the startup time for a server as it must scan the raft.db file for free space instead of loading the already populated free list structure.