Ensuring the health and performance of Consul and the services it supports is essential. Monitoring and observability provide the insights needed to understand system behavior, enabling proactive management and issue prevention.

Consul emits a wealth of telemetry and logs, offering in-depth visibility into the platform's functionality, and the services running on it. While Consul provides basic tools and UI to view these metrics, it is recommended to leverage feature-rich monitoring and observability tools such as Datadog, AppDynamics CNS, Grafana, and others. HashiCorp has partnered with industry-leading APM vendors to create integrations that make it easy to monitor Consul and its services as part of centralized monitoring and observability tools.

Before starting implementation, building a comprehensive observability strategy is vital to success. This strategy should encompass all teams—platform, DevOps, network, and security teams. It is crucial to select scalable and adaptable solutions that can meet growing demands.

HashiCorp advises developing a thorough plan to build and implement an observability strategy. This plan should align with both current objectives and future needs, ensuring it scales as your Consul usage grows.

Prerequisites

We recommend that you have completed the following steps before implementing the guidance in this document:

1. A fully functional, production-ready Consul cluster (LTS version 1.15.0 or higher)

2. Services registered and discovered in Consul

3. A fully deployed observability tool, such as Datadog

Build your observability strategy

To successfully implement a comprehensive observability solution, take a holistic approach. This involves thorough evaluation and standardization of the following elements:

• Strategy
• Personas
• Observability pillars
• Alerts and notifications
• Dashboards
• Processes
• Integrations
• Rollout strategy
• Project plan

Strategy

Begin by defining what success looks like. This involves understanding the needs of different stakeholders, selecting appropriate tools and processes, and aligning these elements with your business objectives.

Vision

As with any successful project or initiative, it is important to have an overarching vision for your observability needs. This vision will help guide teams and priorities, ensuring alignment with the overall goals.

Use cases

Engage with all stakeholders to identify and prioritize use cases based on business objectives. Here are a few examples:

• Proactive monitoring: Detect and address issues before they impact end users.

• Troubleshooting: Provide end-to-end visibility, and meaningful insights across applications, services, and infrastructure to aid in troubleshooting.

• Security and compliance: Provide data and insights for security and compliance assessments/audits.

• Capacity planning: Facilitate capacity planning for resources and infrastructure needed to support business growth.

Objectives

Establishing clear objectives that align with the needs and goals of your organization is crucial. Here are some observability objectives to consider:

Proactive monitoring

• Proactive issue detection and resolution
  • Goal: Identify and address issues before they impact users.
  • Key metrics: Error rates, anomaly detection rates, time to detect and resolve issues.
• Comprehensive visibility across systems
  • Goal: Achieve full visibility into the operational state of all components.
  • Key metrics: Coverage of monitored components, completeness of logging and tracing, and data granularity.

Troubleshooting

• Improved incident response and root cause analysis
  • Goal: Streamline incident response and effectively determine the root cause of issues.
  • Key metrics: Incident response time, root cause identification time, post-incident analysis effectiveness.
• Improved system reliability and availability
  • Goal: Minimize system downtime and ensure high availability of services.
  • Key metrics: Uptime percentage, Mean Time Between Failures (MTBF), Mean Time to Recovery (MTTR).

Security and compliance

• Enhanced security monitoring and compliance

  • Goal: Ensure robust security monitoring and compliance with regulatory requirements.

  • Key metrics: Security incident detection rate, compliance audit success rate, security monitoring coverage.

Capacity planning

• Enhanced performance monitoring

  • Goal: Ensure optimal performance of applications and services.

  • Key metrics: Response times, latency, throughput, resource utilization (CPU, memory).

• Scalability and flexibility of monitoring solutions

  • Goal: Ensure observability solutions can scale with business growth and adapt to changing requirements.

  • Key metrics: Scalability benchmarks, ease of integration with new technologies, adaptability to new use cases.

Other goals

Additionally, establish objectives that focus on business expectations, user experience, and organizational improvements. Here are a few such objectives to consider:

• SLAs/SLOs

  • Goal: Establish Service Level Agreements (SLAs) and Service Level Objectives (SLOs).

  • Importance: These are crucial for setting performance baselines, determining alert thresholds, and severities, and establishing appropriate notification methods.

• User experience optimization

  • Goal: Enhance the end-user experience by ensuring seamless and responsive services.

  • Key metrics: User satisfaction scores, application error rates, response times, and service availability from the end user's perspective.

• Improved collaboration across teams

  • Goal: Facilitate better collaboration and knowledge sharing among development, operations, and security teams.

  • Key metrics: Cross-team resolution times, number of collaborative incidents, shared documentation, and practices.

Personas

Observability solutions should be designed to cater to various personas within your organization. Identifying these relevant personas will help determine why they need the solution, how and when they will interact with it, and what information they will gather to address their issues. Here are some common personas to consider:

• Developers
• Platform engineers
  • Infrastructure specialists
  • DevOps engineers
  • Security architects
  • SREs
• Security engineers
• Network operators
• Cloud engineers
• Service owners
• Management

Identifying personas enables you to standardize various aspects of your observability strategy and implementation, such as—

• Role-Based Access Control (RBAC): Develop RBAC standards based on personas to ensure only the intended teams/people have access to create, update, delete, or view dashboards, alerts, and notifications. Not all personas need access to everything.

• Alerts and notifications: Design persona-based alerts and notifications to keep noise low and ensure relevant information reaches the right people.

• Dashboards: Create persona-based dashboards to speed up the troubleshooting process by presenting individuals with data and information relevant to them.

By tailoring your observability strategy to the specific needs and roles of different personas, you can enhance the effectiveness of your monitoring and incident response processes.

Observability pillars

When monitoring Consul, it is helpful to tackle it based on the two primary components:

Monitoring the control plane

• This includes your Consul servers, which manage the cluster's overall operations. These servers can run on bare metal, virtual machines (VMs), or within Kubernetes environments. However, it is strongly recommended to deploy them on VMs, following HashiCorp's recommended deployment architecture.

Monitoring the data plane

• This involves observing the Consul clients and associated elements. Key components include Consul agents (clients), sidecar proxies, Consul dataplanes (when operating within containerized environments like Kubernetes), and Gateways, which include Mesh, API, and Terminating Gateway.

For both the control plane and data planes, it is essential to establish standards across the three core pillars of observability: metrics, logs, and traces. This approach ensures comprehensive monitoring is configured, which provides visibility for effectively managing and troubleshooting your Consul environment.

Monitor the control plane

Metrics

Consul emits a significant amount of telemetry metrics, which can be overwhelming. To start, focus on The Four Golden Signals of monitoring:

1. Latency: ​​The time it takes to service a request.

2. Traffic: The volume of requests a system handles at any given time.

3. Errors: The rate of failed requests, either explicitly (e.g., HTTP 500s), implicitly (e.g., HTTP 200 success response with incorrect content), or by policy (e.g., any request over a committed response time).

4. Saturation: The percentage of resources consumed, indicating how full your system is.

Key metrics to monitor

• System metrics

  Visibility into system resource usage helps understand server saturation. Key resources include:

  • CPU
  • Memory
  • Disk

  Refer to “Server host metrics” for specific metrics for the above resources

• Consul metrics

  Server health: Please refer to the Consul documentation “Agent telemetry - server health” for details.

Enable Consul telemetry

Configure each Consul agent (server) to expose and capture local agent metrics with Prometheus and DogstatsD:

telemetry = {
 prometheus_retention_time = "1h"
 dogstatsd_addr            = "127.0.0.1:8125"
 disable_hostname = true
}

Recommendations

• Recommended server metrics to monitor:

  • Transaction timing
  • Leadership changes
  • Certificate authority expiration
  • Autopilot
  • Memory usage
  • Garbage collection
  • Network activity - RPC count
  • Raft thread saturation
  • Raft replication capacity issues
  • License expiration
  • Bolt DB performance

  For full details on the specific metrics refer to “Agent telemetry - server health”

• Additionally, monitor the following metrics:

  • consul.rpc.* - specifically, consul.rpc.query and consul.rpc.queries_blocking are the two most relevant.

  • consul.grpc.server.* - specifically, consul.grpc.server.stream.count, and consul.grpc.server.streams show the number of streams being processed by the server.

  • consul.xds.server.streams - number of xDS streams. In version 1.14 and higher, this metric will be a large source of read load.

References

• Datadog, Consul, and Envoy metrics monitoring | VM Edition

• Datadog - Key metrics for monitoring Consul

• Monitor Consul server health and performance with metrics and logs

Logs

Consul generates two types of logs, Consul application logs and audit logs. Each serve distinct purposes for system monitoring and compliance. These logs contain valuable information regarding system interactions and current operational status. Integrating these logs into your Application Performance Monitoring (APM) platform enables the correlation of metrics with log events, facilitating efficient troubleshooting of issues.

Consul logs

By default, Consul generates application logs and outputs to stdout. Consul logs capture information about the internal state of Consul, such as errors and messages. This information can be valuable in troubleshooting issues. The logs can be redirected to syslog or a file.

To redirect Consul logs to a file, add the following to your agent configuration:

log_file = "/var/log/consul/consul.log"
log_level = "INFO"
log_rotate_bytes = 52428800
log_rotate_duration = "24h"
log_rotate_max_files = 7

Recommendations

• Configure Consul logs to file.

• Configure log rotation by setting appropriate values for log_rotate_bytes, log_rotate_duration, and log_rotate_max_files.

• Limit who has access to the logs by setting appropriate file permissions.

• Configure your APM tool to ingest the logs from all Consul agents (e.g. "Datadog - enable log collection").

References

• Consul log configuration parameters

Audit logs

Consul can generate audit logs if enabled. This is an Enterprise feature, so you will need a valid license file. Actionable logs of authenticated events (both attempted and committed) that Consul processes via its HTTP API are captured in these logs. These events are then compiled into a JSON format for easy export and contain a timestamp, the operation performed, and the user who initiated the action.

Audit logging enables security and compliance teams within an organization to get greater insight into Consul access and usage patterns.

To enable audit logging, add the following to your agent configuration:

audit {
  enabled = true
  sink "My sink" {
    type   = "file"
    format = "json"
    path   = "data/audit/audit.json"
    delivery_guarantee = "best-effort"
    rotate_duration = "24h"
    rotate_max_files = 15
    rotate_bytes = 25165824
  }
}

To learn more about configuring and interpreting audit logs, please refer to the “Capture Consul events with audit logging” tutorial.

Recommendations

• Configure log rotation by setting appropriate values for rotate_duration, rotate_max_files, and rotate_bytes.

• Limit who has access to audit logs.

• Configure your APM tool to ingest audit logs from all Consul agents (e.g. "Datadog - enable log collection").

References

• Audit logging
• Audit log configuration parameters

Traces

Traces provide a view of the traffic flow from one service to another as users and applications interact with the system. In the context of Consul, it involves capturing layer 7 metrics from the sidecar proxy running on the Consul dataplane. These metrics offer visibility into the flow of traffic, errors, and response times, helping to identify issues and bottlenecks.

Since this primarily relates to the dataplane, we will cover it in detail in the “Monitor the data plane” section below.

Monitor the data plane

Metrics

Key metrics to monitor

• System metrics

  Similar to the system metrics from Consul servers, it is important to monitor the system resources of your data plane nodes.

  • CPU
  • Memory
  • Disk

  Refer to server host metrics for the specific metrics for the above resources.

• Consul metrics

  The data plane comprises Consul agents (clients), sidecar proxies, and gateways (mesh, ingress, and terminating).

  • Consul agent metrics - Similar to the Consul server agent, agents running as clients emit telemetry data - Refer to metrics reference.

• Gateways (mesh, API, terminating)

  Gateways also emit telemetry related to the network traffic passing through them. Metrics include ingress/egress request details, errors, and performance information.

• Consul dataplane

  When running in containerized environments, Consul supports a lightweight agent called Consul dataplane, which also emits telemetry. Refer to "Consul dataplane telemetry" for more information.

• Services and health checks

  Additionally, monitor the services running in the Consul datacenter. These include:

  • Catalog

    Use the catalog command to query all registered services in a Consul data center.

  • Service endpoints

    Use the /agent/service/:service_id API endpoint to query individual services.

  • Health checks

    Use the local service healthcheck to get the aggregated state of service(s) on the local agent by name

• Network

  In a microservices environment, the real-time traffic and health of the network is critical, so monitoring it is valuable. APM tools like Datadog provide a robust network monitoring solution (NPM) which should be leveraged.

  • Datadog: Refer to "Datadog NPM now supports Consul networking" for more details.

Enable Consul telemetry

Configure each Consul agent (client) to expose and capture local agent metrics with Prometheus and DogstatsD:

telemetry = {
 prometheus_retention_time = "1h"
 dogstatsd_addr            = "127.0.0.1:8125"
 disable_hostname = true
}

Proxy monitoring

Consul proxy metrics provide detailed health and performance information about your service mesh applications. This includes upstream/downstream network traffic metrics, error rates, and additional performance information that you can use to understand your distributed applications.

Once proxy metrics are enabled in Consul, you do not need to configure or instrument your applications in the service mesh to leverage proxy metrics.

Enabling Envoy sidecar proxy metrics

This can be configured in two ways:

1. Global default, which applies to all Envoy proxies in the Consul datacenter.

   # File: proxy-defaults.hcl
   Kind        = "proxy-defaults"
   Name        = "global"
   Config      = {
    envoy_prometheus_bind_addr = "127.0.0.1:9102"
    envoy_dogstatsd_url        = "udp://127.0.0.1:8125"
   }
   

   Refer to “Proxy defaults configuration reference” documentation for more details.

2. Service (sidecar) specific, which applies to the specifically configured proxy.

   # web.hcl - web service and sidecar proxy registration
   service {
    name = "web"
    id   = "web-1"
    port = 8080
   
    connect {
      sidecar_service{
        proxy = {
          config = [
            {
                envoy_prometheus_bind_addr = "127.0.0.1:9102"
               envoy_dogstatsd_url        = "udp://127.0.0.1:8125"
            }
          ]
        }
      }
    }
   

   For the configured changes to take effect, restart Envoy, or re-generate its bootstrap configuration using the consul connect envoy command

Recommendations

For full end-to-end visibility, HashiCorp recommends collecting telemetry from all components of your service mesh namely, agents (client), sidecar proxy (envoy proxy), gateways (envoy proxy), and Consul data plane.

References

• Monitor application health and performance with Consul proxy metrics

• Consul dataplane telemetry

• Configure metrics for Consul on Kubernetes

• Monitoring service-to-service communication with Envoy

Logs

Similar to the Consul control plane, the data plane components also generate both application logs, and access logs. The primary components are the Consul agents (clients).

Consul logs

The Consul agent (client) running in your Consul data center generates logs to stdout by default. Agents can be configured to redirect logs to syslog or a file.

To redirect Consul logs to a file, add the following to all of your agent (client) configurations:

log_file = "/var/log/consul/consul.log"
log_level = "INFO"
log_rotate_bytes = 52428800
log_rotate_duration = "24h"
log_rotate_max_files = 7

Recommendations

• Enable logs from all components of your data plane.

• Configure log rotation by setting appropriate values for log_rotate_bytes, log_rotate_duration, and log_rotate_max_files.

• Limit who has access to the logs by setting appropriate file permissions.

• Configure your APM tool to ingest the logs from all Consul agents (e.g. "Datadog - enable log collection").

References

• Monitor Consul server health and performance with metrics and logs

Access logs

Envoy proxy can emit access logs, which record application connections and requests that pass through proxies in a service mesh, including sidecar proxies and gateways.

This can help in troubleshooting issues, threat detection, and audit compliance.

• Sidecar proxy can be configured to emit access logs. These logs record service-to-service connections and requests in a service mesh.

  It is configured globally as follows:

  Kind      = "proxy-defaults"
  Name      = "global"
  AccessLogs {
     Enabled = true
  }
  

• Gateway logs

  Like the sidecar proxy (Envoy), gateways (mesh, API, and terminating) can emit access logs to record network traffic that passes through them in a service mesh.

Recommendations

• Enable access logs in JSON format so APM tools can ingest them.

• Envoy proxy does not handle log rotation for the access logs it generated. Use an appropriate log rotation tool to periodically rotate logs.

References

• Access logs
• Debug service mesh events and errors with Consul proxy access logs

Audit logs

Consul agents that run as clients in your data plane also generate audit logs. Like your Consul servers, configure all your Consul client agents to enable audit logging.

To enable audit logging add the following to all your agent (client) configurations:

audit {
  enabled = true
  sink "My sink" {
    type   = "file"
    format = "json"
    path   = "data/audit/audit.json"
    delivery_guarantee = "best-effort"
    rotate_duration = "24h"
    rotate_max_files = 15
    rotate_bytes = 25165824
  }
}

Recommendations

• Configure log rotation by setting appropriate values for rotate_duration, rotate_max_files, and rotate_bytes.

• Limit who has access to audit logs.

• Configure your APM tool to ingest audit logs from all Consul agents.

References

• Audit logging

• Audit log configuration parameters

• Capture Consul events with audit logging

Traces

Traces provide a comprehensive and continuous view of an application’s network traffic, aiming to follow a request flow as it moves from one service to another. Think of tracing as representing a single user’s journey through an entire app stack. Its primary purpose is optimization rather than being reactive. By tracing through a stack, developers can pinpoint errors or performance bottlenecks.

When issues arise, tracing allows you to understand how the user encountered the problem by examining:

• Which function was involved
• The function’s duration
• Parameters passed
• How deep into the function the user progressed

The two parts of traces

1. Trace: Represents the entire journey of a request or action as it moves through various nodes of a distributed system, especially containerized applications or microservices architectures.

2. Span: An operation or work taking place on a service. For example, a web server responding to an HTTP request or a single invocation of a function. A span has a start time and an end time. A series of tagged time intervals, known as spans, form a single trace in distributed tracing.

Instrument your application with Datadog

Consul does not automatically implement tracing for your applications. They must be instrumented to support tracing.

If Datadog is your APM solution, you can use Datadog’s SDK and tracing library (Tracer) to instrument your applications.

For more implementation details and information on Datadog SDK’s supported environments and programming languages, please refer to Datadog’s "Application Instrumentation" documentation.

Consul trace configuration

This involves configuring your sidecar proxy (Envoy) by adding spans. The following two keys need to be configured:

• envoy_tracing_json
• envoy_extra_static_cluster_json

For details on trace configuration, please refer to the following articles:

• Consul’s distributed tracing

• Envoy tracers

Configure proxy defaults so the settings will automatically apply to all proxies by default.

Recommendations

• Correlate logs to traces

  • Inject span IDs into your log messages, so your APM tool can correlate them.

    • Datadog: For implementation details please refer to Datadog’s “Correlate Logs and Traces” document.

• While implementing the instrumentation of your application, thoroughly test and verify the tracing data:

  • Use Consul’s (or other 3rd party tools') fault injection feature to inject errors and timeouts.

  • Stress test your complete environment and verify your tracing implementation is identifying the bottlenecks.

• Data security

  As with telemetry and logging data, tracing data can potentially expose sensitive data, such as personal identifiable information (PII). Be mindful and take measures to obfuscate such data.

  • Datadog: You can either leverage Sensitive Data Scanners or implement Datadog’s tracing library to remediate such data before sending it to Datadog.

    Refer to Datadog’s Data Security documentation for more information.

References

• Distributed tracing

• Enabling distributed tracing with HashiCorp Consul

• Consul demo - tracing datadog (Demo)

• Distributed tracing with Consul (Video)

• Leveraging HashiCorp Consul and Datadog for service mesh deployments (Video)

• Proxy defaults configuration reference

• Proxy parameters

Additional observability elements

Alerts and notifications

Alerts and notifications are proactive measures to notify teams of abnormal application behavior. This plays a vital role in observability. They ensure that relevant teams are promptly informed of any issues to prevent potential outages.

When designing alerts and notifications, it is crucial to consider personas and establish consistent standards. Here are key aspects to consider when setting up alerting standards:

• Persona: Determine who will receive the notifications based on their role and responsibilities.

• RBAC (Role-Based Access Control): Define who can create, update, delete, and manage alert configurations.

• Metrics to alert on: Select metrics that are critical to monitor based on your service level agreements (SLAs) and objectives (SLOs).

• Thresholds: Set appropriate thresholds that align with performance targets and operational requirements.

• Notification channels: Choose effective channels such as Slack, PagerDuty, JIRA, email, etc., to ensure alerts reach the right teams promptly.

Recommendations

• Persona based alerts: Customize alerts with relevant titles, thresholds, severity levels, messages, notification channels, and recipients.

• Warning thresholds: Set lower thresholds to receive warnings before critical thresholds are breached.

• Meaningful notification titles: Use variables when possible to automatically populate alert titles and messages, providing quick insights into the alert trigger.

• Monitoring messages: Include detailed incident information and resolution steps, specifying who to contact (e.g., service owner, technical contact) for swift issue resolution.

• Noise reduction: Ensure effective alerts by minimizing noise—avoid excessive alerting or notifications.

• Flexible alert management: Provide the ability to disable alerts during troubleshooting or maintenance windows as needed.

Refer to Consul's agent telemetry documentation for guidance on key Consul metrics and thresholds that are essential for configuring effective alerts.

Refer to Consul’s “Monitoring service-to-service communication with Envoy” documentation for guidance on key Envoy metrics and alerting thresholds.

References

• Consul monitoring and alerts recommendations

• Agent telemetry

Dashboards

Managing dynamically changing systems—both services and infrastructure—poses increasing challenges. Observability through visualization and dashboards offers a unified view of your entire IT landscape, enabling effective monitoring, and troubleshooting of applications and infrastructure.

Observability dashboards provide a robust solution by offering—

• Real-time insights into application and infrastructure health
• Centralized visibility across distributed systems
• Correlation of metrics, logs, and traces for rapid issue identification

Recommendations

• Create persona-based dashboards tailored to meet the specific needs of various roles within your organization:

  • SRE: Broad insights into services, infrastructure, and networking.

  • DevOps: Deeper insights into services, infrastructure, and networking, particularly in response to changes like deployments, feature rollouts, and upgrades.

  • Developers: Application and service-specific dashboards.

  • Network team: Detailed metrics related to network performance and issues.

  • Management: Overall service health and monitor SLA/SLO compliance.

• Based on your chosen visualization solution, leverage vendor-provided dashboards for Consul such as Datadog or Grafana, and customize them to meet your specific requirements.

By tailoring dashboards to the distinct needs of different personas, you ensure that each team member has the visibility necessary to perform their roles effectively.

Processes

Having insights into the health of your services is crucial, but timely and appropriate actions are necessary to address any issues. Established processes ensure that individuals and teams know exactly what steps to follow for enabling monitoring, and for efficient issue resolution.

Recommendations

Processes to implement:

• Establish baselines: Develop and agree upon baseline metrics with service owners, determining how and when these baselines will be established.

• In-take requests: Define how service owners should communicate their monitoring requirements to the platform team for configuration, including metrics, baselines, alerts, notifications, and dashboards.

• Incident response: Outline procedures for teams to follow during an incident, including how to inform customers, steps for troubleshooting, communication protocols with other teams, and escalation to management.

By having clear, standardized processes, you ensure a coordinated and efficient response to any issues that may arise, minimizing downtime and maintaining service quality.

Integrations

HashiCorp recommends integrating Consul with third-party APM tools to achieve end-to-end observability of your Consul control plane, Consul data plane, and all registered services. To facilitate this, HashiCorp has partnered with industry-leading APM vendors to provide simple-to-deploy integrations.

Integrating with your existing automation (CI/CD pipelines) is highly recommended.

Recommendations

• Consul integrations:

  • Datadog

  • Cisco AppDynamics

  • Grafana

  • Dynatrace

• Automation: Use automation tools like Terraform and Ansible to enable/configure monitoring as part of your infrastructure and service deployment process, typically within your CI/CD pipeline. This approach ensures that monitoring is consistent, seamlessly integrated, and operational from the moment your services are deployed.

By leveraging these integrations and automation tools, you can enhance your observability strategy and maintain comprehensive visibility across your systems even when new systems and services come online.

Rollout strategy

To minimize unexpected disruption it is recommended to establish a clear rollout strategy. This approach helps introduce observability-related changes in a controlled manner, reducing the risk of production issues.

Recommendations

• Take a phased approach:

  • Pilot phase:

    • Begin with a small-scale implementation focused on critical applications and infrastructure.

    • Evaluate the implementation and refine it based on feedback.

  • Full rollout:

    • Gradually expand the observability solution across all systems and teams.

    • Ensure proper documentation and training are provided.

  • Continuous monitoring and improvement:

    • Regularly review the performance of your observability baseline and processes.

    • Incorporate changes as they become available through the feedback process.

Project plan

To effectively implement your observability strategy, HashiCorp recommends structuring it as a dedicated project. Develop a comprehensive project plan that outlines all tasks, assigns owners, identifies dependencies, and sets clear timelines. This structured approach ensures efficient execution and timely completion of the project.

Example project plan

1. Project initiation phase

   • Define project scope, objectives, and success criteria.

   • Identify stakeholders and establish communication channels.

   • Assign project manager and core team members.

2. Planning phase

   • Conduct a thorough assessment of current observability capabilities.

   • Define SLAs/SLOs and establish baseline metrics.

   • Develop detailed requirements for metrics, logs, and tracing implementations.

   • Identify the observability and log management tools, as well as any auxiliary tools required to meet your objectives

   • Create a project timeline and milestone schedule.

3. Implementation phase

   • Deploy and configure observability tools and platforms (e.g., Datadog, Prometheus, Grafana, Jaeger).

   • Integrate monitoring agents and instrumentation into Consul control and data planes.

   • Set up centralized logging and implement distributed tracing frameworks.

4. Testing and validation phase

   • Conduct functional testing to ensure monitoring tools capture expected metrics.

   • Validate logging configurations and trace propagation across Consul components.

   • Perform load testing and simulate failure scenarios to verify alerting and response mechanisms.

5. Deployment and rollout phase

   • Plan and execute phased deployment of observability enhancements.

   • Provide training sessions for operations and support teams on new monitoring capabilities.

   • Monitor system performance post-deployment and address any immediate issues.

6. Monitoring and optimization phase

   • Establish ongoing monitoring processes and routines.

   • Continuously optimize metrics collection, logging practices, and tracing configurations.

   • Conduct regular reviews to refine SLAs/SLOs based on operational insights.

7. Documentation and knowledge sharing

   • Document project outcomes, including configurations, processes, and lessons learned.

   • Share best practices and operational guidelines with relevant teams.

   • Update documentation as new tools and practices are adopted.

By following this structured project plan, your organization can effectively implement HashiCorp's recommended observability strategy, ensuring robust monitoring and operational efficiency across your Consul environment.

Conclusion

Consul provides powerful tools for service discovery, health monitoring, and secure communication that integrate seamlessly into a comprehensive observability stack. Tailoring your strategy to include these features will ensure you maintain robust visibility and control over your applications, services, and infrastructure as your company grows.

Platform monitoring recommendations from initial configuration

Configure platform monitoring

After setting up your first datacenter, it is an ideal time to make sure your deployment is healthy and establish a baseline. Here are recommendations for monitoring your Consul control and data plane. By keeping track of these components and setting up alerts, you can better maintain the overall health and resilience of your service mesh.

Consul control plane monitoring

The Consul control plane consists of the following components:

• RPC communication between Consul servers and clients.
• Gossip Traffic: LAN
• Consul cluster peering

It is important to monitor and establish baseline and alert thresholds for Consul control plane components for detection of abnormal behavior. Note that these alerts can also be triggered by some planned events like Consul cluster upgrades, configuration changes, or leadership change.

To help monitor your Consul control plane, we recommend to establish a baseline and standard deviation for the following:

• Server health(opens in new tab)
• Leadership changes(opens in new tab)
• Key metrics(opens in new tab)
• Autopilot(opens in new tab)
• Network activity(opens in new tab)
• Certificate authority expiration(opens in new tab)

We recommend monitoring the following parameters for Consul agents’ health:

• Disk space and file handles
• RAM utilization(opens in new tab)
• CPU utilization
• Network activity and utilization

We recommend using an application performance monitoring (APM) system(opens in new tab) to track these metrics. For a full list of key metrics, visit the Key metrics(opens in new tab) section of Telemetry documentation.

Host-level alerts

When collecting metrics, it is important to establish a baseline. This baseline ensures your Consul deployment is healthy, and serves as a reference point when troubleshooting abnormal Cluster behavior. Once you have established a baseline for your metrics, use them and the following recommendations to configure reasonable alerts for your Consul agent.

Memory

Set up an alert if your RAM usage exceeds a reasonable threshold (for example, 90% of your allocated RAM). Refer to the Memory Alert Recommendations(opens in new tab) for further details.

Metrics to monitor:

Consul agents are running low on memory if consul.runtime.sys_bytes exceeds 90% of mem.total_bytes, mem.used_percent is over 90%, or swap.used_percent is greater than 0. You should increase the memory available to Consul if any of these three conditions are met.

CPU

Set up an alert to detect CPU spikes on your Consul server agents. Refer to the CPU Alert Recommendations(opens in new tab) for further details.

Metrics to monitor:

If cpu.iowait_cpu is greater than 10%, it should be considered critical as Consul is waiting for data to be written to disk. This could be a sign that Raft is writing snapshots to disk too often.

Network

The data sent between all Consul agents must follow latency requirements for total round trip time (RTT):

• Average RTT for all traffic cannot exceed 50ms. RTT for 99 percent of traffic cannot exceed 100ms.
• Set an alert to detect when the RTT exceeds these values. These values should be compared to the host's network latency so the RTT does not exceed these values.
• Refer to the Reference architecture(opens in new tab) to learn more about network latency and bandwidth guidance.

Metrics to monitor:

Sudden increases to the net metrics, greater than 50% deviation from baseline, indicates too many requests that are not being handled.

File descriptors

Metrics to monitor:

By default, process and kernel limits are conservative, you may want to increase the limits beyond the defaults. If the linux_sysctl_fs.file-nr value exceeds 80% of linux_sysctl_fs.file-max, the file handles should be increased.

Disk activity

Metrics to monitor:

Sudden, large changes to the disk I/O metrics, greater than 50% deviation from baseline or more than 3 standard deviations from baseline indicates Consul has too much disk I/O. Too much disk I/O can cause the rest of the system to slow down or become unavailable, as the kernel spends all its time waiting for I/O to complete.

Raft monitoring

Consul uses Raft for consensus protocol(opens in new tab). High saturation of the Raft goroutines can lead to elevated latency in the rest of the system and may cause the Consul cluster to be unstable. As a result, it is important to monitor Raft to track your control plane health. We recommend the following actions to keep control plane healthy:

• Create an alert that notifies you when Raft thread saturation(opens in new tab) exceeds 50%.
• Monitor Raft replication capacity(opens in new tab) when Consul is handling large amounts of data and high write throughput.
• Lower raft_multiplier(opens in new tab) to keep your Consul cluster stable. The value of raft_multiplier defines the scaling factor for Consul. Default value for raft_multiplier is 5.A short multiplier minimizes failure detection and election time but may trigger frequently in high latency situations. This can cause constant leadership churn and associated unavailability. A high multiplier reduces the chances that spurious failures will cause leadership churn but it does this at the expense of taking longer to detect real failures and thus takes longer to restore Consul cluster availability.Wide networks with higher latency will perform better with larger raft_multiplier values.

Raft protocol health

Metrics to monitor:

Saturation of consul.raft.thread of more than 50% can lead to elevated latency in the rest of the system and cause cluster instability.

Snapshot restore happens when a Consul agent is first started, or when specifically instructed to do so via RPC. For Consul servers, consul.raft.fsm.lastRestoreDuration tracks the duration of the operation. For Consul clients, from the leader's perspective when it installs a new snapshot on a follower, consul.raft.rpc.installSnapshot tracks the timing information. Both these metrics should be consistent, without sudden large changes.

Plotting of consul.raft.leader.oldestLogAge should look like a saw-tooth wave increasing linearly with time until the leader takes a snapshot and then jumping down as the oldest Raft logs are truncated. The lowest point on that line should remain comfortably higher (for example, 2x or more) than the time it takes to restore a snapshot.

BoltDB monitoring

Metrics to monitor:

Sudden increases in the consul.raft.boltdb.storeLogs times will directly impact the upper limit to the throughput of write operations within Consul.

If the free space within the database grows excessively large, such as after a large spike in writes beyond the normal steady state and a subsequent slow down in the write rate, then Bolt DB could end up writing a large amount of extra data to disk for each Raft log storage operation. This will lead to an increase in the consul.raft.boltdb.freelistBytes metric - a count of the extra bytes that are being written for each Raft log storage operation beyond the Raft log data itself. Sudden increases in this metric can be correlated to increases in the consul.raft.boltdb.storeLogs metric indicating an issue.

The maximum number of Raft log storage operations that can be performed each second is represented with the consul.raft.boltdb.writeCapacity metric. When Raft log storage operations are becoming slower you may not see an immediate decrease in write capacity due to increased batch sizes of each operation. Sudden changes in this metric should be further investigated.

The consul.raft.boltdb.logsPerBatch metric keeps track of the current batch size for Raft log storage operations. The maximum allowed is 64 Raft logs. Therefore if this metric is near 64 and the consul.raft.boltdb.storeLogs metric is seeing increased time to write each batch to disk, it is likely that increased write latencies and other errors may occur.

Consul dataplane monitoring

Consul data plane collects metrics about its own status and performance. Consul data plane uses the same external metrics store that is configured for Envoy. To enable telemetry for Consul data plane, enable telemetry for Envoy by specifying an external metrics store in the proxy-defaults configuration entry or directly in the proxy.config field of the proxy service definition.

Service monitoring

You can extract the following service-related information:

• Use the catalog(opens in new tab) command or the Consul UI to query all registered services in a Consul datacenter.
• Use the /agent/service/:service_id(opens in new tab) API endpoint to query individual services.