• HashiCorp Developer

  • HashiCorp Cloud Platform
  • Terraform
  • Packer
  • Consul
  • Vault
  • Boundary
  • Nomad
  • Waypoint
  • Vagrant
Consul
  • Install
  • Tutorials
  • Documentation
  • API
  • CLI
  • Try Cloud(opens in new tab)
  • Sign up
Consul Home

Documentation

Skip to main content
  • Documentation
  • What is Consul?


    • Overview
    • How Service Mesh Works
    • Configuration
    • Service-to-service permissions - Intentions
    • Service-to-service permissions - Intentions (Legacy Mode)
    • Transparent Proxy
    • Connectivity Tasks
    • Distributed Tracing
    • Nomad
    • Kubernetes
    • Develop and Debug


  • HCP Consul


  • Resources

  • Tutorial Library
  • Certifications
  • Community Forum
    (opens in new tab)
  • Support
    (opens in new tab)
  • GitHub
    (opens in new tab)
  1. Developer
  2. Consul
  3. Documentation
  4. Service Mesh
  5. Distributed Tracing
  • Consul
  • v1.13.x
  • v1.12.x
  • v1.11.x
  • v1.10.x
  • v1.9.x
  • v1.8.x

»Distributed Tracing

Distributed tracing is a way to track and correlate requests across microservices. Distributed tracing must first be implemented in each application, it cannot be added by Consul. Once implemented in your applications, adding distributed tracing to Consul will add the sidecar proxies as spans in the request path.

Application Changes

Consul alone cannot implement distributed tracing for your applications. Each application must propagate the required headers. Typically this is done using a tracing library such as:

  • https://github.com/opentracing/opentracing-go
  • https://github.com/DataDog/dd-trace-go
  • https://github.com/openzipkin/zipkin-go

Configuration

Once your applications have been instrumented with a tracing library, you are ready to configure Consul to add sidecar proxy spans to the trace. Your eventual config will look something like:

Kind = "proxy-defaults"
Name = "global"
Config {
  protocol           = "http"
  envoy_tracing_json = <<EOF
{
  "http":{
    "name":"envoy.tracers.zipkin",
    "typedConfig":{
      "@type":"type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
      "collector_cluster":"collector_cluster_name",
      "collector_endpoint_version":"HTTP_JSON",
      "collector_endpoint":"/api/v2/spans",
      "shared_span_context":false
    }
  }
}
EOF

  envoy_extra_static_clusters_json = <<EOF
{
  "connect_timeout":"3.000s",
  "dns_lookup_family":"V4_ONLY",
  "lb_policy":"ROUND_ROBIN",
  "load_assignment":{
    "cluster_name":"collector_cluster_name",
    "endpoints":[
      {
        "lb_endpoints":[
          {
            "endpoint":{
              "address":{
                "socket_address":{
                  "address":"collector-url",
                  "port_value":9411,
                  "protocol":"TCP"
                }
              }
            }
          }
        ]
      }
    ]
  },
  "name":"collector_cluster_name",
  "type":"STRICT_DNS"
}
EOF
}

apiVersion: consul.hashicorp.com/v1alpha1
kind: ProxyDefaults
metadata:
  name: global
spec:
  config:
    protocol: http
    envoy_tracing_json: |
      {
        "http":{
          "name":"envoy.tracers.zipkin",
          "typedConfig":{
            "@type":"type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
            "collector_cluster":"collector_cluster_name",
            "collector_endpoint_version":"HTTP_JSON",
            "collector_endpoint":"/api/v2/spans",
            "shared_span_context":false
          }
        }
      }
    envoy_extra_static_clusters_json: |
      {
        "connect_timeout":"3.000s",
        "dns_lookup_family":"V4_ONLY",
        "lb_policy":"ROUND_ROBIN",
        "load_assignment":{
          "cluster_name":"collector_cluster_name",
          "endpoints":[
            {
              "lb_endpoints":[
                {
                  "endpoint":{
                    "address":{
                      "socket_address":{
                        "address":"collector-url",
                        "port_value":9411,
                        "protocol":"TCP"
                      }
                    }
                  }
                }
              ]
            }
          ]
        },
        "name":"collector_cluster_name",
        "type":"STRICT_DNS"
      }
{
  "Kind": "ProxyDefaults",
  "Name": "global",
  "Config": {
    "protocol": "http",
    "envoy_tracing_json": "{\"http\":{\"name\":\"envoy.tracers.zipkin\",\"typedConfig\":{\"@type\":\"type.googleapis.com/envoy.config.trace.v3.ZipkinConfig\",\"collector_cluster\":\"collector_cluster_name\",\"collector_endpoint_version\":\"HTTP_JSON\",\"collector_endpoint\":\"/api/v2/spans\",\"shared_span_context\":false}}}",
    "envoy_extra_static_clusters_json": "{\"connect_timeout\":\"3.000s\",\"dns_lookup_family\":\"V4_ONLY\",\"lb_policy\":\"ROUND_ROBIN\",\"load_assignment\":{\"cluster_name\":\"collector_cluster_name\",\"endpoints\":[{\"lb_endpoints\":[{\"endpoint\":{\"address\":{\"socket_address\":{\"address\":\"collector-url\",\"port_value\":9411,\"protocol\":\"TCP\"}}}}]}]},\"name\":\"collector_cluster_name\",\"type\":\"STRICT_DNS\"}"
  }
}

NOTE: This example uses a proxy defaults config entry which will apply to all proxies but you can also apply this config in the proxy service registration (not supported on Kubernetes).

Within the config there are two keys you need to customize:

  1. envoy_tracing_json: Sets the tracing configuration for your specific tracing type. See the Envoy tracers documentation for your specific collector's configuration. This configuration will reference the cluster name defined in envoy_extra_static_clusters_json.
  2. envoy_extra_static_clusters_json: Defines the address of your tracing collector where Envoy will send its spans. In this example the URL was collector-url:9411.

Applying the configuration

This configuration only applies when proxies are restarted since it changes the bootstrap config for Envoy which can only be applied on startup. This means you must restart all your proxies for changes to this config to take effect.

Note: On Kubernetes this is a matter of restarting your deployments, e.g. kubectl rollout restart deploy/deploy-name.

Considerations

  1. Distributed tracing is only supported for HTTP and gRPC services. You must specify the protocol either globally via a proxy defaults config entry:

    Kind      = "proxy-defaults"
    Name      = "global"
    Config {
      protocol = "http"
    }
    
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ProxyDefaults
    metadata:
      name: global
    spec:
      config:
        protocol: http
    
    {
      "Kind": "proxy-defaults",
      "Name": "global",
      "Config": {
        "protocol": "http"
      }
    }
    

    Or via a service defaults config entry for each service:

    Kind      = "service-defaults"
    Name      = "service-name"
    Protocol  = "http"
    
    apiVersion: consul.hashicorp.com/v1alpha1
    kind: ProxyDefaults
    metadata:
      name: service-name
    spec:
      protocol: http
    
    {
      "Kind": "service-defaults",
      "Name": "service-name",
      "Protocol": "http"
    }
    
  2. Requests through Ingress Gateways will not be traced unless the header x-client-trace-id: 1 is set (see hashicorp/consul#6645).

  3. Consul does not currently support interoperation with OpenTelemetry libraries due to Envoy not yet having support.

  4. Tracing is only supported with Envoy proxies, not the built-in proxy.

  5. When configuring the Zipkin tracer in envoy_tracing_json, set trace_id_128bit to true if your application is configured to generate 128-bit trace IDs. For example:

    {
      "http": {
        "name": "envoy.tracers.zipkin",
        "typedConfig": {
          "@type": "type.googleapis.com/envoy.config.trace.v3.ZipkinConfig",
          "collector_cluster": "zipkin",
          "collector_endpoint_version": "HTTP_JSON",
          "collector_endpoint": "/api/v2/spans",
          "shared_span_context": false,
          "trace_id_128bit": true
        }
      }
    }
    
Edit this page on GitHub

On this page

  1. Distributed Tracing
  2. Application Changes
  3. Configuration
  4. Applying the configuration
  5. Considerations
Give Feedback(opens in new tab)
  • Certifications
  • System Status
  • Terms of Use
  • Security
  • Privacy
  • Trademark Policy
  • Trade Controls
  • Give Feedback(opens in new tab)