Configure auto-instrumentation

Configure Consul Agentic Runtime Fabric proxies to generate OpenTelemetry spans for AI model and Model Context Protocol (MCP) tool calls. When you finish, supported calls produce correlated network and payload spans in your trace backend, with no tracing SDK in your applications.

Beta feature

Beta functionality is stable but possibly incomplete and subject to change. We strongly discourage using beta features in production deployments of Consul.

To learn how auto-instrumentation works, refer to Auto-instrumentation.

Requirements

Consul Enterprise with a valid license for external processing, registered services, and configured mesh connectivity.
The consul-ai feature gate enabled. On Kubernetes, set the ai.enabled Helm value to true.
The AI Inference Gateway and MCP sidecar deployed for the model and tool traffic you want to observe.
Agent, inference, and tool calls must route through the configured sidecar upstreams or gateways, on HTTP-aware listeners.
An OpenTelemetry Collector and a trace backend reachable from the proxies and processors.
An entry point that establishes trace context and a sampling decision for each request. Without a sampled traceparent, payload spans can be missing in your trace backend. Refer to Trace context propagation for more details.
Applications that forward trace context on downstream calls. Refer to Trace context propagation.
Approved rules for content capture, redaction, access control, and retention. Refer to Privacy and content boundaries.

Use the deployment manifests and image versions supplied with your Consul Agentic Runtime Fabric distribution. Adapt addresses and service names to your deployment.

Enable AI Inference Gateway tracing

AI Inference Gateway tracing is off by default. Enable it in the Observability.Tracing block of the inference-gateway configuration entry.

Field	Description
`Enabled`	Set to `true` to export traces.
`OTLP.Endpoint`	The OpenTelemetry Protocol (OTLP) collector URL.
`OTLP.Insecure`	Sends OTLP without TLS when `true`. Use only in non-production environments.
`SampleRatio`	Fraction of requests to trace, from `0` to `1`.

When tracing is enabled, the gateway exports spans over OTLP and joins Envoy's trace through the inbound traceparent header.

Configure Envoy tracing

Configure the Envoy proxies on the observed paths to send network spans to the collector. For example configurations, refer to Distributed tracing.

Add the collector cluster to the proxy bootstrap configuration with envoy_extra_static_clusters_json. Enable HTTP/2 on the cluster for OTLP over gRPC.
Configure the OpenTelemetry tracer with envoy_tracing_json. Set the service name and reference the collector cluster.

Attach the payload processor

Attach the protocol-aware external processor that generates payload spans.

Attach the builtin/ext-proc Envoy extension to the observed outbound proxy listeners. Configure it to send request and response headers and bodies.
Configure buffered request and response bodies for the supported parsers, with explicit size limits.
Connect each processor to Envoy:
- AI Inference Gateway: Envoy reaches the processor through the local local_ext_proc cluster. On Kubernetes, the processor listens on the Unix domain socket /run/consul/ext_proc.sock shared with Envoy.
- MCP sidecar: Envoy reaches the interceptor over loopback on port 21101 by default. To change the port, set port in the service's interceptor block. On Kubernetes, the sidecar listens on a Unix domain socket by default. To use TCP instead, set the consul.hashicorp.com/ai-agent-addr annotation to a port that matches the interceptor port.
Set SERVICE_NAME for the processor.
Set OTLP_ENDPOINT to the collector's OTLP over HTTP /v1/traces endpoint.
Match each observed upstream to its protocol parser. The reference implementation selects parsers by upstream bind-port group.

Warning

Keep the external-processing endpoint private. The loopback connection is plaintext and relies on network-namespace isolation. On Kubernetes, the AI Inference Gateway socket relies on file permissions instead, and Envoy must run as the same user as the processor. Publishing either endpoint on a routable interface exposes an unauthenticated payload reader.

Start the components

Launch the components sequentially so that no spans are lost while the collector initializes.

Start the collector and the trace backend.
Apply the service and proxy configuration.
Start the sidecars and their processors. Recreate any proxy that is already running. Writing a new configuration entry does not replace a running proxy's bootstrap configuration.

Verify spans reach the trace backend

Use synthetic, non-sensitive inputs for every step.

Send a sampled request through the entry point to an agent that makes one model call and one tool call.
Inspect the active proxy configuration. Confirm that it contains the OpenTelemetry tracer and the external-processing filter.
Confirm that the processor receives exchanges. A successful application response does not show that instrumentation works.
Confirm that the collector exports the spans to the trace backend. A collector acknowledgement does not show that the backend stored them.
Retrieve the trace by ID from the trace backend.
Confirm that network spans and payload spans share the trace context. Each payload span appears as a sibling of its Envoy egress span.
Confirm that payload spans carry the processor's service name, separate from the Envoy service name.
Confirm that the tool name, approved arguments, result, model information, and protocol errors match the synthetic request and response.
Confirm that the applications initialize no tracing SDK and emit no manual spans.
Confirm that the applications forward the trace context headers on downstream calls. Without them, the proxy spans do not form one workflow trace.
Test unsampled traffic, unsupported payloads, oversized bodies, and a collector outage. Check application behavior and telemetry-loss indicators.