Auto-instrumentation for AI agents

Consul Agentic Runtime Fabric auto-instrumentation collects telemetry data automatically and pushes it to AI observability tools with the help of OpenTelemetry Collector.

The AI Inference Gateway and Model Context Protocol (MCP) sidecar are specialized proxies that combine Envoy with a protocol-aware external processor to generate OpenTelemetry spans for supported model and tool calls. Applications do not need a tracing SDK, manual spans, or tracing decorators. This approach is sometimes called zero-code instrumentation.

Refer to Configure auto-instrumentation for setup instructions.

Introduction

Network telemetry shows which services communicated and if requests succeeded. To gain insight from AI observability, Consul Agentic Runtime Fabric must collect tool invocation arguments, model interaction, and the result of the interaction when an agent invokes a tool. AI observability details travel in application protocols, not in network span attributes.

The specialized proxies read those protocols as traffic passes through them:

• The AI Inference Gateway instruments supported model requests and responses.
• MCP sidecars instrument supported tool calls, including tool identity, arguments, results, and errors, subject to content-capture policy.

For example, an agent can call the same tool with the same arguments many times while every HTTP request succeeds. The proxy spans capture each call and its result, so the Agentic Analytics component can assess the repetition for cycle detection without custom spans in the application.

Auto-instrumentation covers only traffic that passes through these proxies. Calls that bypass the mesh and local tool execution are not instrumented. For the full list, refer to Limits.

How auto-instrumentation works

Each proxy produces two kinds of spans, and trace context ties them into one workflow trace.

The following diagram shows one request through a proxy and the spans it produces.

Agent ──traceparent──▶ [ Envoy ──ext_proc──▶ Processor ] ──▶ Model or tool
                          │                    │
                          ▼                    ▼
Trace:   entry span
         ├── egress span (Envoy)
         └── payload span (processor)   ← siblings

Envoy and the processor each emit a span. Both spans share the parent from the incoming traceparent, so they appear as siblings under the entry span.

Network spans

Envoy generates network spans for observed ingress and egress requests. These spans describe the service path, request timing, and transport outcomes. Consul supplies the proxy configuration and service connectivity.

Network observation needs HTTP-aware listeners. Opaque TCP forwarding does not expose the HTTP headers and bodies that payload observation reads.

Payload spans

An external processor runs alongside Envoy and receives request and response headers and bodies through Envoy's ext_proc interface. It parses supported protocols and emits a payload span for each observed AI interaction. The processor is infrastructure, not a library that the agent or tool imports.

The reference processor returns continuation responses without modifying the payload. It exports spans through a bounded background queue after the exchange, so the application does not wait for trace storage.

The payload span is a separate span, not extra attributes on Envoy's span. External processing runs before the router creates the egress span, so the payload span and the egress span share a parent and appear as siblings in the trace.

Payload span sensitive data security

Follow these guidelines to protect sensitive data when using auto-instrumentation:

• Do not enable content-bearing capture for a workload when the deployed path cannot enforce filtering or redaction.
• Keep credentials, tokens, and personal data out of telemetry.
• Restrict access to and retention of the remaining evidence.
• Expect redaction to limit downstream semantic analysis, including cycle detection.
• Do not promote trace IDs, arguments, prompts, or results into metric labels.

Trace context propagation

Proxies correlate observations according to the W3C Trace Context specification. You must establish the context and a sampling decision at the entry point. The reference deployment uses an infrastructure trace-root component in front of the API gateway.

The sampling decision determines whether a request is traced. It travels as the sampled flag, the lowest bit of the trace-flags field in the traceparent header. For example, a trace-flags value of 01 means sampled and 00 means not sampled. Make the decision once at the entry point so that every component on the request path receives the same value.

Applications forward incoming context on downstream calls. The reference applications forward traceparent, tracestate, baggage, and x-request-id. This is header propagation, not instrumentation, so applications still create no spans.

A proxy cannot infer which incoming request caused an outgoing request inside an application. Without propagation, observations do not form one workflow trace. In the reference processor, a missing, invalid, or unsampled traceparent prevents payload-span emission.

Applications remain responsible for propagating context and for routing calls through the mesh.

Observed signals

The following table lists what the proxies observe for each type of interaction. Availability depends on the protocol, the visible request and response data, sampling, and size limits.

MCP payload spans use attributes such as mcp.tool, mcp.arguments, mcp.result.text, and mcp.is_error. Inference payload spans use gen_ai.* attributes. The schema is version specific, and not every attribute is available for every provider or protocol version.

A requested tool call in a model response does not show that the agent ran the tool. Correlate it with the actual tool request. To turn these signals into metrics and key performance indicators (KPIs), refer to Operational metrics.

Telemetry pipeline components

For where these components sit in a full deployment, refer to the Agentic Runtime Fabric architecture diagram.

The reference deployment sends Envoy spans to the collector using OpenTelemetry Protocol (OTLP) over gRPC on port 4317. It sends processor spans using OTLP over HTTP at /v1/traces on port 4318.

The collector exports spans to Tempo and can send a copy to the span relay. The relay queues trace IDs, not payloads, on a BullMQ queue. Analytics workers use the IDs to retrieve the trace from storage.

Limitations

• Coverage. Proxies observe only traffic that passes through them with readable HTTP contents. They do not observe TLS passed through as opaque traffic, calls that bypass the mesh, local tool execution, or operations inside an external provider.

• Protocol support. The processor must understand the protocol. Unsupported or opaque payloads produce network spans only.

• Body size. Buffered body processing adds work to the request path and imposes size limits. Envoy can reject oversized requests before the processor receives them, and fail-open external processing does not bypass Envoy's buffer limits. Align listener buffers, processor gRPC limits, exporter capacity, and storage limits.

• Streaming. Buffered JSON observation does not guarantee incremental streaming support. Validate streaming behavior separately.

• Incomplete evidence. Sampling, truncation, missing context, and delivery loss produce incomplete traces. The reference parsers mark truncated attributes. Do not use truncated arguments or results as complete comparison inputs.

• Double counting. One logical call can produce network, payload, and external-processing transport spans, and a gateway can observe the same model exchange on both sides. Do not count every span as a distinct call or billable event. External processing observes the logical exchange, not each Envoy retry.

• Delivery. A collector acknowledgement does not prove that every destination persisted the span. Monitor processor stream counts, skipped spans, export failures, dropped spans, collector queue failures, and backend availability.

• Scope of evidence. Traces do not reveal private model reasoning, prove task completion, establish model quality, or replace application authorization. Trace storage is not a complete audit ledger or a billing system.

  Traces do not reveal private model reasoning, prove task completion, establish model quality, or replace application authorization. Trace storage is not a complete audit ledger or a billing system.

Related documentation

• Configure auto-instrumentation
• AI observability
• Cycle detection
• AI Inference Gateway
• MCP sidecar
• Consul Agentic Runtime Fabric