AI observability and KPIs
Consul Agentic Runtime Fabric observability connects model access, tool calls, policy decisions, and operational outcomes. Metrics, traces, and audit evidence serve different purposes, while higher-order key performance indicators (KPIs) also require application-provided outcomes.
The inference gateway implementation provides metrics and tracing. The compliance audit trail and enforced audit-delivery mode described by the architecture are separate design capabilities, not inference-gateway configuration documented here. Higher-order KPIs require additional application and analysis integrations.
Introduction
The architecture separates three independently configurable signal types:
| Signal | Purpose | Boundary |
|---|---|---|
| Metrics | Aggregate traffic, latency, errors, usage, and policy outcomes. | Aggregation and delivery loss make metrics unsuitable as a billing ledger. |
| Traces | Connect request processing and application workflow activity. | Sampling and incomplete instrumentation limit what a trace can establish. |
| Audit evidence | Describe attributable governance decisions for investigation. | Durable, complete, tamper-evident retention requires delivery and storage controls beyond ordinary telemetry. |
Each component emits the events it can observe. An inference gateway cannot supply tool-business outcomes or agent workflow completion merely because those operations belong to the same application.
Correlation and model attribution
Request and trace identifiers connect observations across processing stages. Inference processing before and after routing must remain correlatable even though it uses separate external-processing streams.
A request identifier is a join key, not a workload identity or authorization credential. Validate untrusted correlation values and keep record identity separate from a caller-controlled identifier.
Requested-model and served-model attribution answer different questions. Preserve both where available, and correlate approved fallback and retry evidence with the relevant attempt. The architecture requires actual-model attribution to be available to authorized users. A client-facing interface alone does not establish which model served a request.
Metrics and traces
Inference metrics describe request outcomes, stage duration, token usage, streaming behavior, identity resolution, and configured PII processing. Personally identifiable information (PII) signals identify detector types and actions, not detected values.
Metrics use bounded dimensions. Do not promote request identifiers, trace identifiers, user identifiers, full workload identities, arbitrary paths, or request content into metric labels. Unbounded labels increase memory use and can expose sensitive information.
Tracing describes the gateway processing path and links it to application spans where applications propagate compatible context. Gateway spans do not automatically instrument agent reasoning, tool business logic, or task outcomes.
The inference implementation exports metrics through Prometheus and optionally through OpenTelemetry Protocol (OTLP). Tracing requires explicit enablement and an OTLP destination. These exporters are best-effort: a telemetry-sink failure must not change the inference request outcome.
Proposed audit delivery
The audit design separates admission evidence from completion evidence. A denied request needs a decision record even though it never reaches a provider. An admitted request can also need a completion record containing outcome and usage attribution.
The design calls for explicit loss accounting and correlation across these records. Sampling, buffer exhaustion, process termination, and sink failures can create gaps. A local buffer or standard-output stream does not by itself provide durable or tamper-evident storage.
The proposed enforced-delivery mode is an explicit exception to best-effort emission: admission can depend on the configured audit-delivery guarantee. Do not infer this guarantee from ordinary metrics, tracing, or logs, and do not assume that the mode exists without component support.
Audit retention, access control, integrity, and downstream delivery remain operational responsibilities. Audit evidence can support compliance work. Its presence does not establish certification or regulatory compliance.
Higher-order KPIs
Higher-order KPIs combine network observations with application semantics. Define the population, outcome criteria, time window, and treatment of incomplete workflows before interpreting a result.
| KPI | Required inputs | Interpretation limit |
|---|---|---|
| Task completion rate | Application-defined task starts, terminal outcomes, and correlation across calls. | Successful HTTP requests or model responses do not prove task completion. |
| Tool success rate | Tool protocol outcomes and application-defined business results. | A protocol success can still represent an unsuccessful business operation. |
| Cost per successful task | Attributed usage, maintained provider pricing, and confirmed task outcomes. | Missing usage, retries, pricing changes, and incomplete tasks affect the result. |
| Autonomous versus escalated outcomes | Application approval events and final task outcomes. | Avoid treating every approval request or timeout as a completed task. |
| Model quality or routing effectiveness | Application evaluations and confirmed served-model attribution. | Latency, token counts, and availability are not quality measurements. |
These are analysis requirements, not a claim that the gateway emits each KPI directly. Keep operational cost estimates separate from an authoritative invoice or financial ledger.
Privacy and content boundaries
Keep prompts, responses, tool arguments, tool results, matched PII, credentials, and tokens out of telemetry and audit records. Provider errors can contain request data, so prefer bounded error categories over raw error messages.
Correlated metadata can still be sensitive. Apply access controls and retention limits to identity, model usage, policy decisions, and application outcomes even when content capture is disabled.
Dependencies and boundaries
The observability component owns signal definitions, privacy rules, delivery modes, and KPI prerequisites. Applications supply workflow outcomes and the instrumentation needed for semantic analysis.