AI Inference Gateway
The AI Inference Gateway governs agent access to large language models (LLMs) in Consul Agentic Runtime Fabric. It provides a supported inference interface between applications and model providers, with policy controls for routing, usage, and sensitive data.
Agents reach the gateway over their Consul service mesh path. The gateway forwards each request to a model registered in the Consul catalog. To learn how model, tool, and identity paths fit together, refer to the Consul Agentic Runtime Fabric architecture.
Enterprise
This feature requires Consul Enterprise. Refer to the Enterprise feature matrix for additional information.
Introduction
Applications send inference requests to a gateway instead of integrating network policy separately for each provider. The gateway accepts an OpenAI-compatible request format and translates supported operations to the selected provider's format. Interface compatibility does not imply support for every provider feature or equivalent behavior between models.
Consul supplies gateway configuration and eligible backend information. Envoy enforces mesh connectivity policy and performs routing and retries. The external processor validates and transforms requests and responses. It does not independently select routes or run a retry loop.
Request processing
The inference path separates processing before and after Envoy selects an upstream:
| Phase | Responsibility |
|---|---|
| Before routing | Resolve caller workload identity, validate the canonical request, and apply configured request-side data protection. Provide request information used by configured routing policy. |
| After routing | Translate the request for the selected provider and rewrite the provider request path. |
| Provider response | Normalize supported responses, process streamed output where supported, apply configured response-side data protection, and capture usage. |
The upstream processor uses information about the backend that Envoy selected. A retry against another backend must use that backend's adapter and model information rather than reuse an incompatible translated request.
Model selection and fallback
Model selection has two distinct inputs: the caller's request and the operator's policy. The architecture requires the gateway to preserve an explicit model choice when policy permits it. Policy can reject a disallowed request. It must not silently substitute an unrelated model.
Caller-approved fallback constrains which alternatives are acceptable. A routing policy's ability to reach another backend is not, by itself, permission to change the requested model. Treat retrying a request and changing models as separate decisions.
For requests that delegate model selection to the gateway, eligible providers and routing policy determine the selection. Provider translation changes the wire format. It does not establish that two models have the same capabilities, quality, or data-handling terms.
The requested model and the model that serves the request are separate attribution values. The architecture requires actual-model attribution to be available to authorized users, including when an approved fallback occurs.
Usage controls
Token-aware usage controls govern admission and account for completed work. Admission can depend on estimated consumption because final provider usage is unavailable before a request completes. Usage accounting must distinguish estimates from reported consumption, including incomplete streams and retries.
Keep rate-limit decisions, observed token usage, and financial accounting distinct. A token count is not a price, and gateway telemetry is not a billing ledger. Cost analysis requires a maintained pricing source and a defined policy for missing or estimated usage.
Data protection and streaming
Personally identifiable information (PII) processing applies the configured detection and handling rules to request or response content. Request-side controls act before data reaches a provider. Response-side controls act before processed content returns to the caller.
Streaming introduces an additional boundary: content already delivered to the caller cannot be recalled. Validate detector coverage, buffering behavior, and supported stream formats for your deployment. Do not interpret configured PII controls as a guarantee that all sensitive content is detected.
Telemetry records decisions and bounded metadata, not prompts, responses, detected PII values, credentials, or provider error text that can contain request content.
Dependencies and boundaries
- Mesh identity and policy establish the workload's permitted network path to the gateway and its backends.
- Provider adapters define supported request formats, response formats, streaming behavior, and translation limits.
- Routing rules define eligible backends. Application consent and operator policy constrain model substitution.
- Usage consumers require consistent attribution across admission, provider usage, and completion evidence.
- Managed-model authentication is separate from delegated user identity and external credential injection.
The inference component owns adapter behavior, streaming, routing-rule integration, accounting, and PII processing. It does not own tool business logic or determine whether a model response satisfies the user's task.