Define API gateway routes on Kubernetes

This topic describes how to configure HTTP and TCP routes and attach them to Consul API gateway listeners in Kubernetes-orchestrated networks. Routes are rule-based configurations that allow external clients to send requests to services in the mesh. For information

Overview

The following steps describe the general workflow for defining and deploying routes:

Define a route configuration that specifies the protocol type, name of the gateway to attach to, and rules for routing requests.
Deploy the configuration to create the routes and attach them to the gateway.

Routes and the gateways they are attached to are eventually-consistent objects. They provide feedback about their current state through a series of status conditions. As a result, you must manually check the route status to determine if the route successfully bound to the gateway.

Requirements

Verify that your environment meets the requirements specified in Technical specifications for Kubernetes.

OpenShift

If your Kubernetes-orchestrated network runs on OpenShift, verify that OpenShift is enabled for your Consul installation. Refer to OpenShift requirements for additional information.

Define routes

Define route configurations and bind them to listeners configured on the gateway so that Consul can route incoming requests to services in the mesh.

Create a configuration file and specify the following fields:

apiVersion: Specifies the Kubernetes API gateway version. This must be set to gateway.networking.k8s.io/v1 for Standard API gateway or consul.hashicorp.com/v1beta1 for Consul API gateway. For more details on API version refer to Consul API gateway reference objects.
kind: Set to HTTPRoute or TCPRoute.
metadata.name: Specify a name for the route. The name is metadata that you can use to reference the configuration when performing Consul operations.
spec.parentRefs.name: Specifies a list of API gateways that the route binds to.
spec. rules: Specifies a list of routing rules for constructing a routing table that maps listeners to services.
Refer to the Routes configuration reference for details about configuring route rules, and review the Enable API gateway for Kubernetes guide for guidance on selecting the right gateway type.

Configure any additional fields necessary for your use case, such as the namespace or admin partition.
Save the configuration.

The following example creates a route named example-route associated with a listener defined in example-gateway.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: example-route
spec:
  parentRefs:
  - name: example-gateway
  rules:
  - backendRefs:
    - kind: Service
      name: echo
      port: 8080

Route to service subsets

When an HTTPRoute sends traffic to a single backend service, Consul API gateway also evaluates the backend service's discovery chain. When the backend service has a ServiceRouter that routes matching requests to a ServiceResolver subset, the API gateway preserves that subset routing when it builds the gateway discovery chain.

You can expose an HTTP route at the gateway while using Consul service mesh traffic management, such as routing a subset of requests to a canary service version. Consul applies the HTTPRoute match and the ServiceRouter match. For example, a request must match the gateway route and the service-router rule before Consul routes it to the subset destination.

The following example routes requests with the x-version: canary header to the canary subset of the api service. All other requests that match the route use the stable default subset.

apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceResolver
metadata:
  name: api
spec:
  defaultSubset: stable
  subsets:
    stable:
      filter: Service.Meta.version == stable
    canary:
      filter: Service.Meta.version == canary
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceRouter
metadata:
  name: api
spec:
  routes:
    - match:
        http:
          header:
            - name: x-version
              exact: canary
      destination:
        service: api
        serviceSubset: canary
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: api-route
spec:
  parentRefs:
  - name: example-gateway
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /api
    backendRefs:
    - kind: Service
      name: api
      port: 8080

TLS SDS backend override for HTTP routes

In Kubernetes, backend-level SDS certificate override for HTTP routes is configured with RouteTLSSDSFilter attached under HTTPRoute.rules[].backendRefs[].filters[].

Filter fields:

spec.sds.certResource (required)
spec.sds.clusterName (optional when inherited from listener/gateway SDS)

Behavior:

If clusterName is omitted, it can be inherited when exactly one listener-level/global SDS cluster is resolved from parent references.
If inherited clusterName is missing or ambiguous, the filter is rejected.
Rule-level filter placement (HTTPRoute.rules[].filters[]) is invalid for this override type.

Example: HTTPRoute backendRef override using RouteTLSSDSFilter

apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: api-route-override
  namespace: default
spec:
  parentRefs:
  - name: sds-gateway
    sectionName: https
  hostnames:
  - b.example.test
  rules:
  - backendRefs:
    - name: svc-b
      port: 5678
      filters:
      - type: ExtensionRef
        extensionRef:
          group: consul.hashicorp.com
          kind: RouteTLSSDSFilter
          name: route-sds-override-http
---
apiVersion: consul.hashicorp.com/v1alpha1
kind: RouteTLSSDSFilter
metadata:
  name: route-sds-override-http
  namespace: default
spec:
  sds:
    clusterName: sds-cluster-2
    certResource: foo.example.com

Effective TLS SDS precedence

When configured, certificate selection precedence is:

Route service override TLS.SDS
Listener TLS.SDS
Gateway-level TLS.SDS

Troubleshooting checklist

Confirm RouteTLSSDSFilter is attached under backendRefs[].filters[].
Ensure spec.sds.certResource is non-empty.
Ensure clusterName is provided or can be inherited from listener/gateway SDS.
For hostname-based routing checks, verify hostnames and request host headers match expected values.

Upstream limits and passive health checks

You can configure upstream circuit-breaker limits and passive health checks for the backend services that an API gateway routes traffic to. Passive health checks configure Envoy outlier detection on the gateway. When an instance of a backend service returns consecutive errors, the gateway removes the instance from its load balancing pool for a period of time and sends requests to the remaining healthy instances.

Upstream limits and passive health checks require Consul v2.1 or later and Consul Helm chart v2.1.0 or later. They are supported for Gateway and HTTPRoute resources in the gateway.networking.k8s.io API group.

You can configure the settings in the following locations:

Gateway-wide defaults: api-gateway.consul.hashicorp.com/default-* annotations on the Gateway. The defaults apply to every backend service that routes attached to the gateway target. Refer to Gateway annotations for the list of annotations.
Backend-level overrides: A RouteUpstreamLimitsFilter resource attached to a backend in HTTPRoute.rules[].backendRefs[].filters[].

Values in a RouteUpstreamLimitsFilter take precedence over the gateway-wide defaults:

Each circuit-breaker limit that you set in the filter overrides the corresponding gateway default. Limits that you do not set inherit the gateway default.
A passive health check in the filter replaces the gateway default passive health check for the backend service. Consul does not merge individual passive health check fields.

Configure gateway-wide defaults

Add the default annotations to the Gateway. The following example limits each backend service to 40 connections and ejects a backend service instance after five consecutive 5xx errors:

my-gateway.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-gateway
  annotations:
    api-gateway.consul.hashicorp.com/default-max-connections: "40"
    api-gateway.consul.hashicorp.com/default-max-pending-requests: "80"
    api-gateway.consul.hashicorp.com/default-max-concurrent-requests: "160"
    api-gateway.consul.hashicorp.com/default-passive-health-check-interval: "15s"
    api-gateway.consul.hashicorp.com/default-passive-health-check-max-failures: "5"
    api-gateway.consul.hashicorp.com/default-passive-health-check-max-ejection-percent: "20"
spec:
  gatewayClassName: consul
  listeners:
  - name: http
    protocol: HTTP
    port: 80
    allowedRoutes:
      namespaces:
        from: All

Consul on Kubernetes ignores annotations that have invalid values, such as negative numbers, percentages greater than 100, or malformed durations, and logs a message in the controller logs.

Override the defaults for a backend service

Create a RouteUpstreamLimitsFilter and attach it to the backend in the HTTPRoute. The filter must be in the same namespace as the HTTPRoute. The following example overrides the defaults for the static-server backend:

limits-route.yaml

apiVersion: consul.hashicorp.com/v1alpha1
kind: RouteUpstreamLimitsFilter
metadata:
  name: static-server-limits
spec:
  maxConnections: 25
  maxPendingRequests: 50
  maxConcurrentRequests: 100
  passiveHealthCheck:
    interval: 10s
    maxFailures: 3
    enforcingConsecutive5xx: 100
    maxEjectionPercent: 50
    baseEjectionTime: 30s
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: limits-route
spec:
  parentRefs:
  - name: my-gateway
  rules:
  - backendRefs:
    - name: static-server
      port: 80
      filters:
      - type: ExtensionRef
        extensionRef:
          group: consul.hashicorp.com
          kind: RouteUpstreamLimitsFilter
          name: static-server-limits

Apply the configuration:

$ kubectl apply --filename limits-route.yaml

When you update the filter or the Gateway annotations, Consul on Kubernetes updates the Consul configuration entries and the gateway applies the new values.

Verify upstream limits and passive health checks

Complete the following steps to verify that the gateway applies the settings:

Read the Consul configuration entries to confirm that Consul on Kubernetes translated the settings. The gateway annotations appear in the Defaults field of the api-gateway configuration entry, and the filter values appear in the Limits field of the backend service in the http-route configuration entry. Run the commands from a host where the consul CLI can reach your Consul servers, and set CONSUL_HTTP_TOKEN if ACLs are enabled.
```
$ consul config read -kind api-gateway -name my-gateway
$ consul config read -kind http-route -name limits-route
```

Forward the Envoy admin port of the gateway pod to your local machine:

$ kubectl port-forward \
    $(kubectl get pods --selector gateway.consul.hashicorp.com/name=my-gateway --output jsonpath='{.items[0].metadata.name}') \
    19000:19000

In a separate terminal, query the Envoy configuration for the static-server cluster. The circuit_breakers block contains the upstream limits, and the outlier_detection block contains the passive health check settings.

$ curl --silent localhost:19000/config_dump \
    | jq '.configs[]
      | select(."@type" | contains("ClustersConfigDump"))
      | .dynamic_active_clusters[]
      | select(.cluster.name | startswith("static-server."))
      | {circuit_breakers: .cluster.circuit_breakers, outlier_detection: .cluster.outlier_detection}'

The output contains the values from the RouteUpstreamLimitsFilter, such as "max_connections": 25 and "consecutive_5xx": 3.

Negate header matches

The Kubernetes Gateway API does not provide a way to negate an HTTP header match. To route requests when a header value does not match a specific value, attach a RouteHeaderMatchInvertFilter to the HTTPRoute rule and list the names of the headers whose match conditions you want to negate. Consul on Kubernetes sets Invert to true on those header matches in the http-route configuration entry.

Negated header matches require Consul v2.1 or later and Consul Helm chart v2.1.0 or later. They are supported for HTTPRoute resources in the gateway.networking.k8s.io API group.

The following behavior applies to negated header matches:

Attach the filter in HTTPRoute.rules[].filters[]. The filter applies to header matches in all matches entries of the rule.
The filter must be in the same namespace as the HTTPRoute.
Header names in the filter are not case sensitive.
A negated Exact or RegularExpression condition matches requests that include the header when the value does not match. It does not match requests that do not include the header.

Configure a negated header match

The following example routes requests to the service-v2 service when the x-version header value is not v1:

version-route.yaml

apiVersion: consul.hashicorp.com/v1alpha1
kind: RouteHeaderMatchInvertFilter
metadata:
  name: invert-version
spec:
  headerNames:
    - x-version
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: version-route
spec:
  parentRefs:
  - name: my-gateway
  rules:
  - matches:
    - headers:
      - type: Exact
        name: x-version
        value: v1
    filters:
    - type: ExtensionRef
      extensionRef:
        group: consul.hashicorp.com
        kind: RouteHeaderMatchInvertFilter
        name: invert-version
    backendRefs:
    - name: service-v2
      port: 80

Apply the configuration:

$ kubectl apply --filename version-route.yaml

When you change the header names in the filter, reference a different filter, or remove the filter from the rule, Consul on Kubernetes updates the http-route configuration entry. When you remove the filter, Consul on Kubernetes sets Invert to false on the header matches.

Verify negated header matches

Complete the following steps to verify that the gateway applies the negated header match:

Read the http-route configuration entry and confirm that the header match includes "Invert": true. Run the command from a host where the consul CLI can reach your Consul servers, and set CONSUL_HTTP_TOKEN if ACLs are enabled.
```
$ consul config read -kind http-route -name version-route
```
The header match in the output is similar to the following example:
```
{
  "Match": "exact",
  "Name": "x-version",
  "Value": "v1",
  "Invert": true
}
```

Forward the Envoy admin port of the gateway pod to your local machine:

$ kubectl port-forward \
    $(kubectl get pods --selector gateway.consul.hashicorp.com/name=my-gateway --output jsonpath='{.items[0].metadata.name}') \
    19000:19000

In a separate terminal, query the Envoy route configuration. The header matcher for x-version includes "invert_match": true.

$ curl --silent localhost:19000/config_dump \
    | jq '.configs[]
      | select(."@type" | contains("RoutesConfigDump"))
      | .dynamic_route_configs[].route_config.virtual_hosts[].routes[].match.headers'

Deploy the route configuration

Apply the configuration to your cluster using the kubectl command. The following command applies the configuration to the consul namespace:

$ kubectl apply -f my-route.yaml -n consul