Kubernetes service mesh workload scenarios

This page provides example workflows for registering workloads on Kubernetes into Consul's service mesh in different scenarios, including multiport deployments. Each scenario provides an example Kubernetes manifest to demonstrate how to use Consul's service mesh with a specific Kubernetes workload type.

Kubernetes Pods running as a deployment

The following example shows a Kubernetes configuration that specifically enables service mesh connections for the static-server service. Consul starts and registers a sidecar proxy that listens on port 20000 by default and proxies valid inbound connections to port 8080.

apiVersion: v1
kind: Service
metadata:
  # This name will be the service name in Consul.
  name: static-server
spec:
  selector:
    app: static-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-server
  template:
    metadata:
      name: static-server
      labels:
        app: static-server
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
    spec:
      containers:
        - name: static-server
          image: hashicorp/http-echo:latest
          args:
            - -text="hello world"
            - -listen=:8080
          ports:
            - containerPort: 8080
              name: http
      # If ACLs are enabled, the serviceAccountName must match the Consul service name.
      serviceAccountName: static-server

To establish a connection to the upstream Pod using service mesh, a client must dial the upstream workload using a mesh proxy. The client mesh proxy will use Consul service discovery to find all available upstream proxies and their public ports.

Connecting to mesh-enabled Services

The example Deployment specification below configures a Deployment that is capable of establishing connections to our previous example "static-server" service. The connection to this static text service happens over an authorized and encrypted connection via service mesh.

apiVersion: v1
kind: Service
metadata:
  # This name will be the service name in Consul.
  name: static-client
spec:
  selector:
    app: static-client
  ports:
    - port: 80
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: static-client
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: static-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: static-client
  template:
    metadata:
      name: static-client
      labels:
        app: static-client
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
    spec:
      containers:
        - name: static-client
          image: curlimages/curl:latest
          # Just spin & wait forever, we'll use `kubectl exec` to demo
          command: ['/bin/sh', '-c', '--']
          args: ['while true; do sleep 30; done;']
      # If ACLs are enabled, the serviceAccountName must match the Consul service name.
      serviceAccountName: static-client

By default when ACLs are enabled or when ACLs default policy is allow, Consul will automatically configure proxies with all upstreams from the same datacenter. When ACLs are enabled with default deny policy, you must supply an intention to tell Consul which upstream you need to talk to.

When upstreams are specified explicitly with the consul.hashicorp.com/connect-service-upstreams annotation, the injector will also set environment variables <NAME>_CONNECT_SERVICE_HOST and <NAME>_CONNECT_SERVICE_PORT in every container in the Pod for every defined upstream. This is analogous to the standard Kubernetes service environment variables, but point instead to the correct local proxy port to establish connections via service mesh.

You cannot reference auto-generated environment variables when the upstream annotation contains a dot. This is because Consul also renders the environment variables to include a dot. For example, Consul renders the variables generated for static-server.svc:8080 as STATIC-SERVER.SVC_CONNECT_SERVICE_HOST and STATIC_SERVER.SVC_CONNECT_SERVICE_PORT, which makes the variables unusable. You can verify access to the static text server using kubectl exec. Because transparent proxy is enabled by default, use Kubernetes DNS to connect to your desired upstream.

$ kubectl exec deploy/static-client -- curl --silent http://static-server/
"hello world"

You can control access to the server using intentions. If you use the Consul UI or CLI to deny communication between "static-client" and "static-server", connections are immediately rejected without updating either of the running pods. You can then remove this intention to allow connections again.

$ kubectl exec deploy/static-client -- curl --silent http://static-server/
command terminated with exit code 52

Kubernetes Jobs

Kubernetes Jobs run pods that only make outbound requests to services on the mesh and successfully terminate when they are complete. In order to register a Kubernetes Job with the mesh, you must provide an integer value for the consul.hashicorp.com/sidecar-proxy-lifecycle-shutdown-grace-period-seconds annotation. Then, issue a request to the http://127.0.0.1:20600/graceful_shutdown API endpoint so that Kubernetes gracefully shuts down the consul-dataplane sidecar after the job is complete.

Below is an example Kubernetes manifest that deploys a job correctly.

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: test-job
  namespace: default
---
apiVersion: v1
kind: Service
metadata:
  name: test-job
  namespace: default
spec:
  selector:
    app: test-job
  ports:
    - port: 80
---
apiVersion: batch/v1
kind: Job
metadata:
   name: test-job
   namespace: default
   labels:
     app: test-job
spec:
  template:
    metadata:
      annotations:
        'consul.hashicorp.com/connect-inject': 'true'
        'consul.hashicorp.com/sidecar-proxy-lifecycle-shutdown-grace-period-seconds': '5'
      labels:
        app: test-job
    spec:
      containers:
      - name: test-job
        image: alpine/curl:3.14
        ports:
        - containerPort: 80
        command:
          - /bin/sh
          - -c
          - |
            echo "Started test job"
            sleep 10
            echo "Killing proxy"
            curl --max-time 2 -s -f -X POST http://127.0.0.1:20600/graceful_shutdown
            sleep 10
            echo "Ended test job"
      serviceAccountName: test-job
      restartPolicy: Never

Upon completing the job you should be able to verify that all containers are shut down within the pod.

$ kubectl get pods
NAME             READY   STATUS      RESTARTS   AGE
test-job-49st7   0/2     Completed   0          3m55s

$ kubectl get job
NAME       COMPLETIONS   DURATION   AGE
test-job   1/1           30s        4m31s

In addition, based on the logs emitted by the pod you can verify that the proxy was shut down before the Job completed.

$ kubectl logs test-job-49st7 -c test-job
Started test job
Killing proxy
Ended test job

Kubernetes Pods with multiple ports

To add a Pod with multiple ports to the service mesh, create a single Kubernetes Service that exposes the named ports and register it with one service account. Consul registers the Pod as a single multi-port service that routes mesh traffic to each named port. Multi-port service registration requires Consul Enterprise.

For step-by-step instructions, example manifests, and connection examples with transparent proxy enabled or disabled, refer to Configure multi-port services on Kubernetes. For general multi-port concepts and limitations that apply across all platforms, refer to multi-port services.