Save 10-15% Register for HashiConf and save big when you buy 2+ tickets Get your passes
Vault
Deploy Vault

Retrieve secrets for Kubernetes workloads with Vault Agent

  • 13min
  • |
  • Vault

Most requests to Vault require an authentication token. This includes all API requests, Vault CLI commands, and other libraries. If you can securely get the first secret from an originator to a consumer, later secrets transmitted between the originator and consumer authenticate with the trust established by the successful distribution and user of the first secret.

Secure Introduction

Challenge

The applications running in a Kubernetes environment is no exception. Vault provides Kubernetes auth method to authenticate the clients using a Kubernetes Service Account Token.

Kubernetes

However, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the next challenge becomes how to manage the lifecycle of tokens in a standard way without having to write custom logic.

Solution

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

  • Automatic authentication
  • Secure delivery/storage of tokens
  • Lifecycle management of these tokens (renewal & re-authentication)
Vault Agent

Note

Though this tutorial focuses on the demonstration of the Vault Agent Auto-Auth using the kubernetes auth method for application integration, there are two other options. The other options are the Vault Secrets Operator and the Agent Sidecar Injector.

Refer to Vault Secrets Operator and Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar for a step-by-step tutorial on these options.

Prerequisites

To perform the tasks described in this tutorial, you need to have:

Install supporting tools

This tutorial was last tested 13 November 2023 on a macOS 13.5.2 using the following software versions.

$ docker version
Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.6
...

$ kubectl version --short
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
...

$ minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

  1. Retrieve the configuration by cloning the hashicorp-education/learn-vault-agent repository from GitHub.

    $ git clone https://github.com/hashicorp-education/learn-vault-agent

    This repository has supporting content for the Vault tutorials. The content specific to this tutorial is in a sub-directory.

  2. Go into the learn-vault-agent/vault-agent-k8s-demo directory.

    $ cd learn-vault-agent/vault-agent-k8s-demo

Working directory

Run the remainder of commands in the tutorial from this directory.

Start a Vault server

  1. To go through this tutorial, start a Vault dev server which listens for requests locally at 0.0.0.0:8200 with root as the root token ID.

    $ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

    Setting the -dev-listen-address to 0.0.0.0:8200 overrides the default address of a Vault dev server (127.0.0.1:8200) and enables Vault to be addressable by the Kubernetes cluster and its pods because it binds to a shared network.

    Insecure operation

    Do not run a Vault dev server in production. This approach simplifies the unsealing process for this tutorial.

  2. Export an environment variable for the vault CLI to connect to the Vault server.

    $ export VAULT_ADDR=http://0.0.0.0:8200

Create a service account

  1. Start a Kubernetes cluster running in minikube.

    $ minikube start --driver=docker

  2. Wait a couple of minutes for the minikube environment to become fully available.

    $ minikube status

minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

  3. In Kubernetes, a service account provides an identity for processes that run in a Pod so that the processes can contact the API server. Open the provided vault-auth-service-account.yaml file in your preferred text editor and examine its content used for the service account definition in this tutorial.

    vault-auth-service-account.yaml

    apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default

  4. Create the vault-auth service account.

    $ kubectl apply --filename vault-auth-service-account.yaml

Kubernetes 1.24+

  1. In Kubernetes 1.24+, you need to create the service account secret.

    vault-auth-secret.yaml

    apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-secret
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token

  2. Create a vault-auth-secret secret.

    $ kubectl apply --filename vault-auth-secret.yaml

Configure Kubernetes auth method

  1. Create the myapp-kv-ro Vault policy.

    $ vault policy write myapp-kv-ro - <<EOF
path "secret/data/myapp/*" {
    capabilities = ["read", "list"]
}
EOF

  2. Create some test data at the secret/myapp path.

    $ vault kv put secret/myapp/config \
      username='appuser' \
      password='suP3rsec(et!' \
      ttl='30s'

    Output:

    ====== Secret Path ======
secret/data/myapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2022-03-24T06:09:49.99472Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

  3. Set the environment variables to point to the running minikube environment.

    Set the SA_SECRET_NAME environment variable value to the vault-auth service account secret.

    $ export SA_SECRET_NAME=$(kubectl get secrets --output=json \
    | jq -r '.items[].metadata | select(.name|startswith("vault-auth-")).name')

    Set the SA_JWT_TOKEN environment variable value to the service account JWT used to access the TokenReview API

    $ export SA_JWT_TOKEN=$(kubectl get secret $SA_SECRET_NAME \
    --output 'go-template={{ .data.token }}' | base64 --decode)

    Set the SA_CA_CRT environment variable value to the PEM encoded CA cert used to talk to Kubernetes API.

    $ export SA_CA_CRT=$(kubectl config view --raw --minify --flatten --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode | awk '{printf "%s\\n", $0}')

    Set the K8S_HOST environment variable value to minikube IP address.

    $ export K8S_HOST=$(kubectl config view --raw --minify --flatten \
    --output 'jsonpath={.clusters[].cluster.server}')

  4. Now, enable and configure the Kubernetes auth method.

    Enable the Kubernetes auth method at the default path ("auth/kubernetes").

    $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

    Tell Vault how to communicate with the Kubernetes (minikube) cluster.

    $ vault write auth/kubernetes/config \
     token_reviewer_jwt="$SA_JWT_TOKEN" \
     kubernetes_host="$K8S_HOST" \
     kubernetes_ca_cert="$SA_CA_CRT" \
     issuer="https://kubernetes.default.svc.cluster.local"

    Output:

    Success! Data written to: auth/kubernetes/config

    You can validate the issuer name of your Kubernetes cluster found under Discovering the service account issuer.

  5. Create a role named, example, that maps the Kubernetes Service Account to Vault policies and default token TTL.

    $ vault write auth/kubernetes/role/example \
     bound_service_account_names=vault-auth \
     bound_service_account_namespaces=default \
     token_policies=myapp-kv-ro \
     ttl=24h

    Output:

    Success! Data written to: auth/kubernetes/role/example

Note

The pattern Vault uses to authenticate Pods depends on sharing the JWT token over the network. Given the security model of Vault, this is allowable because Vault is part of the trusted compute base. In general, Kubernetes applications should not share this JWT with other applications. It allows API calls on behalf of the Pod and can result in unintended access from 3rd parties.

Determine the Vault address

A service bound to all networks on the host, as you configured Vault, is addressable by pods within minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.

  1. Start a minikube SSH session.

    $ minikube ssh
## ... minikube ssh login

  2. Within this SSH session, retrieve the value of the minikube host.

    $ dig +short host.docker.internal
192.168.65.2

    Docker networking

    The host has a changing IP address (or none if you have no network access). Connect to the special DNS name host.docker.internal which resolves to the internal IP address used by the host. This is for development purpose and will not work in production. For more information, review the documentation for Mac, Windows.

  3. Next, retrieve the status of the Vault server to verify network connectivity.

    $ dig +short host.docker.internal | xargs -I{} curl -s http://{}:8200/v1/sys/seal-status | python3 -m json.tool
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.4.1+ent",
  "migration": false,
  "cluster_name": "vault-cluster-3de6c2d3",
  "cluster_id": "10fd177e-d55a-d740-0c54-26268ed86e31",
  "recovery_seal": false,
  "storage_type": "inmem"
}

    The output displays that the Vault server initialization status of false. This confirms that pods within your cluster are able to reach Vault given that each pod uses the gateway address.

  4. Exit the minikube SSH session.

    $ exit

  5. Create a variable named EXTERNAL_VAULT_ADDR to capture the minikube gateway address.

    $ EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')

  6. Verify that the variable has the IP address you saw when executed in the minikube shell.

    $ echo $EXTERNAL_VAULT_ADDR
192.168.65.2

Optional: Verify the Kubernetes auth method configuration

  1. Define a Pod with a container.

    $ cat > devwebapp.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: devwebapp
  labels:
    app: devwebapp
spec:
  serviceAccountName: vault-auth
  containers:
    - name: devwebapp
      image: burtlo/devwebapp-ruby:k8s
      env:
        - name: VAULT_ADDR
          value: "http://$EXTERNAL_VAULT_ADDR:8200"
EOF

    The Pod devwebapp runs with the vault-auth service account.

  2. Create the devwebapp pod in the default Kubernetes namespace.

    $ kubectl apply --filename devwebapp.yaml --namespace default
pod/devwebapp created

  3. Display all the pods in the default namespace. Wait until the devwebapp pod is running and ready (1/1).

    $ kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
devwebapp   1/1     Running   0          77s

  4. Start an interactive shell session on the devwebapp pod.

    $ kubectl exec --stdin=true --tty=true devwebapp -- /bin/sh
#

    The command connects to the devwebapp container.

  5. Set KUBE_TOKEN to the service account token.

    $ KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

  6. Authenticate with Vault through the example role with the KUBE_TOKEN.

    $ curl --request POST \
       --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \
       $VAULT_ADDR/v1/auth/kubernetes/login | python3 -m json.tool

    Example output:

    {
   ...snip...
   "auth": {
      "client_token": "hvs.CAESIBxfvzds7M4tas017ls36Dl_kA-3YpCCBT9wczP1E41DGh4KHGh2cy5JYW1NMmFkb2gwTVBhZVhsN0pDM0tOaWM",
      "accessor": "1Miu5tbfuZzYuYWCntb4Ztke",
      "policies": [
         "default",
         "myapp-kv-ro"
      ],
      "token_policies": [
         "default",
         "myapp-kv-ro"
      ],
      "metadata": {
         "role": "example",
         "service_account_name": "vault-auth",
         "service_account_namespace": "default",
         "service_account_secret_name": "",
         "service_account_uid": "a293b5fb-96a2-43b0-acee-03d9ffce9423"
      },
      "lease_duration": 86400,
      "renewable": true,
      "entity_id": "8925670b-d8d6-0792-ee56-eec1a9e740cb",
      "token_type": "service",
      "orphan": true,
      "mfa_requirement": null,
      "num_uses": 0
   }
}

    The request generates the client_token with myapp-kv-ro policy attached to the token. The metadata displays that its service account name (service_account_name) is vault-auth.

  7. Exit the pod.

    $ exit

Start Vault Agent with Auto-Auth

Now that you have verified the Kubernetes auth method configuration, start a client Pod which leverages Vault Agent to automatically authenticate with Vault and retrieve a client token.

  1. First, open the provided configmap.yaml file in your preferred text editor and review its content.

    configmap.yaml

    apiVersion: v1
data:
  vault-agent-config.hcl: |
    # Comment this out if running as sidecar instead of initContainer
    exit_after_auth = true

    pid_file = "/home/vault/pidfile"

    auto_auth {
        method "kubernetes" {
            mount_path = "auth/kubernetes"
            config = {
                role = "example"
            }
        }

        sink "file" {
            config = {
                path = "/home/vault/.vault-token"
            }
        }
    }

    template {
    destination = "/etc/secrets/index.html"
    contents = <<EOT
    <html>
    <body>
    <p>Some secrets:</p>
    {{- with secret "secret/data/myapp/config" }}
    <ul>
    <li><pre>username: {{ .Data.data.username }}</pre></li>
    <li><pre>password: {{ .Data.data.password }}</pre></li>
    </ul>
    {{ end }}
    </body>
    </html>
    EOT
    }
kind: ConfigMap
metadata:
  name: example-vault-agent-config
  namespace: default

    This creates a Vault Agent configuration file, vault-agent-config.hcl. The Vault Agent auto_auth block uses the kubernetes auth method enabled at the auth/kubernetes path. The Vault Agent will use the example role which you created in Configure Kubernetes auth method.

    The sink block specifies the location on disk where to write tokens. You can configure different Vault Agent Auto-Auth sink blocks if you want Vault Agent to place the token into multiple locations. In this example, set the sink to /home/vault/.vault-token.

    The template block creates a file which retrieves username and password values at the secret/data/myapp/config path.

  2. Create a ConfigMap containing a Vault Agent configuration.

    $ kubectl create --filename configmap.yaml
configmap/example-vault-agent-config created

  3. View the created ConfigMap.

    $ kubectl get configmap example-vault-agent-config --output yaml

  4. Review the provided example Pod spec file, example-k8s-spec.yaml.

    example-k8s-spec.yaml

    apiVersion: v1
kind: Pod
metadata:
  name: vault-agent-example
  namespace: default
spec:
  serviceAccountName: vault-auth

  volumes:
    - configMap:
        items:
          - key: vault-agent-config.hcl
            path: vault-agent-config.hcl
        name: example-vault-agent-config
      name: config
    - emptyDir: {}
      name: shared-data

  initContainers:
    - args:
        - agent
        - -config=/etc/vault/vault-agent-config.hcl
        - -log-level=debug
      env:
        - name: VAULT_ADDR
          value: http://EXTERNAL_VAULT_ADDR:8200
      image: vault
      name: vault-agent
      volumeMounts:
        - mountPath: /etc/vault
          name: config
        - mountPath: /etc/secrets
          name: shared-data

  containers:
    - image: nginx
      name: nginx-container
      ports:
        - containerPort: 80
      volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: shared-data

    The example Pod spec (example-k8s-spec.yaml) spins up two containers in vault-agent-example pod. A Vault container which runs Vault Agent as an Init Container and an nginx container exposing port 80.

    Vault Agent on Kubernetes workflow

    The Vault address environment variable, VAULT_ADDR, uses a placeholder value EXTERNAL_VAULT_ADDR.

  5. Generate the Pod spec with EXTERNAL_VAULT_ADDR variable value in its place.

    $ cat example-k8s-spec.yaml | \
   sed -e s/"EXTERNAL_VAULT_ADDR"/"$EXTERNAL_VAULT_ADDR"/ \
   > vault-agent-example.yaml

  6. Create the vault-agent-example pod defined in vault-agent-example.yaml.

    $ kubectl apply --filename vault-agent-example.yaml --record

    This takes a minute or so for the pod to become fully up and running.

Verification

  1. In another terminal, launch the minikube dashboard.

    $ minikube dashboard

  2. Under Workloads click Pods to verify that vault-agent-example pod has is running.

    Kubernetes

  3. Select vault-agent-example to see its details.

    Kubernetes

  4. In another terminal, port forward all requests made to http://localhost:8080 to port 80 on the vault-agent-example pod.

    $ kubectl port-forward pod/vault-agent-example 8080:80

  5. In a web browser, go to localhost:8080

    Kubernetes

    The page displays the values for username and password from secret/myapp/config.

  6. View the HTML source.

    $ kubectl exec -it vault-agent-example --container nginx-container -- cat /usr/share/nginx/html/index.html
<html>
<body>
<p>Some secrets:</p>
<ul>
<li><pre>username: appuser</pre></li>
<li><pre>password: suP3rsec(et!</pre></li>
</ul>

</body>
</html>

Clean up

  1. Stop the cluster.

    $ minikube stop
✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🛑  1 node stopped.

  2. Delete the cluster.

    $ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "minikube" ...
🔥  Removing /Users/mrken/.minikube/machines/minikube ...
💀  Removed all traces of the "minikube" cluster

  3. Delete the supporting code repository copied locally.

    $ cd ../.. && rm -rf learn-vault-agent

  4. Stop the Vault server process.

    $ pkill vault

Help and reference

Other Vault Agent tutorials:

Previous

Mount Vault secrets through CSI Volume

 Next Collection

Deploy Vault on the cloud

This tutorial also appears in:

  • 7 tutorials
    Vault Agent
    Use Vault Agent to authenticate and read secrets from Vault with little to no change in your application code.
    • Vault