HashiConf Last chance to register for our cloud conference October 10-12. Register today
Vault
Vault Agent

Vault Agent with Kubernetes

  • 17min
  • |
  • Vault

Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret.

Secure Introduction

Challenge

The applications running in a Kubernetes environment is no exception. Luckily, Vault provides Kubernetes auth method to authenticate the clients using a Kubernetes Service Account Token.

Kubernetes

However, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the next challenge becomes how to manage the lifecycle of tokens in a standard way without having to write custom logic.

Solution

Vault Agent provides a number of different helper features, specifically addressing the following challenges:

  • Automatic authentication
  • Secure delivery/storage of tokens
  • Lifecycle management of these tokens (renewal & re-authentication)
Vault Agent

Note

This tutorial focuses on the demonstration of the Vault Agent Auto-Auth using the kubernetes auth method. Vault Helm introduced Agent Sidecar Injector. Refer to Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar for a step-by-step tutorial on the sidecar usage.

Prerequisites

This lab was tested on macOS using an x86_64 based processor. If you are running macOS on an Apple silicon-based processor, use a x86_64 based Linux virtual machine in your preferred cloud provider.

To perform the tasks described in this tutorial, you need:

Note

For the purpose of demonstration, this tutorial runs Minikube as a Kubernetes environment. If you wish to test against an Azure Kubernetes Service (AKS) cluster instead, follow the steps in the Azure Kubernetes Service Cluster section to create an AKS cluster. If you want to test against a Google Kubernetes Engine (GKE) cluster, refer to the steps described in the Google Kubernetes Engine Cluster section.

Retrieve the additional configuration by cloning the hashicorp/learn-vault-agent repository from GitHub.

$ git clone https://github.com/hashicorp/learn-vault-agent.git

This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found in a sub-directory.

Go into the learn-vault-agent/vault-agent-k8s-demo directory.

$ cd learn-vault-agent/vault-agent-k8s-demo

Working directory

This tutorial assumes that the remainder of commands are executed in this directory.

Start a Vault server

To go through this tutorial, start a Vault dev server which listens for requests locally at 0.0.0.0:8200 with root as the root token ID.

$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200

Setting the -dev-listen-address to 0.0.0.0:8200 overrides the default address of a Vault dev server (127.0.0.1:8200) and enables Vault to be addressable by the Kubernetes cluster and its pods because it binds to a shared network.

Insecure operation

Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://0.0.0.0:8200

Create a service account

  1. Start a Kubernetes cluster running in Minikube.

    $ minikube start --driver=docker

    Wait a couple of minutes for the minikube environment to become fully available.

    $ minikube status

minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

  2. In Kubernetes, a service account provides an identity for processes that run in a Pod so that the processes can contact the API server. Open the provided vault-auth-service-account.yaml file in your preferred text editor and examine its content for the service account definition to be used for this tutorial.

    vault-auth-service-account.yaml
    apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: vault-auth
  namespace: default

  3. Create the vault-auth service account.

    $ kubectl apply --filename vault-auth-service-account.yaml

Kubernetes 1.24+ only

  1. The service account generated a secret that is required for configuration automatically in Kubernetes 1.23. In Kubernetes 1.24+, you need to create the secret explicitly.

    vault-auth-secret.yaml
    apiVersion: v1
kind: Secret
metadata:
  name: vault-auth-secret
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token

  2. Create a vault-auth-secret secret.

    $ kubectl apply --filename vault-auth-secret.yaml

Configure Kubernetes auth method

  1. Create a read-only policy, myapp-kv-ro in Vault.

    $ vault policy write myapp-kv-ro - <<EOF
path "secret/data/myapp/*" {
    capabilities = ["read", "list"]
}
EOF

  2. Create some test data at the secret/myapp path.

    $ vault kv put secret/myapp/config \
      username='appuser' \
      password='suP3rsec(et!' \
      ttl='30s'

    Output:

    ====== Secret Path ======
secret/data/myapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2022-03-24T06:09:49.99472Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

  3. Set the environment variables to point to the running Minikube environment.

    Set the SA_SECRET_NAME environment variable value to the vault-auth service account secret.

    $ export SA_SECRET_NAME=$(kubectl get secrets --output=json \
    | jq -r '.items[].metadata | select(.name|startswith("vault-auth-")).name')

    Set the SA_JWT_TOKEN environment variable value to the service account JWT used to access the TokenReview API

    $ export SA_JWT_TOKEN=$(kubectl get secret $SA_SECRET_NAME \
    --output 'go-template={{ .data.token }}' | base64 --decode)

    Set the SA_CA_CRT environment variable value to the PEM encoded CA cert used to talk to Kubernetes API.

    $ export SA_CA_CRT=$(kubectl config view --raw --minify --flatten \
    --output 'jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)

    Set the K8S_HOST environment variable value to minikube IP address.

    $ export K8S_HOST=$(kubectl config view --raw --minify --flatten \
    --output 'jsonpath={.clusters[].cluster.server}')

  4. Now, enable and configure the Kubernetes auth method.

    Enable the Kubernetes auth method at the default path ("auth/kubernetes").

    $ vault auth enable kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

    Tell Vault how to communicate with the Kubernetes (Minikube) cluster.

    $ vault write auth/kubernetes/config \
     token_reviewer_jwt="$SA_JWT_TOKEN" \
     kubernetes_host="$K8S_HOST" \
     kubernetes_ca_cert="$SA_CA_CRT" \
     issuer="https://kubernetes.default.svc.cluster.local"

    Output:

    Success! Data written to: auth/kubernetes/config

    You can validate the issuer name of your Kubernetes cluster using this method.

  5. Create a role named, example, that maps the Kubernetes Service Account to Vault policies and default token TTL.

    $ vault write auth/kubernetes/role/example \
     bound_service_account_names=vault-auth \
     bound_service_account_namespaces=default \
     token_policies=myapp-kv-ro \
     ttl=24h

    Output:

    Success! Data written to: auth/kubernetes/role/example

Note

The pattern Vault uses to authenticate Pods depends on sharing the JWT token over the network. Given the security model of Vault, this is allowable because Vault is part of the trusted compute base. In general, Kubernetes applications should not share this JWT with other applications, as it allows API calls to be made on behalf of the Pod and can result in unintended access being granted to 3rd parties.

Determine the Vault address

A service bound to all networks on the host, as you configured Vault, is addressable by pods within Minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.

  1. Start a minikube SSH session.

    $ minikube ssh
## ... minikube ssh login

  2. Within this SSH session, retrieve the value of the Minikube host.

    $ dig +short host.docker.internal
192.168.65.2

    Docker networking

    The host has a changing IP address (or none if you have no network access). We recommend that you connect to the special DNS name host.docker.internal which resolves to the internal IP address used by the host. host.docker.internal. This is for development purpose and will not work in production. For more information, review the documentation for Mac, Windows.

  3. Next, retrieve the status of the Vault server to verify network connectivity.

    $ dig +short host.docker.internal | xargs -I{} curl -s http://{}:8200/v1/sys/seal-status
{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.4.1+ent",
  "migration": false,
  "cluster_name": "vault-cluster-3de6c2d3",
  "cluster_id": "10fd177e-d55a-d740-0c54-26268ed86e31",
  "recovery_seal": false,
  "storage_type": "inmem"
}

    The output displays that Vault is initialized and unsealed. This confirms that pods within your cluster are able to reach Vault given that each pod is configured to use the gateway address.

  4. Next, exit the Minikube SSH session.

    $ exit

  5. Finally, create a variable named EXTERNAL_VAULT_ADDR to capture the Minikube gateway address.

    $ EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')

  6. Verify that the variable contains the IP address you saw when executed in the Minikube shell.

    $ echo $EXTERNAL_VAULT_ADDR
192.168.65.2

Optional: Verify the Kubernetes auth method configuration

  1. Define a Pod with a container.

    $ cat > devwebapp.yaml <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: devwebapp
  labels:
    app: devwebapp
spec:
  serviceAccountName: vault-auth
  containers:
    - name: devwebapp
      image: burtlo/devwebapp-ruby:k8s
      env:
        - name: VAULT_ADDR
          value: "http://$EXTERNAL_VAULT_ADDR:8200"
EOF

    The Pod is named devwebapp and runs with the vault-auth service account.

  2. Create the devwebapp pod in the default namespace

    $ kubectl apply --filename devwebapp.yaml --namespace default
pod/devwebapp created

  3. Display all the pods in the default namespace.

    $ kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
devwebapp   1/1     Running   0          77s

    Wait until the devwebapp pod is running and ready (1/1).

  4. Start an interactive shell session on the devwebapp pod.

    $ kubectl exec --stdin=true --tty=true devwebapp -- /bin/sh
#

    Your system prompt is replaced with a new prompt #.

    Note

    The prompt within this section is shown as $ but the commands are intended to be executed within this interactive shell on the devwebapp container.

  5. Set KUBE_TOKEN to the service account token.

    $ KUBE_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

  6. Authenticate with Vault through the example role with the KUBE_TOKEN.

    $ curl --request POST \
       --data '{"jwt": "'"$KUBE_TOKEN"'", "role": "example"}' \
       $VAULT_ADDR/v1/auth/kubernetes/login

    Example output:

    {
   ...snip...
   "auth": {
      "client_token": "hvs.CAESIBxfvzds7M4tas017ls36Dl_kA-3YpCCBT9wczP1E41DGh4KHGh2cy5JYW1NMmFkb2gwTVBhZVhsN0pDM0tOaWM",
      "accessor": "1Miu5tbfuZzYuYWCntb4Ztke",
      "policies": [
         "default",
         "myapp-kv-ro"
      ],
      "token_policies": [
         "default",
         "myapp-kv-ro"
      ],
      "metadata": {
         "role": "example",
         "service_account_name": "vault-auth",
         "service_account_namespace": "default",
         "service_account_secret_name": "",
         "service_account_uid": "a293b5fb-96a2-43b0-acee-03d9ffce9423"
      },
      "lease_duration": 86400,
      "renewable": true,
      "entity_id": "8925670b-d8d6-0792-ee56-eec1a9e740cb",
      "token_type": "service",
      "orphan": true,
      "mfa_requirement": null,
      "num_uses": 0
   }
}

    Notice that client_token is successfully generated and myapp-kv-ro policy is attached with the token. The metadata displays that its service account name (service_account_name) is vault-auth.

  7. Lastly, exit the pod.

    $ exit

Start Vault Agent with Auto-Auth

Now that you have verified that the Kubernetes auth method has been configured on the Vault server, it is time to spin up a client Pod which leverages Vault Agent to automatically authenticate with Vault and retrieve a client token.

  1. First, open the provided configmap.yaml file in your preferred text editor and review its content.

    configmap.yaml
    1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031323334353637383940414243apiVersion: v1
data:
  vault-agent-config.hcl: |
    # Comment this out if running as sidecar instead of initContainer
    exit_after_auth = true

    pid_file = "/home/vault/pidfile"

    auto_auth {
        method "kubernetes" {
            mount_path = "auth/kubernetes"
            config = {
                role = "example"
            }
        }

        sink "file" {
            config = {
                path = "/home/vault/.vault-token"
            }
        }
    }

    template {
    destination = "/etc/secrets/index.html"
    contents = <<EOT
    <html>
    <body>
    <p>Some secrets:</p>
    {{- with secret "secret/data/myapp/config" }}
    <ul>
    <li><pre>username: {{ .Data.data.username }}</pre></li>
    <li><pre>password: {{ .Data.data.password }}</pre></li>
    </ul>
    {{ end }}
    </body>
    </html>
    EOT
    }
kind: ConfigMap
metadata:
  name: example-vault-agent-config
  namespace: default

    This creates a Vault Agent configuration file, vault-agent-config.hcl. Notice that the Vault Agent Auto-Auth (auto_auth block) is configured to use the kubernetes auth method enabled at the auth/kubernetes path on the Vault server. The Vault Agent will use the example role which you created in Step 2.

    The sink block specifies the location on disk where to write tokens. Vault Agent Auto-Auth sink can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, the sink is set to /home/vault/.vault-token.

    Finally, the template block creates a templated file which retrieves username and password values at the secret/data/myapp/config path.

  2. Create a ConfigMap containing a Vault Agent configuration.

    $ kubectl create --filename configmap.yaml
configmap/example-vault-agent-config created

  3. View the created ConfigMap.

    $ kubectl get configmap example-vault-agent-config --output yaml

  4. An example Pod spec file is provided. Review the provided example Pod spec file, example-k8s-spec.yaml.

    example-k8s-spec.yaml
    apiVersion: v1
kind: Pod
metadata:
  name: vault-agent-example
  namespace: default
spec:
  serviceAccountName: vault-auth

  volumes:
    - configMap:
        items:
          - key: vault-agent-config.hcl
            path: vault-agent-config.hcl
        name: example-vault-agent-config
      name: config
    - emptyDir: {}
      name: shared-data

  initContainers:
    - args:
        - agent
        - -config=/etc/vault/vault-agent-config.hcl
        - -log-level=debug
      env:
        - name: VAULT_ADDR
          value: http://EXTERNAL_VAULT_ADDR:8200
      image: vault
      name: vault-agent
      volumeMounts:
        - mountPath: /etc/vault
          name: config
        - mountPath: /etc/secrets
          name: shared-data

  containers:
    - image: nginx
      name: nginx-container
      ports:
        - containerPort: 80
      volumeMounts:
        - mountPath: /usr/share/nginx/html
          name: shared-data

    The example Pod spec (example-k8s-spec.yaml) spins up two containers in vault-agent-example pod. A vault container which runs Vault Agent as an Init Container. And an nginx container exposing port 80.

    Kubernetes

    The Vault address, VAULT_ADDR, is set to a placeholder value EXTERNAL_VAULT_ADDR.

  5. Generate the Pod spec with EXTERNAL_VAULT_ADDR variable value in its place.

    $ cat example-k8s-spec.yaml | \
   sed -e s/"EXTERNAL_VAULT_ADDR"/"$EXTERNAL_VAULT_ADDR"/ \
   > vault-agent-example.yaml

  6. Create the vault-agent-example pod defined in vault-agent-example.yaml.

    $ kubectl apply --filename vault-agent-example.yaml --record

    This takes a minute or so for the pod to become fully up and running.

Verification

  1. In another terminal, launch the Minikube dashboard.

    $ minikube dashboard

  2. Click Pods under Workloads to verify that vault-agent-example pod has been created successfully.

    Kubernetes

  3. Select vault-agent-example to see its details.

    Kubernetes

  4. In another terminal, port forward all requests made to http://localhost:8080 to port 80 on the vault-agent-example pod.

    $ kubectl port-forward pod/vault-agent-example 8080:80

  5. In a web browser, go to localhost:8080

    Kubernetes

    Notice that the username and password values were successfully read from secret/myapp/config.

  6. Optionally, you can view the HTML source.

    $ kubectl exec -it vault-agent-example --container nginx-container -- cat /usr/share/nginx/html/index.html
<html>
<body>
<p>Some secrets:</p>
<ul>
<li><pre>username: appuser</pre></li>
<li><pre>password: suP3rsec(et!</pre></li>
</ul>

</body>
</html>

Azure Kubernetes Service Cluster

If you wish to test the Kubernetes auth method against an Azure Kubernetes Service (AKS) cluster instead of Minikube, you can run Terraform to provision an AKS cluster.

First, follow the instruction in the Terraform documentation to create a service principal.

Be sure to add the following Azure Active Directory Graph API permission to your app.

API permissions

Store the credentials as Environment Variables.

$ export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
$ export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
$ export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

  1. Set your working directory to where the learn-vault-agent/vault-agent-k8s-demo/terraform-azure folder is located.

  2. Modify terraform.tfvars.example and provide Azure credentials: client_id and client_secret, and save it as terraform.tfvars.

    Client ID: Same as the Application ID

    Client secret: The password (credential) set on your application

  3. Execute the Terraform commands to provision a new AKS cluster.

    Pull necessary plugins.

    $ terraform init

    Now, execute the apply command to build a new AKS cluster.

    $ terraform apply -auto-approve

    Note

    If you received Resource group 'education-XXXXX-rg' could not be found error, run terraform refresh command.

    $ terraform refresh
...

Outputs:

kubernetes_cluster_name = education-xxxxxx-aks
resource_group_name = education-xxxxxx-rg

  4. Now, view the information about the AKS cluster.

    $ kubectl cluster-info

Kubernetes master is running at https://education.hcp.westus2.azmk8s.io:443
Heapster is running at ...
...

    NOTE: Copy the Kubernetes master address (https://education.hcp.westus2.azmk8s.io:443 in this example).

  5. In the /vault-agent-k8s-demo/setup-k8s-auth.sh file, replace Line 48 to point to the AKS cluster address rather than export K8S_HOST=$(minikube ip). Also, replace Line 54 to point to the correct host address.

    Example:

    setup-k8s-auth.sh
    ...
# Set K8S_HOST to minikube IP address
# export K8S_HOST=$(minikube ip)
export K8S_HOST="https://education.hcp.westus2.azmk8s.io:443"

# Enable the Kubernetes auth method at the default path ("auth/kubernetes")
vault auth enable kubernetes

# Tell Vault how to communicate with the Kubernetes (Minikube) cluster
# vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
...

  6. Set the working directory to where scripts are located (learn-vault-agent/vault-agent-k8s-demo).

    $ cd ..

  7. Create a service account, vault-auth.

    $ kubectl create serviceaccount vault-auth

  8. Update the vault-auth service account based on the definition in the vault-auth-service-account.yaml file.

    $ kubectl apply --filename vault-auth-service-account.yaml

  9. Setup the Kubernetes auth method on the Vault server.

    $ ./setup-k8s-auth.sh

  10. Resume with the determine the Vault address step and on.

AKS Kubernetes Dashboard

Click on the Monitor containers in the Azure portal.

Kubernetes

Click the Enable button so that you can monitor the Kubernetes cluster utilization metrics.

Clean up

When you are done experimenting this demo, execute the following commands to clean up.

Execute the terraform command.

$ terraform apply -destroy -auto-approve

Remove the Terraform state files.

$ rm *tfstate*

Google Kubernetes Engine Cluster

  1. Set your working directory to where the learn-vault-agent/vault-agent-k8s-demo/terraform-gcp folder is located.

  2. Modify terraform.tfvars.example and provide GCP credentials: account_file_path and project, and save it as terraform.tfvars.

    Example:

    terraform.tfvars
    account_file_path = "/usr/student/gcp/vault-test-project.json"
project  = "vault-test-project"

  3. Execute the Terraform commands to provision a new GKE cluster.

    Pull necessary plugins.

    $ terraform init

    Now, execute the apply command to build a new GKE cluster.

    $ terraform apply -auto-approve

  4. Connect to the GKE cluster.

    $ gcloud container clusters get-credentials $(terraform output gcp_cluster_name) \
        --zone $(terraform output gcp_zone) \
        --project $(terraform output gcp_project)

  5. Get the cluster information.

    $ kubectl cluster-info

Kubernetes master is running at https://198.51.100.24
GLBCDefaultBackend is running at https://198.51.100.24/api/v1/namespaces/...
...

    NOTE: If you don't have the gcloud command line tool installed, follow the online documentation.

  6. Copy the Kubernetes master address (https://198.51.100.24 in this example).

  7. In the /vault-agent-k8s-demo/setup-k8s-auth.sh file, replace Line 48 to point to the GKE cluster address rather than export K8S_HOST=$(minikube ip). Also, replace Line 54 to point to the correct host address.

    Example:

    setup-k8s-auth.sh
    ...
# Set K8S_HOST to minikube IP address
# export K8S_HOST=$(minikube ip)
export K8S_HOST="https://198.51.100.24"

# Enable the Kubernetes auth method at the default path ("auth/kubernetes")
vault auth enable kubernetes

# Tell Vault how to communicate with the Kubernetes (Minikube) cluster
# vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="https://$K8S_HOST:8443" kubernetes_ca_cert="$SA_CA_CRT"
vault write auth/kubernetes/config token_reviewer_jwt="$SA_JWT_TOKEN" kubernetes_host="$K8S_HOST" kubernetes_ca_cert="$SA_CA_CRT"
...

  8. Set the working directory to where scripts are located (learn-vault-agent/vault-agent-k8s-demo).

    $ cd ..

  9. Create a service account, vault-auth.

    $ kubectl create serviceaccount vault-auth

  10. Update the vault-auth service account with definition provided in the vault-auth-service-account.yaml file.

    $ kubectl apply --filename vault-auth-service-account.yaml

  11. Setup the Kubernetes auth method on the Vault server.

    $ ./setup-k8s-auth.sh

  12. Resume with the determine the Vault address step and on.

Help and reference

Other Vault Agent tutorials:

Previous

Vault Agent with AWS

 Next

Vault Agent Caching

This tutorial also appears in:

  • 22 tutorials
    Vault on Kubernetes
    Vault secures, stores, and tightly controls access to passwords, certificates, and other secrets in modern computing. Here are a series of tutorials that are all about running Vault on Kubernetes.
    • Vault
  • 17 tutorials
    Authentication methods
    Vault clients must authenticate with Vault first and acquire a valid token. Learn available auth methods.
    • Vault