Read secrets for Kubernetes workloads from HCP Vault Dedicated

17min
|
HCP
Vault

Kubernetes workloads can authenticate with HCP Vault Dedicated to access secrets managed by Vault.

Challenge

Managing access to secrets is a critical process for organizations that run workloads on Kubernetes. A Vault client (a user, an application, Kubernetes pod, etc.) must authenticate with Vault before it can read secrets managed by Vault.

Solution

HCP Vault Dedicated can manage secrets for Kubernetes workloads from outside the Kubernetes cluster.

In this tutorial, you will setup:

Your local environment to support Vault Dedicated.
Start a Kubernetes cluster using minikube.
Authenticate with Vault Dedicated using:
- Vault token
- Kubernetes auth method
- Kubernetes auth method via the Vault Agent injector service
Read secrets stored in Vault Dedicated.

Running Vault in Kubernetes

Running self hosted Vault in a Kubernetes cluster is explored in the Vault Installation to minikube via Helm and Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar tutorials.

Prerequisites

HCP Vault Dedicated dev tier or higher cluster available
Vault
kubectl
Helm
minikube installed and configured
ngrok installed and configured with an auth token

This tutorial was tested against the following specific versions:

HCP Vault Dedicated 1.12.3
Vault CLI 1.13
Kubectl 1.26
Helm 3.112
minikube 1.29
Kubernetes 1.26

Personas

The scenario described in this tutorial involves two personas:

admin with privileged permissions to manage Vault and Kubernetes
apps reads the secrets from Vault (client)

Lab setup

Vault setup

Note

If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
Warning
Organizations running HCP Vault Dedicated should consider disabling public access to the Vault cluster. Pods running in Kubernetes can connect to the private Vault Dedicated address using connectivity options with a supported public cloud provider such as using an AWS transit gateway or an Azure VNet peering connection.
Open a new terminal and set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.
Note
For these tasks, you can use HCP Vault Dedicated's admin token. However, it is recommended that admin tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

Enable the KV secret engine.

$ vault secrets enable -version=2 -path=secret kv

Success! Enabled the kv secrets engine at: secret/

Create a secret at path secret/exampleapp/config with a username and password.

$ vault kv put secret/exampleapp/config username='jalbertson' password='bestpasswordever'

======== Secret Path ========
secret/data/exampleapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2022-06-06T18:26:14.070155Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

The Vault Dedicated server is ready.

minikube setup

minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes cluster locally inside Virtual Machines (VM) on your system.

Start a Kubernetes cluster.

$ minikube start

😄  minikube v1.25.2 on Darwin 12.3
✨  Automatically selected the docker driver. Other choices: hyperkit, virtualbox, ssh
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8100MB) ...
🐳  Preparing Kubernetes v1.23.3 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

Verify the status of the minikube cluster.

$ minikube status

minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

Create a Kubernetes service account named exampleapp with a service account token.

$ kubectl create -f - <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: exampleapp
---
apiVersion: v1
kind: Secret
metadata:
  name: exampleapp
  annotations:
    kubernetes.io/service-account.name: exampleapp
type: kubernetes.io/service-account-token
---
EOF

Example output:

serviceaccount/exampleapp created
secret/exampleapp created

minikube is ready.

Token auth method authentication

(Persona: admin)

A pod can authenticate directly with a Vault Dedicated cluster using a token. In this example, the admin token is used for simplicity but any token can be supplied in the pod configuration.

Tokens can be created in Vault using a auth method and attaching a policy. Policies should follow the concept of least privilege to ensure the pod only has access to the secrets required.

Create a pod named exampleapp-token that sets the VAULT_ADDR, VAULT_NAMESPACE, and VAULT_TOKEN environment variables.

Note

This pod configuration includes examples of how to set environment variables that your application may require to interact with Vault. In this tutorial, you are interacting with the pod using the kubectl command-line interface so the environment variables in subsequent commands are also interpreted from your local shell session.

$ kubectl create -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: exampleapp-token
  labels:
    app: exampleapp-token
spec:
  serviceAccountName: exampleapp
  containers:
    - name: app
      image: ubuntu
      command: [ "/bin/sh" , "-c", "tail -f /dev/null" ]
      env:
      - name: VAULT_ADDR
        value: $VAULT_ADDR
      - name: VAULT_TOKEN
        value: $VAULT_TOKEN
      - name: VAULT_NAMESPACE
        value: $VAULT_NAMESPACE
EOF

Example output:

pod/exampleapp-token created

Get all the pods in the default namespace.

$ kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
exampleapp-token   1/1     Running   0          4m

Wait until the exampleapp-token pod reports that is running and ready (1/1).

Update the containers apt repositories.

$ kubectl exec exampleapp-token -- apt update

Install cURL in the container.
Note
cURL is used to simulate an application retrieving secrets from the HCP Vault Dedicated cluster.
```
$ kubectl exec exampleapp-token -- apt install curl -y
```

Verify connectivity from the Ubuntu container to the Vault Dedicated cluster.

(Persona: apps)

$ kubectl exec exampleapp-token -- \
   curl --silent $VAULT_ADDR/v1/sys/seal-status | jq

Example output:

{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.12.3+ent",
  "build_date": "2023-02-03T12:10:29Z",
  "migration": false,
  "cluster_name": "vault-cluster-be46d550",
  "cluster_id": "51ac8751-45d4-1970-2177-b30f6229568f",
  "recovery_seal": true,
  "storage_type": "raft"
}

Read the secret/exampleapp/config secret created in the lab setup section.

$ kubectl exec exampleapp-token -- curl -silent \
   --header "X-Vault-Request: true" \
   --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/data/exampleapp/config | jq

The API returns the secret defined at the path secret/data/exampleapp/config.

Example output:

{
  "request_id": "508cdab5-166e-5a0d-519a-266e8a658ea6",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "data": {
      "password": "bestpasswordever",
      "username": "jalbertson"
    },
    "metadata": {
      "created_time": "2023-03-24T18:25:22.991166575Z",
      "custom_metadata": null,
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Stop the exampleapp-token pod.

$ kubectl delete pod exampleapp-token

pod "exampleapp-token" deleted

Kubernetes auth method authentication

Rather than generating and assigning a token though the token auth method, pods can authenticate directly with Vault using a Kubernetes service account and an associated service account token. Vault uses the service account to validate the identity of the pod using the Kubernetes API.

In this scenario, Vault requires access to the Kubernetes API. In production environments, this would be done by connecting your HCP HVN to a transit gateway or peering connection with the network where Kubernetes is running.

For the purposes of this tutorial, you will configure ngrok to expose the Kubernetes API.

Kubernetes configuration

(Persona: admin)

Create a Kubernetes service account for Vault to authenticate with the Kubernetes, a service account token for the service account, and a role to allow Vault to access the appropriate API.

$ kubectl create -f - <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-auth
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-auth
type: kubernetes.io/service-account-token
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: role-tokenreview-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-auth
    namespace: default
EOF

Example output:

serviceaccount/vault-auth created
secret/vault-auth created
clusterrolebinding.rbac.authorization.k8s.io/role-tokenreview-binding created

Retrieve the vault-auth secret and store it as an environment variable.

$ VAULTAUTH_SECRET=$(kubectl get secret vault-auth -o json | jq -r '.data')

Review the secret.

$ echo $VAULTAUTH_SECRET

Example output:

{
  "ca.crt": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURC..<snip>..lwNGN6cmFpb0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==",
  "namespace": "ZGVmYXVsdA==",
  "token": "ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNkl..<snip>..TWR3T0kwRVlxcE0zZGJ1d3JMUW5vMjNhSnJWaU5SaEp3"
}

The secret includes the public key ca.crt and token as base64 encoded strings.

Decode the ca.crt certificate and store it as an environment variable.

$ K8S_CA_CRT=$(echo $VAULTAUTH_SECRET | jq -r '."ca.crt"' | base64 -d)

Decode the token and store it as an environment variable.
```
$ VAULTAUTH_TOKEN=$(echo $VAULTAUTH_SECRET | jq -r '.token' | base64 -d)
```
You have collected the necessary information for the vault-auth service account to configure the Kubernetes auth method.

Configure networking

(Persona: admin)

In another terminal, start a proxy to expose the Kubernetes API.

$ kubectl proxy --disable-filter=true

Request filter disabled, your proxy is vulnerable to XSRF attacks, please be cautious
Starting to serve on 127.0.0.1:8001

Leave this tab open with the proxy running.

In another terminal, start ngrok and create a tunnel to the proxy listening on port 8001.

Warning

ngrok is used to expose the Kubernetes API to HCP Vault Dedicated. Using --scheme=http exposes the API without encryption to avoid TLS certificate errors.

For production workloads, use a private peering or transit gateway connection with trusted certificates.

$ ngrok http --scheme=http 127.0.0.1:8001

Example output:

ngrok                                                                                                                                                              (Ctrl+C to quit)

Session Status                online
Account                       username (Plan: Free)
Update                        update available (version 3.0.5, Ctrl-U to update)
Version                       3.1.1
Region                        United States (us)
Latency                       32.791235ms
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://d12b-34-567-89-10.ngrok.io -> 127.0.0.1:8001

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

Copy the ngrok forwarding address.
Return to the terminal where you set the VAULT_ADDR environment variable and set an environment variable for the ngrok forwarding address.
```
$ export K8S_URL=<actual-address-from-ngrok>
```

Configure Vault

(Persona: admin)

Enable the Kubernetes auth method.

$ vault auth enable kubernetes

Success! Enabled kubernetes auth method at: kubernetes/

Configure the Kubernetes auth method.

$ vault write auth/kubernetes/config \
 token_reviewer_jwt=$VAULTAUTH_TOKEN \
 kubernetes_host=$K8S_URL \
 kubernetes_ca_cert=$K8S_CA_CRT

Example output:

Success! Data written to: auth/kubernetes/config

Create a Vault policy that permits read access to secret/exampleapp/config

$ vault policy write exampleapp-read - << EOF
path "secret/data/exampleapp/config" {
  capabilities = ["read"]
}
EOF

Example output:

Success! Uploaded policy: exampleapp-read

Create a role for the Kubernetes auth method and include the exampleapp-read Vault policy.

$ vault write auth/kubernetes/role/exampleapp \
bound_service_account_names=exampleapp \
bound_service_account_namespaces=default \
policies=default,exampleapp-read \
ttl=1h

Example output:

Success! Data written to: auth/kubernetes/role/exampleapp

Authenticate and read secret

(Persona: apps)

Retrieve the exampleapp service account token and store it as an environment variable.

$ EXAMPLEAPP_TOKEN=$(kubectl get secret exampleapp -o json | jq -r '.data | .token' | base64 -d)

Create a pod named exampleapp-auth that sets the EXAMPLEAPP_AUTH, VAULT_ADDR, and VAULT_NAMESPACE environment variables.
VAULT_TOKEN is not required because the pod will authenticate with Vault using the Kubernetes service account defined in the auth method configuration to retrieve a token.
Note
This pod configuration includes examples of how to set environment variables that your application may require to interact with Vault. In this tutorial, you are interacting with the pod using the kubectl command-line interface so the environment variables in subsequent commands are also interpreted from your local shell session.
```
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: exampleapp-auth
  labels:
    app: exampleapp-auth
spec:
  serviceAccountName: exampleapp
  containers:
    - name: app
      image: ubuntu
      command: [ "/bin/sh" , "-c", "tail -f /dev/null" ]
      env:
      - name: VAULT_ADDR
        value: $VAULT_ADDR
      - name: VAULT_NAMESPACE
        value: $VAULT_NAMESPACE
      - name: EXAMPLEAPP_TOKEN
        value: $EXAMPLEAPP_TOKEN
EOF
```
Example output:
```
pod/exampleapp-auth created
```

Get all the pods in the default namespace.

$ kubectl get pods
NAME               READY   STATUS    RESTARTS   AGE
exampleapp-auth    1/1     Running   0          4m

Wait until the exampleapp-auth pod reports that is running and ready (1/1).

Update the containers apt repositories.

$ kubectl exec exampleapp-auth -- apt update

Install cURL in the container.
Note
cURL is used to simulate an application retrieving secrets from the HCP Vault Dedicated cluster.
```
$ kubectl exec exampleapp-auth -- apt install curl -y
```

Verify connectivity from the Ubuntu container to the Vault Dedicated cluster.

$ kubectl exec exampleapp-auth -- \
   curl --silent $VAULT_ADDR/v1/sys/seal-status | jq

Example output:

{
  "type": "shamir",
  "initialized": true,
  "sealed": false,
  "t": 1,
  "n": 1,
  "progress": 0,
  "nonce": "",
  "version": "1.12.3+ent",
  "build_date": "2023-02-03T12:10:29Z",
  "migration": false,
  "cluster_name": "vault-cluster-be46d550",
  "cluster_id": "51ac8751-45d4-1970-2177-b30f6229568f",
  "recovery_seal": true,
  "storage_type": "raft"
}

Use the exampleapp Vault role to authenticate and store the generated Vault token as an environment variable.

$ POD_TOKEN=$(kubectl exec exampleapp-auth -- curl --silent \
   --request POST \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --data '{"jwt": "'$EXAMPLEAPP_TOKEN'", "role": "exampleapp"}' \
   $VAULT_ADDR/v1/auth/kubernetes/login | jq -r '.auth | .client_token')

Read the secret/exampleapp/config secret created in the lab setup section using the POD_TOKEN.

$ kubectl exec exampleapp-auth -- curl --silent \
   --header "X-Vault-Request: true" \
   --header "X-Vault-Token: $POD_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   $VAULT_ADDR/v1/secret/data/exampleapp/config | jq

The API returns the secret defined at the path secret/data/exampleapp/config because the API call used the token obtained using the Kubernetes auth method with the exampleapp-read policy attached.

Example output:

{
  "request_id": "a32ec3f2-c6c4-67dd-6de5-c3ef208381af",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "data": {
      "password": "bestpasswordever",
      "username": "jalbertson"
    },
    "metadata": {
      "created_time": "2023-03-30T17:08:44.218378893Z",
      "custom_metadata": null,
      "deletion_time": "",
      "destroyed": false,
      "version": 1
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}

Stop the exampleapp-auth pod.

$ kubectl delete pod exampleapp-auth

pod "exampleapp-token" deleted

Vault Agent injector service

While a pod can access Vault directly, there are times when network restrictions do not allow one or more pods to connect to external services such as Vault Dedicated.

When requirements limit egress network traffic, one option to consider is using the Vault Agent injector service. The injector service enables the authentication and secret retrieval on behalf of the applications. The Vault Agent pod, however, will require an exception to permit egress communications to Vault Dedicated.

In this section, you will install the Vault Helm chart to run the injector service, and configure the pod with the necessary annotations to work with the injector service.

Install Vault Agent

(Persona: admin)

Add the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

Update all the repositories to ensure helm is aware of the latest versions.

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

Install the Vault agent and configure with the Vault Dedicated public address.
Note
The Vault Helm chart also creates a service account named vault. In the previous section you created a dedicated service account for Vault so the vault account will not be used in this tutorial.
```
$ helm install vault hashicorp/vault \
    --set "global.externalVaultAddr=$VAULT_ADDR" \
```
The global.externalVaultAddr is assigned the address of Vault Dedicated cluster.
Get all the pods in the default namespace.
```
$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-agent-injector-7b6cd469d8-8svg5   1/1     Running   0          15s
```
The Vault Agent Injector pod is deployed in the default namespace.
Wait until the vault-agent-injector pod reports that it is running and ready (1/1).

Inject secrets into the pod

(Persona: apps)

The Vault Agent Injector only modifies a pod or deployment if it has a specific set of annotations. Refer to the Vault documentation for a complete list of available annotations.

Review the pod configuration.
```
apiVersion: v1
kind: Pod
metadata:
  name: exampleapp-secret-agent
  labels:
    app: exampleapp-secret-agent
  annotations:
    vault.hashicorp.com/agent-inject: 'true'
    vault.hashicorp.com/role: 'exampleapp'
    vault.hashicorp.com/agent-inject-secret-credentials.txt: 'secret/data/exampleapp/config'
    vault.hashicorp.com/namespace: 'admin'
spec:
  serviceAccountName: exampleapp
  containers:
    - name: app
      image: ubuntu
```
These annotations define a partial structure of the deployment schema and are prefixed with vault.hashicorp.com.
- agent-inject enables the Vault Agent Injector service
- role specifies the Vault Kubernetes authentication role
- agent-inject-secret-FILEPATH prefixes the path of the file, credentials.txt written to the /vault/secrets directory. The value is the path to the secret defined in Vault.
- namespace defines the Vault Dedicated namespace to be used. In this tutorial you are using the default admin namespace.

Create the exampleapp-secret-agent pod.

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: exampleapp-secret-agent
  labels:
    app: exampleapp-secret-agent
  annotations:
    vault.hashicorp.com/agent-inject: 'true'
    vault.hashicorp.com/role: 'exampleapp'
    vault.hashicorp.com/agent-inject-secret-credentials.txt: 'secret/data/exampleapp/config'
    vault.hashicorp.com/namespace: 'admin'
spec:
  serviceAccountName: exampleapp
  containers:
    - name: app
      image: ubuntu
      command: [ "/bin/sh" , "-c", "tail -f /dev/null" ]
EOF

Example output:

pod/exampleapp-secret-agent created

Get all the pods in the default namespace.

$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
exampleapp-secret-agent                 1/1     Running   0          84s
vault-agent-injector-7b6cd469d8-8svg5   1/1     Running   0          17m

Wait until the exampleapp-secret-agent pod reports that it is running and ready (2/2).

Display the secrets written to the file /vault/secrets/secret-credentials.txt on the exampleapp-secret-agent pod.
```
$ kubectl exec -it exampleapp-secret-agent -c app -- cat /vault/secrets/credentials.txt
```
The result displays the unformatted secret data present on the container.
```
data: map[password:salsa username:giraffe]
metadata: map[created_time:2023-04-03T05:26:14.070155Z custom_metadata:<nil> deletion_time: destroyed:false version:1]
```
Formatting data
A template can be applied to structure this data to meet the needs of the application.
The application in this pod still retrieves the secrets directly, but now that the injector service is deployed and capable of retrieving secrets for the application, future updates can remove that application logic.

Cleanup

Delete the exampleapp-secret-agent pod.

$ kubectl delete pod exampleapp-secret-agent

Remove the Vault agent pod.
```
$ helm uninstall vault
```
Stop minikube.
```
$ minikube stop
```
Delete minikube.
Warning
This will remove all existing minikube configurations.
```
$ minikube delete
```
Return to the terminal where the proxy is running and press ctrl-c to stop the proxy.
Return to the terminal where ngrok is running and press ctrl-c to stop ngrok.

Next steps

You deployed Vault Dedicated external to a Kubernetes cluster and deployed pods that leveraged it as a secrets store. You configured a pod to retrieve secrets using the token auth method, using the Kubernetes auth method, and finally through the Vault injector service.

Learn more about the Vault Helm chart by reading the documentation, exploring the project source code, exploring how pods can retrieve secrets through the Vault Injector service via annotations, or secrets mounted on ephemeral volumes.

Collection Overview

Integrate with HCP Vault

Create Kubernetes native secrets