Vault
Integrate a Kubernetes cluster with an external Vault
Vault can manage secrets for Kubernetes application pods from outside the cluster. This could be HashiCorp Cloud Platform (HCP) Vault or another Vault service within your organization.
Running Vault in Kubernetes
Vault running in the cluster is explored in the Vault installation to minikube via Helm with Consul and Injecting secrets into Kubernetes pods via Vault Helm sidecar tutorials.
In this tutorial, you will run Vault locally, and start a Kubernetes cluster with minikube. You will deploy an application that retrieves secrets directly from Vault via a Kubernetes service and secret injection via Vault Agent Injector.
Prerequisites
- Vault version 1.13.0 or later
- Docker
- Kubernetes CLI
- Minikube
- Helm CLI
- Vault CLI
This tutorial was last tested 7 August 2023 on a macOS 13.4.2 using this configuration.
$ docker version
Client:
Cloud integration: v1.0.25
Version: 20.10.16
## ...
$ kubectl version --short
Client Version: v1.27.1
Kustomize Version: v5.0.1
Server Version: v1.27.3
$ minikube version
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0
$ helm version
version.BuildInfo{Version:"v3.12.2", GitCommit:"1e210a2c8cc5117d1055bfaa5d40f51bbc2e345e", GitTreeState:"clean", GoVersion:"go1.20.6"}
$ vault version
Vault v1.14.1 (bf23fe8636b04d554c0fa35a756c75c2f59026c0), built 2023-07-21T10:15:14Z
These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.
The GitHub repository
Next, retrieve the web application and additional configuration by cloning the hashicorp-education/learn-vault-external-kubernetes repository from GitHub.
$ git clone https://github.com/hashicorp-education/learn-vault-external-kubernetes.git
This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found in a sub-directory.
Go into the
learn-vault-external-kubernetes
directory.$ cd learn-vault-external-kubernetes
Working directory
This tutorial assumes that the remainder of commands are executed in this directory.
Start Vault
Vault running external of a Kubernetes cluster can be addressed by any of its pods as long as the Vault server is network addressable. Running Vault locally alongside of minikube is possible if the Vault server is bound to the same network as the cluster.
Open a new terminal, start a Vault dev server with
root
as the root token that listens for requests at0.0.0.0:8200
.$ vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200
Setting the
-dev-listen-address
to0.0.0.0:8200
overrides the default address of a Vault dev server (127.0.0.1:8200
) and enables Vault to be addressable by the Kubernetes cluster and its pods because it binds to a shared network.Insecure operation
Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the
vault
CLI to address the Vault server.$ export VAULT_ADDR=http://0.0.0.0:8200
The web application that you deploy, expects Vault to store a username and password stored at the path
secret/devwebapp/config
. To create this secret requires that a key-value secret engine is enabled and a username and password is put at the specified path. By default the Vault dev server starts with a key-value secrets engine enabled at the path prefixed withsecret
.Login with the root token.
$ vault login root Success! You are now authenticated. The token information displayed below is already stored in the token helper. You do NOT need to run "vault login" again. Future Vault requests will automatically use this token. Key Value --- ----- token root token_accessor 6NYAtL0ANmAGVmLX3tx4bVgu token_duration ∞ token_renewable false token_policies ["root"] identity_policies [] policies ["root"]
Create a secret at path
secret/devwebapp/config
with ausername
andpassword
.$ vault kv put secret/devwebapp/config username='giraffe' password='salsa' ======== Secret Path ======== secret/data/devwebapp/config ======= Metadata ======= Key Value --- ----- created_time 2022-06-06T18:26:14.070155Z custom_metadata <nil> deletion_time n/a destroyed false version 1
Verify that the secret is stored at the path
secret/devwebapp/config
.$ vault kv get -format=json secret/devwebapp/config | jq ".data.data" { "password": "salsa", "username": "giraffe" }
Learn more
This tutorial focuses on Vault's integration with Kubernetes and not interacting the key-value secrets engine. For more information refer to the Versioned Key/Value Secrets Engine tutorial.
The Vault server, with secret, is ready to be addressed by a Kubernetes cluster and the pods deployed in it.
Start minikube
Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters locally inside Virtual Machines (VM) on your system.
Start a Kubernetes cluster.
$ minikube start 😄 minikube v1.25.2 on Darwin 12.3 ✨ Automatically selected the docker driver. Other choices: hyperkit, virtualbox, ssh 👍 Starting control plane node minikube in cluster minikube 🚜 Pulling base image ... 🔥 Creating docker container (CPUs=2, Memory=8100MB) ... 🐳 Preparing Kubernetes v1.23.3 on Docker 20.10.12 ... ▪ kubelet.housekeeping-interval=5m ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔎 Verifying Kubernetes components... ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5 🌟 Enabled addons: storage-provisioner 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.
Verify the status of the minikube cluster.
$ minikube status minikube type: Control Plane host: Running kubelet: Running apiserver: Running kubeconfig: Configured
Additional waiting
Even if this last command completed successfully, you may have to wait for minikube to be available. If an error is displayed, try again after a few minutes.
The host, kubelet, apiserver report that they are running. The
kubectl
, a command line interface (CLI) for running commands against Kubernetes cluster, is also configured to communicate with this recently started cluster.Minikube provides a visual representation of the status in a web-based dashboard. This interface displays the cluster activity in a visual interface that can be used to explore the issues affecting it.
In another terminal, launch the minikube dashboard.
$ minikube dashboard
The operating system's default browser opens and displays the dashboard.
Determine the Vault address
A service bound to all networks on the host, as you configured Vault, is addressable by pods in minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.
Start a minikube SSH session.
$ minikube ssh ## ... minikube ssh login
Within this SSH session, retrieve the value of the minikube host.
$ dig +short host.docker.internal 192.168.65.2
Docker networking
The host has a changing IP address (or none if you have no network access). We recommend that you connect to the special DNS name host.docker.internal which resolves to the internal IP address used by the host.
host.docker.internal
. This is for development purpose and will not work in production. For more information, review the documentation for Mac, Windows.Retrieve the status of the Vault server to verify network connectivity.
$ dig +short host.docker.internal | xargs -I{} curl -s http://{}:8200/v1/sys/seal-status | python3 -m json.tool { "type": "shamir", "initialized": true, "sealed": false, "t": 1, "n": 1, "progress": 0, "nonce": "", "version": "1.5.0", "migration": false, "cluster_name": "vault-cluster-44ba824c", "cluster_id": "adc0bb6a-e330-3e7a-e0c7-38061c3bf191", "recovery_seal": false, "storage_type": "inmem" }
Note
Your JSON output might not be formatted, but the contents will be similar.
The output displays that Vault is initialized and unsealed. This confirms that
pods in the cluster are able to reach Vault given that each pod is configured to
use the gateway address. The contents of this will be identical to running vault status
.
Exit the minikube SSH session.
$ exit
Create a variable named EXTERNAL_VAULT_ADDR to capture the minikube. gateway address.
$ EXTERNAL_VAULT_ADDR=$(minikube ssh "dig +short host.docker.internal" | tr -d '\r')
Verify that the variable has an ip address.
$ echo $EXTERNAL_VAULT_ADDR 192.168.65.2
Deploy application with hard-coded Vault address
The most direct way for a pod in the cluster to address Vault is with a hard-coded network address defined in the application code or provided as an environment variable. We've created and published a web application that allows override of the Vault address.
Create a Kubernetes service account named
internal-app
.$ kubectl create sa internal-app
Define a pod named
devwebapp
with the web application that sets theVAULT_ADDR
toEXTERNAL_VAULT_ADDR
.$ cat > devwebapp.yaml <<EOF apiVersion: v1 kind: Pod metadata: name: devwebapp labels: app: devwebapp spec: serviceAccountName: internal-app containers: - name: app image: burtlo/devwebapp-ruby:k8s env: - name: VAULT_ADDR value: "http://$EXTERNAL_VAULT_ADDR:8200" - name: VAULT_TOKEN value: root EOF
Create the
devwebapp
pod.$ kubectl apply --filename devwebapp.yaml pod/devwebapp created
Get all the pods in the default namespace.
$ kubectl get pods NAME READY STATUS RESTARTS AGE devwebapp 1/1 Running 0 4m
Wait until the
devwebapp
pod reports that is running and ready (1/1
).Request content served at
localhost:8080
from within thedevwebapp
pod.$ kubectl exec devwebapp -- curl -s localhost:8080 ; echo
The result displays the secret is defined at the path
secret/data/devwebapp/config
.{"password"=>"salsa", "username"=>"giraffe"}
The web application authenticates with the external Vault server using the root
token and returns the secret defined at the path secret/data/devwebapp/config
.
This hard-coded approach is an effective solution if the address to the Vault
server does not change.
Deploy service and endpoints to address an external Vault
An external Vault may not have a static network address that services in the cluster can rely upon. When Vault's network address changes each service also needs to change to continue its operation. Another approach to manage this network address is to define a Kubernetes service and endpoints.
A service creates an abstraction around pods or an external service. When an application running in a pod requests the service, that request is routed to the endpoints that share the service name.
Define a service named
external-vault
and a corresponding endpoint configured to address theEXTERNAL_VAULT_ADDR
.$ cat > external-vault.yaml <<EOF --- apiVersion: v1 kind: Service metadata: name: external-vault namespace: default spec: ports: - protocol: TCP port: 8200 --- apiVersion: v1 kind: Endpoints metadata: name: external-vault subsets: - addresses: - ip: $EXTERNAL_VAULT_ADDR ports: - port: 8200 EOF
Create the
external-vault
service.$ kubectl apply --filename external-vault.yaml service/external-vault created endpoints/external-vault created
Verify that the
external-vault
service is addressable from within thedevwebapp
pod.$ kubectl exec devwebapp -- curl -s http://external-vault:8200/v1/sys/seal-status | jq
The result displays the status of the Vault server.
Example output:
{ "type": "shamir", "initialized": true, "sealed": false, "t": 1, "n": 1, "progress": 0, "nonce": "", "version": "1.10.3", "migration": false, "cluster_name": "vault-cluster-92405644", "cluster_id": "e95a3cae-1321-e8d3-28f7-b6592ad23dee", "recovery_seal": false, "storage_type": "inmem" }
Optionally, run a
vault status
and compare the results.Create a pod that sets the
VAULT_ADDR
to theexternal-vault
service.$ kubectl apply --filename=pod-devwebapp-through-service.yaml deployment.apps/devwebapp-through-service created
The
devwebapp-through-service
pod addresses Vault through the service instead of the hard-coded network address.Get all the pods in the default namespace.
$ kubectl get pods NAME READY STATUS RESTARTS AGE devwebapp 1/1 Running 0 36m devwebapp-through-service 1/1 Running 0 20s
Wait until the
devwebapp-through-service
pod is running and ready (1/1
).Request content served at
localhost:8080
from within thedevwebapp-through-service
pod.$ kubectl exec devwebapp-through-service -- curl -s localhost:8080 ; echo
The result displays the secret is defined at the path
secret/data/devwebapp/config
.{"password"=>"salsa", "username"=>"giraffe"}
The web application authenticates and requests the secret from the external
Vault server that it found through the external-vault
service.
Install the Vault Helm chart configured to address an external Vault
The Vault Helm chart can deploy only the Vault Agent Injector service configured to target an external Vault. The injector service enables the authentication and secret retrieval for the applications, by adding Vault Agent containers as they are written to the pod automatically it includes specific annotations.
In this section, you will install the Vault Helm chart to run only the injector service, configure Vault's Kubernetes authentication, create a role to access a secret, and patch a deployment.
Install the Vault Helm chart
The Vault Helm chart is able to install only the Vault Agent Injector service.
Add the HashiCorp Helm repository.
$ helm repo add hashicorp https://helm.releases.hashicorp.com "hashicorp" has been added to your repositories
Update all the repositories to ensure
helm
is aware of the latest versions.$ helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "hashicorp" chart repository Update Complete. ⎈Happy Helming!⎈
Install the latest version of the Vault server running in external mode.
$ helm install vault hashicorp/vault \ --set "global.externalVaultAddr=http://external-vault:8200"
The
global.externalVaultAddr
is assigned the address of the Kubernetes service defined in the Deploy service and endpoints to address an external Vault step.The Vault Agent Injector pod is deployed in the default namespace.
Get all the pods in the default namespace.
$ kubectl get pods NAME READY STATUS RESTARTS AGE devwebapp 1/1 Running 0 84m devwebapp-through-service 1/1 Running 0 48m vault-agent-injector-7b6cd469d8-8svg5 1/1 Running 0 15s
Wait until the
vault-agent-injector
pod reports that it is running and ready (1/1
).The Helm chart creates the
vault
service account. The service account secret is necessary to configure Vault's Kubernetes auth method.Describe the
vault
service account.$ kubectl describe serviceaccount vault
The output should resemble the following:
Name: vault Namespace: default Labels: app.kubernetes.io/instance=vault app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=vault helm.sh/chart=vault-0.25.0 Annotations: meta.helm.sh/release-name: vault meta.helm.sh/release-namespace: default Image pull secrets: <none> Mountable secrets: <none> Tokens: <none> Events: <none>
Kubernetes 1.24+ only: The name of the mountable secret is displayed in Kubernetes 1.23. In Kubernetes 1.24+, the token is not created automatically, and you must create it explicitly.
$ cat > vault-secret.yaml <<EOF apiVersion: v1 kind: Secret metadata: name: vault-token-g955r annotations: kubernetes.io/service-account.name: vault type: kubernetes.io/service-account-token EOF
Create a secret.
$ kubectl apply -f vault-secret.yaml secret/vault-token-g955r created
Create a variable named
VAULT_HELM_SECRET_NAME
that stores the secret name.$ VAULT_HELM_SECRET_NAME=$(kubectl get secrets --output=json | jq -r '.items[].metadata | select(.name|startswith("vault-token-")).name')
This command filters the secrets by those that start with
vault-token-
and returns the name of token.Describe the
vault-token
secret.$ kubectl describe secret $VAULT_HELM_SECRET_NAME
The secret displays its metadata and token value.
Example output:
$ kubectl describe serviceaccount vault Name: vault-token-g955r Namespace: default Labels: <none> Annotations: kubernetes.io/service-account.name: vault kubernetes.io/service-account.uid: 7d816e2f-c436-40a1-ab06-163642d00f04 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1111 bytes namespace: 7 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IjV2TjI4UFR0ckJPdHVFNzNYdVZvTXVJdlJlUGZaczFjT0t0bE9CckRDSkUifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJkZWZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6InZhdWx0LXRva2VuLWc5NTVyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InZhdWx0Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiN2Q4MTZlMmYtYzQzNi00MGExLWFiMDYtMTYzNjQyZDAwZjA0Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OmRlZmF1bHQ6dmF1bHQifQ.URxcuvPVxnwrwEkoXvnuBAnZyHqpl-YtgsHMRakhIa1ilLG4P4GsoeBF1fQtS-3n4vi-770q87zp95TpRDINYeTx_waOwk_UjInlxDF81C_ryx9AkZHZcok-iO68tZovMlMXIi7URBImihiUrBuKkmT-Bw_hso80A_HotNjQ_egW3YrQwuEVnWW-08Njwy-BR-OGnQSpeIYJy3EjvdiddLOnwlX4ocTcSukOdCfGt9OlIV_1HtCQcAHRL_3Xpvy8jLb_DpUCWeyDzz5KQ6yfWsD9JFWgEfhW6keE097Mq-OLi35TU_Ov4y_fdxv6laglx9Xq3AipU8M5nFIOyMZpCg
Describe the service account and notice it has been updated with the secret and the token.
$ kubectl describe serviceaccount vault Name: vault Namespace: default Labels: app.kubernetes.io/instance=vault app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=vault helm.sh/chart=vault-0.20.1 Annotations: meta.helm.sh/release-name: vault meta.helm.sh/release-namespace: default Image pull secrets: <none> Mountable secrets: <none> Tokens: vault-token-g955r Events: <none>
Configure Kubernetes authentication
Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.
Enable the Kubernetes authentication method.
$ vault auth enable kubernetes Success! Enabled kubernetes auth method at: kubernetes/
Vault accepts this service token from any client within the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a configured Kubernetes endpoint. To configure it correctly requires capturing the JSON web token (JWT) for the service account, the Kubernetes CA certificate, and the Kubernetes host URL.
Get the JSON web token (JWT) from the secret.
$ TOKEN_REVIEW_JWT=$(kubectl get secret $VAULT_HELM_SECRET_NAME --output='go-template={{ .data.token }}' | base64 --decode)
Retrieve the Kubernetes CA certificate.
$ KUBE_CA_CERT=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.certificate-authority-data}' | base64 --decode)
Retrieve the Kubernetes host URL.
$ KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
Configure the Kubernetes authentication method to use the service account token, the location of the Kubernetes host, its certificate, and its service account issuer name.
You can validate the issuer name of your Kubernetes cluster using this method.
$ vault write auth/kubernetes/config \ token_reviewer_jwt="$TOKEN_REVIEW_JWT" \ kubernetes_host="$KUBE_HOST" \ kubernetes_ca_cert="$KUBE_CA_CERT" \ issuer="https://kubernetes.default.svc.cluster.local"
For a Vault client to read the secret data defined in the Start Vault section requires that the read capability be granted for the path
secret/data/devwebapp/config
.Write out the policy named
devwebapp
that enables theread
capability for secrets at pathsecret/data/devwebapp/config
$ vault policy write devwebapp - <<EOF path "secret/data/devwebapp/config" { capabilities = ["read"] } EOF
Create a Kubernetes authentication role named
devweb-app
.$ vault write auth/kubernetes/role/devweb-app \ bound_service_account_names=internal-app \ bound_service_account_namespaces=default \ policies=devwebapp \ ttl=24h
The role connects the Kubernetes service account,
internal-app
, and namespace,default
, with the Vault policy,devwebapp
. The tokens returned after authentication are valid for 24 hours.
Inject secrets into the pod
The Vault Agent Injector only modifies a pod or deployment if it contains a specific set of annotations.
Use your preferred text editor and examine the pod with annotations
pod-devwebapp-with-annotations.yaml
.pod-devwebapp-with-annotations.yaml
apiVersion: v1 kind: Pod metadata: name: devwebapp-with-annotations labels: app: devwebapp-with-annotations annotations: vault.hashicorp.com/agent-inject: 'true' vault.hashicorp.com/role: 'devweb-app' vault.hashicorp.com/agent-inject-secret-credentials.txt: 'secret/data/devwebapp/config' spec: serviceAccountName: internal-app containers: - name: app image: burtlo/devwebapp-ruby:k8s
These annotations define a partial structure of the deployment schema and are prefixed with
vault.hashicorp.com
.- the
agent-inject
enables the Vault Agent Injector service - the
role
is the Vault Kubernetes authentication role - the
agent-inject-secret-FILEPATH
prefixes the path of the file,credentials.txt
written to the/vault/secrets
directory. The value is the path to the secret defined in Vault.
- the
Create the
devwebapp-with-annotations
pod.$ kubectl apply --filename pod-devwebapp-with-annotations.yaml pod/devwebapp-with-annotations created
Get all the pods in the default namespace.
$ kubectl get pods NAME READY STATUS RESTARTS AGE devwebapp 1/1 Running 0 84m devwebapp-through-service 1/1 Running 0 48m devwebapp-with-annotations 2/2 Running 0 39s vault-agent-injector-7b6cd469d8-8svg5 1/1 Running 0 17m
Wait until the
devwebapp-with-annotations
pod reports that it is running and ready (2/2
).The Vault Agent Injector service created an additional container in the pod that automatically writes the secrets to the
app
container at the filepath/vault/secrets/credentials.txt
.Display the secrets written to the file
/vault/secrets/secret-credentials.txt
on thedevwebapp-with-annotations
pod.$ kubectl exec -it devwebapp-with-annotations -c app -- cat /vault/secrets/credentials.txt
The result displays the unformatted secret data present on the container.
data: map[password:salsa username:giraffe] metadata: map[created_time:2022-06-06T18:26:14.070155Z custom_metadata:<nil> deletion_time: destroyed:false version:1]
Formatting data
A template can be applied to structure this data to meet the needs of the application.
The application in this pod still retrieves the secrets directly, but now that the injector service is deployed and capable of retrieving secrets for the application, future updates can remove that application logic.
Clean up
Use these steps to clean up the tutorial.
Clean up the minikube instance.
$ minikube delete 🔥 Deleting "minikube" in docker ... 🔥 Deleting container "minikube" ... 🔥 Removing /Users/mrken/.minikube/machines/minikube ... 💀 Removed all traces of the "minikube" cluster.
Move up to the parent directory.
$ cd ../
Delete the cloned repository.
$ rm -rf learn-vault-external-kubernetes
Next steps
You deployed Vault external to a Kubernetes cluster and deployed pods that leveraged it as a secrets store. First, through a hard-coded network address. Second, aliased behind a Kubernetes service and endpoint. And finally, through the Vault Helm's chart and the injector service with annotations applied to a deployment. Learn more about the Vault Helm chart by reading the documentation, exploring the project source code, exploring how pods can retrieve secrets through the Vault Injector service via annotations, or secrets mounted on ephemeral volumes. Also try using Kubernetes with HCP Vault Dedicated.
You also can consider using the Vault Secrets Operator instead of the sidecar injector to manage static and dynamic secrets. The Vault Secrets Operator allows a Kubernetes developer to Kubernetes Secrets and behind the scenes, Vault handles Secrets Lifecycle Management.