Vault Agent with Kubernetes
Nearly all requests to Vault must be accompanied by an authentication token. This includes all API requests, as well as via the Vault CLI and other libraries. If you can securely get the first secret from an originator to a consumer, all subsequent secrets transmitted between this originator and consumer can be authenticated with the trust established by the successful distribution and user of that first secret.

Challenge
The applications running in a Kubernetes environment is no exception. Luckily, Vault provides Kubernetes auth method to authenticate the clients using a Kubernetes Service Account Token.

However, the client is still responsible for managing the lifecycle of its Vault tokens. Therefore, the next challenge becomes how to manage the lifecycle of tokens in a standard way without having to write custom logic.
Solution
Vault Agent provides a number of different helper features, specifically addressing the following challenges:
- Automatic authentication
- Secure delivery/storage of tokens
- Lifecycle management of these tokens (renewal & re-authentication)

Note
This tutorial focuses on the demonstration of the Vault Agent Auto-Auth using
the kubernetes
auth method. Vault Helm introduced Agent Sidecar
Injector. Refer to
Injecting Secrets into Kubernetes Pods via Vault Helm
Sidecar for a step-by-step tutorial on the
sidecar usage.
Prerequisites
This lab was tested on macOS using an x86_64 based processor. If you are running macOS on an Apple silicon-based processor, use a x86_64 based Linux virtual machine in your preferred cloud provider.
To perform the tasks described in this tutorial, you need:
- Minikube installed
- A running Vault environment reachable from your Kubernetes environment. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed
Note
For the purpose of demonstration, this tutorial runs Minikube as a Kubernetes environment. If you wish to test against an Azure Kubernetes Service (AKS) cluster instead, follow the steps in the Azure Kubernetes Service Cluster section to create an AKS cluster. If you want to test against a Google Kubernetes Engine (GKE) cluster, refer to the steps described in the Google Kubernetes Engine Cluster section.
Retrieve the additional configuration by cloning the hashicorp/learn-vault-agent repository from GitHub.
This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found in a sub-directory.
Go into the learn-vault-agent/vault-agent-k8s-demo
directory.
Working directory
This tutorial assumes that the remainder of commands are executed in this directory.
Start a Vault server
To go through this tutorial, start a Vault dev server which listens for requests
locally at 0.0.0.0:8200
with root
as the root token ID.
Setting the -dev-listen-address
to 0.0.0.0:8200
overrides the default
address of a Vault dev server (127.0.0.1:8200
) and enables Vault to be
addressable by the Kubernetes cluster and its pods because it binds to a
shared network.
Insecure operation
Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault
CLI to address the Vault server.
Create a service account
Start a Kubernetes cluster running in Minikube.
Wait a couple of minutes for the minikube environment to become fully available.
In Kubernetes, a service account provides an identity for processes that run in a Pod so that the processes can contact the API server. Open the provided
vault-auth-service-account.yaml
file in your preferred text editor and examine its content for the service account definition to be used for this tutorial.vault-auth-service-account.yamlCreate the
vault-auth
service account.
Kubernetes 1.24+ only
The service account generated a secret that is required for configuration automatically in Kubernetes 1.23. In Kubernetes 1.24+, you need to create the secret explicitly.
vault-auth-secret.yamlCreate a
vault-auth-secret
secret.
Configure Kubernetes auth method
Create a read-only policy,
myapp-kv-ro
in Vault.Create some test data at the
secret/myapp
path.Output:
Set the environment variables to point to the running Minikube environment.
Set the
SA_SECRET_NAME
environment variable value to thevault-auth
service account secret.Set the
SA_JWT_TOKEN
environment variable value to the service account JWT used to access the TokenReview APISet the
SA_CA_CRT
environment variable value to the PEM encoded CA cert used to talk to Kubernetes API.Set the
K8S_HOST
environment variable value to minikube IP address.Now, enable and configure the Kubernetes auth method.
Enable the Kubernetes auth method at the default path ("auth/kubernetes").
Tell Vault how to communicate with the Kubernetes (Minikube) cluster.
Output:
You can validate the issuer name of your Kubernetes cluster using this method.
Create a role named,
example
, that maps the Kubernetes Service Account to Vault policies and default token TTL.Output:
Note
The pattern Vault uses to authenticate Pods depends on sharing the JWT token over the network. Given the security model of Vault, this is allowable because Vault is part of the trusted compute base. In general, Kubernetes applications should not share this JWT with other applications, as it allows API calls to be made on behalf of the Pod and can result in unintended access being granted to 3rd parties.
Determine the Vault address
A service bound to all networks on the host, as you configured Vault, is addressable by pods within Minikube's cluster by sending requests to the gateway address of the Kubernetes cluster.
Start a minikube SSH session.
Within this SSH session, retrieve the value of the Minikube host.
Docker networking
The host has a changing IP address (or none if you have no network access). We recommend that you connect to the special DNS name host.docker.internal which resolves to the internal IP address used by the host.
host.docker.internal
. This is for development purpose and will not work in production. For more information, review the documentation for Mac, Windows.Next, retrieve the status of the Vault server to verify network connectivity.
The output displays that Vault is initialized and unsealed. This confirms that pods within your cluster are able to reach Vault given that each pod is configured to use the gateway address.
Next, exit the Minikube SSH session.
Finally, create a variable named
EXTERNAL_VAULT_ADDR
to capture the Minikube gateway address.Verify that the variable contains the IP address you saw when executed in the Minikube shell.
Optional: Verify the Kubernetes auth method configuration
Define a Pod with a container.
The Pod is named
devwebapp
and runs with thevault-auth
service account.Create the
devwebapp
pod in thedefault
namespaceDisplay all the pods in the default namespace.
Wait until the
devwebapp
pod is running and ready (1/1
).Start an interactive shell session on the
devwebapp
pod.Your system prompt is replaced with a new prompt
#
.Note
The prompt within this section is shown as
$
but the commands are intended to be executed within this interactive shell on thedevwebapp
container.Set
KUBE_TOKEN
to the service account token.Authenticate with Vault through the
example
role with theKUBE_TOKEN
.Example output:
Notice that
client_token
is successfully generated andmyapp-kv-ro
policy is attached with the token. Themetadata
displays that its service account name (service_account_name
) isvault-auth
.Lastly, exit the pod.
Start Vault Agent with Auto-Auth
Now that you have verified that the Kubernetes auth method has been configured on the Vault server, it is time to spin up a client Pod which leverages Vault Agent to automatically authenticate with Vault and retrieve a client token.
First, open the provided
configmap.yaml
file in your preferred text editor and review its content.configmap.yaml1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031323334353637383940414243
This creates a Vault Agent configuration file,
vault-agent-config.hcl
. Notice that the Vault Agent Auto-Auth (auto_auth
block) is configured to use thekubernetes
auth method enabled at theauth/kubernetes
path on the Vault server. The Vault Agent will use theexample
role which you created in Step 2.The
sink
block specifies the location on disk where to write tokens. Vault Agent Auto-Authsink
can be configured multiple times if you want Vault Agent to place the token into multiple locations. In this example, thesink
is set to/home/vault/.vault-token
.Finally, the
template
block creates a templated file which retrievesusername
andpassword
values at thesecret/data/myapp/config
path.Create a ConfigMap containing a Vault Agent configuration.
View the created ConfigMap.
An example Pod spec file is provided. Review the provided example Pod spec file,
example-k8s-spec.yaml
.example-k8s-spec.yamlThe example Pod spec (
example-k8s-spec.yaml
) spins up two containers invault-agent-example
pod. Avault
container which runs Vault Agent as an Init Container. And annginx
container exposing port 80.The Vault address,
VAULT_ADDR
, is set to a placeholder valueEXTERNAL_VAULT_ADDR
.Generate the Pod spec with
EXTERNAL_VAULT_ADDR
variable value in its place.Create the
vault-agent-example
pod defined invault-agent-example.yaml
.This takes a minute or so for the pod to become fully up and running.
Verification
In another terminal, launch the Minikube dashboard.
Click Pods under Workloads to verify that
vault-agent-example
pod has been created successfully.Select vault-agent-example to see its details.
In another terminal, port forward all requests made to
http://localhost:8080
to port80
on thevault-agent-example
pod.In a web browser, go to
localhost:8080
Notice that the
username
andpassword
values were successfully read fromsecret/myapp/config
.Optionally, you can view the HTML source.
Azure Kubernetes Service Cluster
If you wish to test the Kubernetes auth method against an Azure Kubernetes Service (AKS) cluster instead of Minikube, you can run Terraform to provision an AKS cluster.
First, follow the instruction in the Terraform documentation to create a service principal.
Be sure to add the following Azure Active Directory Graph API permission to your app.
Store the credentials as Environment Variables.
Subscription ID: Navigate to the Subscriptions blade within the Azure Portal and copy the SUBSCRIPTION ID
Tenant ID: Navigate to the Azure Active Directory > Properties in the Azure Portal, and copy the Directory ID which is your tenant ID
Client ID: Same as the Application ID
Client secret: The password (credential) set on your application
Set your working directory to where the
learn-vault-agent/vault-agent-k8s-demo/terraform-azure
folder is located.Modify
terraform.tfvars.example
and provide Azure credentials:client_id
andclient_secret
, and save it asterraform.tfvars
.Client ID: Same as the Application ID
Client secret: The password (credential) set on your application
Execute the Terraform commands to provision a new AKS cluster.
Pull necessary plugins.
Now, execute the
apply
command to build a new AKS cluster.Note
If you received
Resource group 'education-XXXXX-rg' could not be found
error, runterraform refresh
command.Now, view the information about the AKS cluster.
NOTE: Copy the Kubernetes master address (
https://education.hcp.westus2.azmk8s.io:443
in this example).In the
/vault-agent-k8s-demo/setup-k8s-auth.sh
file, replace Line 48 to point to the AKS cluster address rather thanexport K8S_HOST=$(minikube ip)
. Also, replace Line 54 to point to the correct host address.Example:
setup-k8s-auth.shSet the working directory to where scripts are located (
learn-vault-agent/vault-agent-k8s-demo
).Create a service account,
vault-auth
.Update the
vault-auth
service account based on the definition in thevault-auth-service-account.yaml
file.Setup the Kubernetes auth method on the Vault server.
Resume with the determine the Vault address step and on.
AKS Kubernetes Dashboard
Click on the Monitor containers in the Azure portal.
Click the Enable button so that you can monitor the Kubernetes cluster utilization metrics.
Clean up
When you are done experimenting this demo, execute the following commands to clean up.
Execute the terraform
command.
Remove the Terraform state files.
Google Kubernetes Engine Cluster
Set your working directory to where the
learn-vault-agent/vault-agent-k8s-demo/terraform-gcp
folder is located.Modify
terraform.tfvars.example
and provide GCP credentials:account_file_path
andproject
, and save it asterraform.tfvars
.Example:
terraform.tfvarsExecute the Terraform commands to provision a new GKE cluster.
Pull necessary plugins.
Now, execute the
apply
command to build a new GKE cluster.Connect to the GKE cluster.
Get the cluster information.
NOTE: If you don't have the
gcloud
command line tool installed, follow the online documentation.Copy the Kubernetes master address (
https://198.51.100.24
in this example).In the
/vault-agent-k8s-demo/setup-k8s-auth.sh
file, replace Line 48 to point to the GKE cluster address rather thanexport K8S_HOST=$(minikube ip)
. Also, replace Line 54 to point to the correct host address.Example:
setup-k8s-auth.shSet the working directory to where scripts are located (
learn-vault-agent/vault-agent-k8s-demo
).Create a service account,
vault-auth
.Update the
vault-auth
service account with definition provided in thevault-auth-service-account.yaml
file.Setup the Kubernetes auth method on the Vault server.
Resume with the determine the Vault address step and on.
Help and reference
- Blog post: Why Use the Vault Agent for Secrets Management?
- Video: Streamline Secrets Management with Vault Agent and Vault 0.11
- Secure Introduction of Vault Clients
- Vault Agent Auto-Auth
- Kubernetes Auth Method
- Kubernetes Auth Method (API)
- Vault Kubernetes Workshop on Google Cloud Platform
- Kubernetes Tutorials
- Consul Template