Injecting Secrets into Kubernetes Pods via Vault Agent Containers
Deploying applications that act as secret consumers of Vault require the application to:
- Authenticate and acquire a client token.
- Manage the lifecycle of the token.
- Retrieve secrets from Vault.
- Manage the leases of any dynamic secrets.
Vault Agent takes responsibility for these tasks and enables your applications to remain unaware of Vault. However, this introduces a new requirement that deployments install and configure Vault Agent alongside the application as a sidecar.
The Vault Helm chart enables you to run Vault and the Vault Agent Injector service. This injector service leverages the Kubernetes mutating admission webhook to intercept pods that define specific annotations and inject a Vault Agent container to manage these secrets. This is beneficial because:
- Applications remain Vault unaware as the secrets are stored on the file-system in their container.
- Existing deployments require no change; as annotations can be patched.
- Access to secrets can be enforced via Kubernetes service accounts and namespaces
In this tutorial, you setup Vault and this injector service with the Vault Helm chart. Then you will deploy several applications to demonstrate how this new injector service retrieves and writes these secrets for the applications to use.
Prerequisites
This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, and additional configuration to bring it all together.
This tutorial was last tested 23 Apr 2021 on a macOS 11.2.3 using this configuration.
Docker version.
Minikube version.
Helm version.
These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.
First, follow the directions to install Minikube, including VirtualBox or similar.
Next, install kubectl CLI and helm CLI.
Next, retrieve the web application and additional configuration by cloning the hashicorp/vault-guides repository from GitHub.
This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found in a sub-directory.
Go into the
vault-guides/operations/provision-vault/kubernetes/minikube/vault-agent-sidecar
directory.
Working directory: This tutorial assumes that the remainder of commands are executed in this directory.
Start Minikube
Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters locally inside Virtual Machines (VM) on your system.
Start a Kubernetes cluster.
The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.
Verify the status of the Minikube cluster.
Additional waiting: Even if this last command completed successfully, you may have to wait for Minikube to be available. If an error is displayed, try again after a few minutes.
The host, kubelet, and apiserver report that they are running. The kubectl
, a
command line interface (CLI) for running commands against Kubernetes cluster, is
also configured to communicate with this recently started cluster.
Minikube provides a visual representation of the status in a web-based dashboard. This interface displays the cluster activity in a visual interface that can assist in delving into the issues affecting it.
In another terminal, launch the minikube dashboard.
The operating system's default browser opens and displays the dashboard.
Install the Vault Helm chart
The recommended way to run Vault on Kubernetes is via the Helm chart. Helm is a package manager that installs and configures all the necessary components to run Vault in several different modes. A Helm chart includes templates that enable conditional and parameterized execution. These parameters can be set through command-line arguments or defined in YAML.
Add the HashiCorp Helm repository.
Update all the repositories to ensure helm
is aware of the latest versions.
Install the latest version of the Vault server running in development mode.
The Vault pod and Vault Agent Injector pod are deployed in the default namespace.
Display all the pods in the default namespace.
The vault-0
pod runs a Vault server in development mode. The
vault-agent-injector
pod performs the injection based on the annotations
present or patched on a deployment.
Development mode: Running a Vault server in development is automatically initialized and unsealed. This is ideal in a learning environment but NOT recommended for a production environment.
Wait until the vault-0
pod and vault-agent-injector
pod are running and
ready (1/1
).
Set a secret in Vault
The applications that you deploy in the Inject secrets into the
pod section expect Vault to store a username and
password stored at the path internal/database/config
. To create this secret
requires that a key-value secret
engine is enabled and a
username and password is put at the specified path.
Start an interactive shell session on the vault-0
pod.
Your system prompt is replaced with a new prompt / $
. Commands issued at this
prompt are executed on the vault-0
container.
Enable kv-v2 secrets at the path internal
.
Learn more: This tutorial focuses on Vault's integration with Kubernetes and not interacting the key-value secrets engine. For more information refer to the Static Secrets: Key/Value Secret tutorial.
Create a secret at path internal/database/config
with a username
and
password
.
Verify that the secret is defined at the path internal/database/config
.
The secret is ready for the application.
Lastly, exit the vault-0
pod.
Configure Kubernetes authentication
Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token. This token is provided to each pod when it is created.
Start an interactive shell session on the vault-0
pod.
Your system prompt is replaced with a new prompt / $
. Commands issued at this
prompt are executed on the vault-0
container.
Enable the Kubernetes authentication method.
Vault accepts a service token from any client in the Kubernetes cluster. During authentication, Vault verifies that the service account token is valid by querying a token review Kubernetes endpoint.
Configure the Kubernetes authentication method to use the location of the Kubernetes API.
For the best compatibility with recent Kubernetes versions, ensure you are using Vault v1.9.3 or greater.
Successful output from the command resembles this example:
The environment variable KUBERNETES_PORT_443_TCP_ADDR
is defined and references
the internal network address of the Kubernetes host.
For a client to read the secret data defined at internal/database/config
,
requires that the read capability be granted for the path
internal/data/database/config
. This is an example of a
policy. A policy
defines a set of capabilities.
Write out the policy named internal-app
that enables the read
capability
for secrets at path internal/data/database/config
.
Create a Kubernetes authentication role named internal-app
.
Successful output from the command resembles this example:
The role connects the Kubernetes service account, internal-app
, and namespace,
default
, with the Vault policy, internal-app
. The tokens returned after
authentication are valid for 24 hours.
Lastly, exit the vault-0
pod.
Define a Kubernetes service account
The Vault Kubernetes authentication role defined a Kubernetes service account
named internal-app
.
Get all the service accounts in the default namespace.
Create a Kubernetes service account named internal-app
in the default
namespace.
Verify that the service account has been created.
The name of the service account here aligns with the name assigned to the
bound_service_account_names
field when the internal-app
role was created.
Launch an application
We've created a sample application, published it to DockerHub, and created a Kubernetes deployment that launches this application.
Display the deployment for the orgchart
application.
The name of this deployment is orgchart
. The
spec.template.spec.serviceAccountName
defines the service account
internal-app
to run this container.
Apply the deployment defined in deployment-orgchart.yaml
.
Get all the pods in the default namespace.
The orgchart pod is displayed here as the pod prefixed with orgchart
.
Additional waiting: The deployment of the pod requires the retrieval
of the application container from Docker Hub. This
displays the STATUS of ContainerCreating
. The pod reports that it is not
ready (0/1
).
The Vault-Agent injector looks for deployments that define specific annotations.
None of these annotations exist in the current deployment. This means that
no secrets are present on the orgchart
container in the orgchart
pod.
Verify that no secrets are written to the orgchart
container in the
orgchart
pod.
The output displays that there is no such file or directory named
/vault/secrets
:
Inject secrets into the pod
The deployment is running the pod with the internal-app
Kubernetes service
account in the default namespace. The Vault Agent Injector only modifies a
deployment if it contains a specific set of annotations. An existing deployment
may have its definition patched to include the necessary annotations.
Display the deployment patch patch-inject-secrets.yaml
.
These
annotations
define a partial structure of the deployment schema and are prefixed with
vault.hashicorp.com
.
agent-inject
enables the Vault Agent Injector servicerole
is the Vault Kubernetes authentication roleagent-inject-secret-FILEPATH
prefixes the path of the file,database-config.txt
written to the/vault/secrets
directory. The value is the path to the secret defined in Vault.
Patch the orgchart
deployment defined in patch-inject-secrets.yaml
.
A new orgchart
pod starts alongside the existing pod. When it is ready the
original terminates and removes itself from the list of active pods.
Get all the pods in the default namespace.
Wait until the re-deployed orgchart
pod reports that
it is
Running
and ready (2/2
).
This new pod now launches two containers. The application container, named
orgchart
, and the Vault Agent container, named vault-agent
.
Display the logs of the vault-agent
container in the new orgchart
pod.
Vault Agent manages the token lifecycle and the secret retrieval. The secret is
rendered in the orgchart
container at the path
/vault/secrets/database-config.txt
.
Finally, display the secret written to the orgchart
container.
The unformatted secret data is present on the container:
Apply a template to the injected secrets
The structure of the injected secrets may need to be structured in a way for an application to use. Before writing the secrets to the file system a template can structure the data. To apply this template a new set of annotations need to be applied.
Display the annotations file that contains a template definition.
This patch contains two new annotations:
agent-inject-status
set toupdate
informs the injector reinject these values.agent-inject-template-FILEPATH
prefixes the file path. The value defines the Vault Agent template to apply to the secret's data.
The template formats the username and password as a PostgreSQL connection string.
Apply the updated annotations.
Get all the pods in the default namespace.
Wait until the re-deployed orgchart
pod reports that
it is
Running
and ready (2/2
).
Finally, display the secret written to the orgchart
container in the
orgchart
pod.
The secrets are rendered in a PostgreSQL connection string is present on the container:
Pod with annotations
The annotations may patch these secrets into any deployment. Pods require that the annotations be included in their initial definition.
Display the pod definition for the payroll
application.
Apply the pod defined in pod-payroll.yaml
.
Get all the pods in the default namespace.
Wait until the payroll
pod reports that
it is
Running
and ready (2/2
).
Finally, display the secret written to the payroll
container in the payroll
pod.
The secrets are rendered in a PostgreSQL connection string is present on the container:
Secrets are bound to the service account
Pods run with a Kubernetes service account other than the ones defined in the Vault Kubernetes authentication role are NOT able to access the secrets defined at that path.
Display the deployment and service account for the website
application.
Apply the deployment and service account defined in deployment-website.yaml
.
Get all the pods in the default namespace.
The website
deployment creates a pod but it is NEVER ready.
Display the logs of the vault-agent-init
container in the website
pod.
The initialization process failed because the service account name is not authorized:
The service account, external-app
is not assigned to any Vault Kubernetes authentication role. This failure to authenticate causes the deployment to fail initialization.
Display the deployment patch patch-website.yaml
.
The patch modifies the deployment definition to use the service account
internal-app
. This Kubernetes service account is authorized by the Vault
Kubernetes authentication role.
Patch the website
deployment defined in patch-website.yaml
.
Get all the pods in the default namespace.
Wait until the website
pod reports that it is
Running
and ready (2/2
).
Finally, display the secret written to the website
container in the website
pod.
The secrets are rendered in a PostgreSQL connection string is present on the container:
Vault Kubernetes Roles: Alternatively, you can define a new Vault Kubernetes role, that enables the original service account access, and patch the deployment.
Secrets are bound to the namespace
Pods run in a namespace other than the ones defined in the Vault Kubernetes authentication role are NOT able to access the secrets defined at that path.
Create the offsite
namespace.
Set the current context to the offsite namespace.
Create a Kubernetes service account named internal-app
in the offsite
namespace.
Display the deployment for the issues
application.
Apply the deployment defined in deployment-issues.yaml
.
Get all the pods in the offsite namespace.
Current context: The same command is issued but the results are different because you are now in a different namespace.
The issues
deployment creates a pod but it is NEVER ready.
Display the logs of the vault-agent-init
container in the issues
pod.
The initialization process fails because the namespace is not authorized:
The
namespace, offsite
is not assigned to any Vault Kubernetes authentication
role. This failure to authenticate causes the deployment to fail initialization.
Start an interactive shell session on the vault-0
pod in the default
namespace.
Your system prompt is replaced with a new prompt / $
. Commands issued at this
prompt are executed on the vault-0
container.
Create a Kubernetes authentication role named offsite-app
.
Successful output from the command resembles this example:
Exit the vault-0
pod.
Display the deployment patch patch-issues.yaml
.
The patch performs an update to set the vault.hashicorp.com/role
to the
Vault Kubernetes role offsite-app
.
Patch the issues
deployment defined in patch-issues.yaml
.
A new issues
pod starts alongside the existing pod. When it is ready the
original terminates and removes itself from the list of active pods.
Get all the pods in the offsite namespace.
Wait until the re-deployed issues
pod reports that
it is
Running
and ready (2/2
).
Finally, display the secret written to the issues
container in the issues
pod.
The secrets are rendered in a PostgreSQL connection string is present on the container:
Next Steps
You launched Vault and the injector service with the Vault Helm chart. Learn more about the Vault Helm chart by reading the documentation, exploring the project source code, reading the blog post announcing the "Injecting Vault Secrets into Kubernetes Pods via a Sidecar", or the documentation for Agent Sidecar Injector
Then you deployed several applications to demonstrate how this new injector service retrieves and writes these secrets for the applications to use. Explore how pods can retrieve them directly via network requests or secrets mounted on ephemeral volumes.