The Vault Secrets Operator on Kubernetes
Challenge
Vault offers a complete solution for secrets lifecycle management, but developers and operators would be required to learn a completely new tool. Instead, developers want a cloud native way to access the secrets through Kubernetes and not be required to understand Vault in great depth. Through the Vault Secrets Operator, secrets are accessed as native Kubernetes secrets, but with the advantage of being managed by HashiCorp Vault.
Solution
A Kubernetes operator is a software extension that uses custom resources to manage applications hosted on Kubernetes.
The Vault Secrets Operator is a Kubernetes operator that syncs secrets between Vault and Kubernetes natively without requiring the users to learn details of Vault use.
Currently, Vault secrets operator is available and supports kv-v1 and kv-v2, TLS certificates in PKI and full range of static and dynamic secrets.
The Vault Secrets Operator syncs the secrets between Vault and the Kubernetes secrets in a specified namespace. Within that namespace, applications have access to the secrets. The secrets are still managed by Vault, but accessed through the standard way on Kubernetes.
Prerequisites
- Recent version of Vault binary. Please see Supported Vault versions.
- Docker
- Kubernetes command-line interface (CLI)
- Minikube
- k9s
- Helm CLI
Install supporting tools
This tutorial was last tested 21 May 2023 on a macOS 13.3.1 using the following software versions.
Clone the GitHub repository
Clone the repository at learn-vault-secrets-operator.
Move into that folder.
Start minikube
Minikube allows you to run a miniature Kubernetes cluster on your local machine.
Create a minikube cluster.
The output should resemble the following:
You have created a Kubernetes cluster running on Docker.
Install Vault cluster
Using Helm install Vault on a local instance of minikube. Vault is installed it's own virtual cluster called a namespace.
If you have not already, add the HashiCorp Repo.
In order to have the latest version of the HashiCorp Helm charts, update the repo.
Details of the output might differ, the important thing is the Update Complete message.
Determine the latest version of Vault.
Vault Secrets Operator supports for the latest three versions of Vault. Please see Supported Vault versions for details.
Using the YAML file in
/vault
install Vault on your Minikube clusterThe output should resemble the following:
Configure Vault
Here you connect to the Vault instance on minikube, enable and configure Kubernetes authentication, KV secrets engine, a role and policy for Kubernetes, and create a static secret.
Connect to the Vault instance. Until you
exit
you will be executing from inside the Vault instance.Enable the Kubernetes auth method.
Configure the auth method.
The output should resemble the following:
Enable the kv v2 Secrets Engine.
Create a read only policy.
The output should resemble the following:
Create a role in Vault to enable access to secret.
The output should resemble the following:
Notice that the bound_service_account_namespaces is app, limiting where the secret is synced to.
Create a secret.
Exit the Vault instance.
Install the Vault Secrets Operator
Use helm to deploy the Vault Secrets Operator.
Examine the file
vault/vault-operator-values.yaml
:This sets up the basics of the operator.
Deploy and sync a secret
In this section you set up a namespace with a Kubernetes secret. That secret is managed through the Vault Secrets Operator.
Create a namespace called app on your Kubernetes cluster.
Set up the Kubernetes authentication for the secret.
Create the secret names
secretkv
in the app namespace.
Rotate the static secret
In this section you use the k9s tool to look at the secret. Then you use Vault to manually rotate the secret, and k9s again to verify that within Kubernetes the secret is rotated.
Open a new terminal and start up
k9s
.If not already displayed, list the namespaces by typing
:ns
.Choose the app namespace and press
enter
.This area is blank, so type in
:secrets
and hit enter.Now the secrets named secretkv is displayed, highlight it.
Display the secret by pressing the
x
key.In the original terminal, connect to the Vault instance.
Rotate the secret.
Return to
k9s
, and escape back to the secret page and hitx
again to display the updated secret.The secret has changed, and now will be different than noted earlier.
At the first window, exit Vault.
Dynamic secrets
Manually rotating secrets is cumbersome and prone to human error. Enter dynamic secrets.
Now you will create a dynamic secret with the database secrets engine. Dynamic secrets lifecycle is managed by Vault and will be automatically rotated. The lifecycle management includes deleting and recreating the secrets regularly. In this section we will use the Vault Secrets Operator to rotate the Kubernetes secrets every 1 minute.
Install Postgres pod
You will create a pod and install Postgres into it. This is used later to generate credentials for the database secrets engine created in the next section.
Create a namespace for the postgres pod.
Add the bitnami repository to your local Helm.
Install postgres.
The output should resemble the following:
Export the Postgres database password as an environment variable.
Make a note of the
POSTGRES_PASSWORD
value. This is an admin password for the Postgres cluster and you will use this value to set up the database secrets engine later.
Setup Postgres
Connect to the Vault instance, and set up a database secrets engine in Vault with a corresponding role and policy.
Connect to the Vault instance.
Enable an instance of the Database Secrets Engine.
Configure the Database Secrets Engine, replacing the
<<POSTGRES_PASSWORD>>
with the password from above.The output should resemble the following:
Create a role for the postgres pod.
The output should resemble the following:
Note
In production you do not want
default_ttl="1m"
ormax_ttl="1m"
set. This is too short and a longer TTL should be used. It is only done this way in this tutorial for demonstration purposes.Create the
demo-auth-policy-db
policy.The output should resemble the following:
Transit encryption
Vault Secrets Operator can be configured to maintain an internal, encrypted persistent cache of client tokens. This is particularly helpful for being able to transparently renew leases for dynamic secrets should the operator restart. Without using the client cache, the operator would need to fetch new client tokens on restarts, and re-issue credentials for dynamic secrets, causing downtime for applications. With transit encryption configured and enabled, the client token cache is end-to-end encrypted using Transit Encryption so that the persisted tokens cannot be accessed.
Enable an instance of the Transit Secrets Engine at the path
demo-transit
.Create a encryption key.
Create a policy for the operator role to access the encryption key.
The output should resemble the following:
Create Kubernetes auth role for the operator.
The output should resemble the following:
Setup dynamic secrets
A variety of different clients can be used to generate Dynamic secrets but for this tutorial, Vault will call Postgres to generate access credentials. This allows the application to access Postgres, and Vault to rotate them as needed. The details of this depends on how the Secret Engine is set up, and are managed by Vault.
See Dynamic Secrets for another example of how this is done.
Create a role to allow access to the Kubernetes secrets engine for the demo-ns
.
Create a new role for the dynamic secret.
The output should resemble the following:
Exit the shell.
Create the application
This section creates a namespace demo-ns
that has a dynamic secret available, and an application to access it. By using this secret the application has access to Postgres. Later you will be k9s
to examine the variable and verify that it automatically rotates.
Create a new namespace.
Create the app, Vault connection, authentication, service account and corresponding secrets.
Note
If you receive warning messages resembling the following:
Warning: resource
vaultauths/default
is missing thekubectl.kubernetes.io/last-applied-configuration
annotation which is required by kubectl apply. Thekubectl apply
command should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.This warning message is expected. No preexisting information for annotation
kubectl.kubernetes.io/last-applied-configuration
exists, therefore this first configuration and will be used to compare for changes later.
Examine the dynamic secret
Using k9s
examine the dynamic secret, and confirm the secrets changes every minute.
Go back to the window with k9s and display all the available namespaces with the
:ns
command.Find and choose the demo-ns by highlighting it and hitting return.
Take a look at the secrets in the demo-ns pod by typing
:secrets
and enter.Choose the first secret displayed and type in
x
.The secrets should resemble the following, remember to note down the password displayed
Exit screen with escape key and wait a minimum of 60 seconds, and check the values once again with
x
. Notice the password has automatically changed.
Clean up
Delete the minikube cluster.
Additional discussion
A Kubernetes operator is a software extension that uses custom resources to manage applications hosted on Kubernetes. The Vault Secret Operator leverages the HashiCorp Vault as a complete secrets management solution.
Secrets exist within Namespaces, which are virtual clusters with a Kubernetes Cluster. The secrets operator allows you to administer the secrets through Vault, but access as a Kubernetes native structure.
Kubernetes cluster administrators interested in using the the Vault Secrets Operator to rotate Dynamic secrets are encouraged to look at the demo include with the source for the Vault Secrets Operator.
In this tutorial you begun to learn about the Vault Secrets Operator by set up a Minikube cluster with a Vault instance, Vault Secrets Operator controller and created a secret in a namespace called app. Then you displayed the secret in k9s, and used Vault to rotate the secret. You then validated the change in the Vault secret value was reflected in the Kubernetes secret.
In the Dynamic Secrets section you set up a PostGresSQL pod, dummy application and created a dynamic secret. Then, using k9s you watched the secret rotate automatically.
The Vault Secrets Operator is a first class Kubernetes operator pattern for use with HashiCorp Vault responsible for syncing Vault secrets to Kubernetes Secrets natively.
Features of the Vault Secrets Operator include support for all secrets engines and Kubernetes, AWS, JWT and AppRole authentication to Vault.