The Vault Secrets Operator on Kubernetes

16min
|
Vault
Interactive

Challenge

Vault offers a complete solution for secrets lifecycle management, but developers and operators are required to learn a completely new tool. Instead, developers want a cloud native way to access the secrets through Kubernetes and not be required to understand Vault in great depth. Through the Vault Secrets Operator, secrets are accessed as native Kubernetes secrets, but with the advantage of being managed by HashiCorp Vault.

Solution

A Kubernetes operator is a software extension that uses custom resources to manage applications hosted on Kubernetes.

The Vault Secrets Operator is a Kubernetes operator that syncs secrets between Vault and Kubernetes natively without requiring the users to learn details of Vault use.

Currently, Vault secrets operator is available and supports kv-v1 and kv-v2, TLS certificates in PKI and full range of static and dynamic secrets.

Secrets Operator Diagram

The Vault Secrets Operator syncs the secrets between Vault and the Kubernetes secrets in a specified namespace. Within that namespace, applications have access to the secrets. The secrets are still managed by Vault, but accessed through the standard way on Kubernetes.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Prerequisites

Recent version of Vault binary. Please see Supported Vault versions.
Docker
Helm CLI
k9s
Kubernetes command-line interface (CLI)
Minikube
Recent version of the Vault binary installed. Refer to the Getting Started tutorial.

Install supporting tools

This tutorial was last tested 21 May 2023 on a macOS 13.3.1 using the following software versions.

Docker version

$ docker version
Client:
Cloud integration: v1.0.25
Version:           20.10.16
## ...

Helm version

$ helm version
version.BuildInfo{Version:"v3.12.0", GitCommit:"c9f554d75773799f72ceef38c51210f1842a1dea", GitTreeState:"clean", GoVersion:"go1.20.4"}

k9s version

$ k9s version
____  __.________
|    |/ _/   __   \______
|      < \____    /  ___/
|    |  \   /    /\___ \
|____|__ \ /____//____  >
      \/            \/

Version:    0.27.3
Commit:     7c76691c389e4e7de29516932a304f7029307c6d
Date:       n/a

Kubectl version

$ kubectl version --short
Client Version: v1.27.1
Kustomize Version: v5.0.1
Server Version: v1.26.3

Minikube version

$ minikube version
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

Clone the GitHub repository

Clone the repository at learn-vault-secrets-operator.

$ git clone https://github.com/hashicorp-education/learn-vault-secrets-operator.git
Cloning into 'learn-vault-secrets-operator'...
remote: Enumerating objects: 15, done.
remote: Counting objects: 100% (15/15), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 15 (delta 0), reused 15 (delta 0), pack-reused 0
Receiving objects: 100% (15/15), done.

Move into that folder.
```
$ cd learn-vault-secrets-operator
```

Start minikube

Minikube allows you to run a miniature Kubernetes cluster on your local machine.

Create a minikube cluster.

$ minikube start

The output should resemble the following:

😄  minikube v1.30.1 on Darwin 13.4 (arm64)
✨  Automatically selected the docker driver
📌  Using Docker Desktop driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
🐳  Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...
   ▪ Generating certificates and keys ...
   ▪ Booting up control plane ...
   ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
   ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

You have created a Kubernetes cluster running on Docker.

Install Vault cluster

Diagram for k8s cluster

Using Helm install Vault on a local instance of minikube. Vault is installed it's own virtual cluster called a namespace.

If you have not already, add the HashiCorp Repo.

$ helm repo add hashicorp https://helm.releases.hashicorp.com

In order to have the latest version of the HashiCorp Helm charts, update the repo.

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
...Successfully got an update from the "open" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

Details of the output might differ, the important thing is the Update Complete message.

Determine the latest version of Vault.

$ helm search repo hashicorp/vault
NAME            CHART VERSION   APP VERSION     DESCRIPTION
hashicorp/vault 0.23.0          1.13.1          Official HashiCorp Vault Chart

Vault Secrets Operator supports for the latest three versions of Vault. Please see Supported Vault versions for details.

Using the YAML file in /vault install Vault on your Minikube cluster

$ helm install vault hashicorp/vault -n vault --create-namespace --values vault/vault-values.yaml

The output should resemble the following:

NAME: vault
LAST DEPLOYED: Fri Mar 31 09:37:42 2023
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/

Your release is named vault. To learn more about the release, try:
$ helm status vault
$ helm get manifest vault

Configure Vault

setup Vault cluster

Here you connect to the Vault instance on minikube, enable and configure Kubernetes authentication, KV secrets engine, a role and policy for Kubernetes, and create a static secret.

Connect to the Vault instance. Until you exit you will be executing from inside the Vault instance.
```
$ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
```

Enable the Kubernetes auth method.

$ vault auth enable -path demo-auth-mount kubernetes
Success! Enabled kubernetes auth method at: kubernetes/

Configure the auth method.

$ vault write auth/demo-auth-mount/config \
   kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"

The output should resemble the following:

Success! Data written to: auth/demo-auth-mount/config

Enable the kv v2 Secrets Engine.

$ vault secrets enable -path=kvv2 kv-v2
Success! Enabled the kv-v2 secrets engine at: kvv2/

Create a read only policy.

$ vault policy write dev - <<EOF
path "kvv2/*" {
   capabilities = ["read"]
}
EOF

The output should resemble the following:

Success! Uploaded policy: dev

Create a role in Vault to enable access to secret.

$ vault write auth/demo-auth-mount/role/role1 \
   bound_service_account_names=default \
   bound_service_account_namespaces=app \
   policies=dev \
   audience=vault \
   ttl=24h

The output should resemble the following:

 Success! Data written to: auth/demo-auth-mount/role/role1

Notice that the bound_service_account_namespaces is app, limiting where the secret is synced to.

Create a secret.

$ vault kv put kvv2/webapp/config username="static-user" password="static-password"
===== Secret Path =====
kvv2/data/webapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2023-04-03T16:35:56.1103993Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Exit the Vault instance.
```
$ exit
```

Install the Vault Secrets Operator

Vault Secrets Operator

Use helm to deploy the Vault Secrets Operator.

$ helm install vault-secrets-operator hashicorp/vault-secrets-operator -n vault-secrets-operator-system --create-namespace --values vault/vault-operator-values.yaml
NAME: vault-secrets-operator
LAST DEPLOYED: Fri Mar 31 10:00:29 2023
NAMESPACE: vault-secrets-operator-system
STATUS: deployed
REVISION: 1

Examine the file vault/vault-operator-values.yaml:
```
$ cat vault/vault-operator-values.yaml
defaultVaultConnection:
   enabled: true
   address: "http://vault.vault.svc.cluster.local:8200"
   skipTLSVerify: false
...
```
The defaultVaultConnection sets up the default connection to Vault. This is used if no other connection is set.
- address is the address on the Kubernetes cluster.
- skipTLSVerify set to false enables TLS certificate verification.

Deploy and sync a secret

Create secret in App pod

In this section you set up a namespace with a Kubernetes secret. That secret is managed through the Vault Secrets Operator.

Create a namespace called app on your Kubernetes cluster.
```
$ kubectl create ns app
namespace/app created
```

Set up the Kubernetes authentication for the secret.

$ kubectl apply -f vault/vault-auth-static.yaml
vaultauth.secrets.hashicorp.com/static-auth created

Create the secret names secretkv in the app namespace.

$ kubectl apply -f vault/static-secret.yaml
vaultstaticsecret.secrets.hashicorp.com/vault-kv-app created

Rotate the static secret

In this section you use the k9s tool to display the secret. Then you use Vault to manually rotate the secret, and k9s again to verify that within Kubernetes the secret is rotated.

Open a new terminal and start up k9s.
```
$ k9s
```
If not already displayed, list the namespaces by typing :ns.
Use the up and down arrows to choose the app namespace and press enter.
This area is blank, so type in :secrets and hit enter.
Now the secrets named secretkv is displayed, highlight it.
Display the secret by pressing the x key.

In the original terminal, connect to the Vault instance.

$ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
/ $

Rotate the secret.

$ vault kv put kvv2/webapp/config username="static-user2" password="static-password2"
===== Secret Path =====
kvv2/data/webapp/config

======= Metadata =======
Key                Value
---                -----
created_time       2023-04-03T16:40:31.274411719Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

Return to k9s, and escape back to the secret page and hit x again to display the updated secret.
The secret has changed, and now will be different than noted earlier.
At the first window, exit Vault.
```
$ exit
```

Dynamic secrets

Manually rotating secrets is cumbersome and prone to human error. Enter dynamic secrets.

Now you will create a dynamic secret with the database secrets engine. Dynamic secrets lifecycle is managed by Vault and will be automatically rotated. The lifecycle management includes deleting and recreating the secrets regularly. In this section we will use the Vault Secrets Operator to rotate the Kubernetes secrets every 1 minute.

Install Postgres pod

install PostgreSQL

You will create a pod and install Postgres into it. This is used later to generate credentials for the database secrets engine created in the next section.

Create a namespace for the postgres pod.
```
$ kubectl create ns postgres
```

Add the bitnami repository to your local Helm.

$ helm repo add bitnami https://charts.bitnami.com/bitnami

Install postgres.

$ helm upgrade --install postgres bitnami/postgresql --namespace postgres --set auth.audit.logConnections=true  --set auth.postgresPassword=secret-pass

The output should resemble the following:

Release "postgres" does not exist. Installing it now.
NAME: postgres
LAST DEPLOYED: Wed Jun  7 05:56:05 2023
NAMESPACE: postgres
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: postgresql
CHART VERSION: 12.5.6
APP VERSION: 15.3.0

Setup Postgres

Connect to the Vault instance, and set up a database secrets engine in Vault with a corresponding role and policy.

Connect to the Vault instance.

$ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh

Enable an instance of the Database Secrets Engine.

$ vault secrets enable -path=demo-db database
Success! Data written to: demo-db/config/demo-db

Configure the Database Secrets Engine.

$ vault write demo-db/config/demo-db \
   plugin_name=postgresql-database-plugin \
   allowed_roles="dev-postgres" \
   connection_url="postgresql://{{username}}:{{password}}@postgres-postgresql.postgres.svc.cluster.local:5432/postgres?sslmode=disable" \
   username="postgres" \
   password="secret-pass"

The output should resemble the following:

Success! Data written to: demo-db/config/demo-db

Create a role for the postgres pod.

$ vault write demo-db/roles/dev-postgres \
   db_name=demo-db \
   creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
      GRANT ALL PRIVILEGES ON DATABASE postgres TO \"{{name}}\";" \
   backend=demo-db \
   name=dev-postgres \
   default_ttl="1m" \
   max_ttl="1m"

The output should resemble the following:

Success! Data written to: demo-db/roles/dev-postgres

Note

In production you do not want default_ttl="1m" or max_ttl="1m" set. This is too short and a longer TTL should be used. It is only done this way in this tutorial for demonstration purposes.

Create the demo-auth-policy-db policy.

$ vault policy write demo-auth-policy-db - <<EOF
path "demo-db/creds/dev-postgres" {
   capabilities = ["read"]
}
EOF

The output should resemble the following:

Success! Uploaded policy: demo-auth-policy-db

Disconnect to the Vault instance.
```
$ exit
```

Transit encryption

Vault Secrets Operator can be configured to maintain an internal, encrypted persistent cache of client tokens. This is particularly helpful for being able to transparently renew leases for dynamic secrets should the operator restart. Without using the client cache, the operator would need to fetch new client tokens on restarts, and re-issue credentials for dynamic secrets, causing downtime for applications. With transit encryption configured and enabled, the client token cache is end-to-end encrypted using Transit Encryption so that the persisted tokens cannot be accessed.

The helm chart in vault/vault-operator-values.yaml already set up the Vault secrets operator for client token cache.
```
$ cat vault/vault-operator-values.yaml
...
controller:
   manager:
      clientCache:
         persistenceModel: direct-encrypted
         storageEncryption:
            enabled: true
            mount: demo-auth-mount
            keyName: vso-client-cache
            transitMount: demo-transit
            kubernetes:
               role: auth-role-operator
               serviceAccount: demo-operator
```
Much of the remaining tutorial is setting up Vault to these specifications already present in vault-operator-values.yaml.
The clientCache section has settings for the cache to use the direct encrypted mode.
The storageEncryption sub-section specifies details of the transit secrets engine, authentication that is used by the transit secrets engine.
- transitMount sets the transit secrets engine mount named demo-transit.
- mount sets the Kubernetes auth method engine named demo-auth-transit used by the demo-transit.
- keyName specifies a key named vso-client-cache used for encrypt/decrypt operations.
- kubernetes sub-section configures the Vault Kubernetes auth method, specifying the role and the service account.
Refer to Vault Secrets Operator helm chart section on storageEncryption documentation for details.

Connect back to the Vault instance.

$ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh

Enable an instance of the Transit Secrets Engine at the path demo-transit.

$ vault secrets enable -path=demo-transit transit
Success! Enabled the transit secrets engine at: demo-transit/

Create a encryption key.

$ vault write -force demo-transit/keys/vso-client-cache
Success! Data written to: demo-transit/keys/vso-client-cache

Create a policy for the operator role to access the encryption key.

$ vault policy write demo-auth-policy-operator - <<EOF
path "demo-transit/encrypt/vso-client-cache" {
   capabilities = ["create", "update"]
}
path "demo-transit/decrypt/vso-client-cache" {
   capabilities = ["create", "update"]
}
EOF

The output should resemble the following:

Success! Uploaded policy: demo-auth-policy-operator

Create Kubernetes auth role for the operator.

$ vault write auth/demo-auth-mount/role/auth-role-operator \
   bound_service_account_names=demo-operator \
   bound_service_account_namespaces=vault-secrets-operator-system \
   token_ttl=0 \
   token_period=120 \
   token_policies=demo-auth-policy-db \
   audience=vault

The output should resemble the following:

Success! Data written to: auth/demo-auth-mount/role/auth-role-operator

Setup dynamic secrets

Vault includes a number of dynamic secrets engines capable of generating temporary credentials for various applications. In this tutorial, we will use Vault's dynamic secrets engine for Postgres to generate temporary client credentials to the Postgres database.

See Dynamic Secrets for another example of how this is done.

Create a role to allow access to the Kubernetes secrets engine for the demo-ns.

Create a new role for the dynamic secret.

$ vault write auth/demo-auth-mount/role/auth-role \
   bound_service_account_names=default \
   bound_service_account_namespaces=demo-ns \
   token_ttl=0 \
   token_period=120 \
   token_policies=demo-auth-policy-db \
   audience=vault

The output should resemble the following:

Success! Data written to: auth/demo-auth-mount/role/auth-role

Exit the shell.
```
 $ exit
```

Create the application

Dynamic secrets

In this section you create a namespace demo-ns that has a dynamic secret available, and an application to access it. By using this secret the application has access to Postgres. Later you will use k9s to examine the variable and verify that it automatically rotates.

Create a new namespace.

$ kubectl create ns demo-ns
namespace/demo-ns created

Create the app, Vault connection, authentication, service account and corresponding secrets.
```
$ kubectl apply -f dynamic-secrets/.
deployment.apps/vso-db-demo created
secret/vso-db-demo created
vaultauth.secrets.hashicorp.com/default configured
vaultauth.secrets.hashicorp.com/demo-operator created
vaultconnection.secrets.hashicorp.com/default configured
vaultdynamicsecret.secrets.hashicorp.com/vso-db-demo-create created
vaultdynamicsecret.secrets.hashicorp.com/vso-db-demo created
serviceaccount/demo-operator created
```
Note
If you receive warning messages resembling the following:
Warning: resource vaultauths/default is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. The kubectl apply command should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
This warning message is expected. No preexisting information for annotation kubectl.kubernetes.io/last-applied-configuration exists, therefore this first configuration and will be used to compare for changes later.

Examine the dynamic secret

Using k9s examine the dynamic secret, and confirm the secrets changes every minute.

Go back to the window with k9s and display all the available namespaces with the :ns command.
Find and choose the demo-ns by highlighting it and hitting return.
Display the secrets in the demo-ns pod by typing :secrets and enter.
Choose the first secret displayed and type in x.
The secrets should resemble the following, remember to note down the password displayed
Exit screen with escape key and wait a minimum of 60 seconds, and check the values once again with x. Notice the password has automatically changed.

Clean up

Delete the minikube cluster.

$ minikube delete
🔥  Deleting "minikube" in docker ...
🔥  Deleting container "vault-secrets-operator" ...
🔥  Removing /Users/mrken/.minikube/machines/vault-secrets-operator ...
💀  Removed all traces of the "vault-secrets-operator" cluster.

Additional discussion

A Kubernetes operator is a software extension that uses custom resources to manage applications hosted on Kubernetes. The Vault Secret Operator leverages the HashiCorp Vault as a complete secrets management solution.

Secrets exist within Namespaces, which are virtual clusters with a Kubernetes Cluster. The secrets operator allows you to administer the secrets through Vault, but access as a Kubernetes native structure.

Kubernetes cluster administrators interested in using the the Vault Secrets Operator to rotate Dynamic secrets are encouraged to look at the demo include with the source for the Vault Secrets Operator.

In this tutorial you begun to learn about the Vault Secrets Operator by set up a Minikube cluster with a Vault instance, Vault Secrets Operator controller and created a secret in a namespace called app. Then you displayed the secret in k9s, and used Vault to rotate the secret. You then validated the change in the Vault secret value was reflected in the Kubernetes secret.

In the Dynamic Secrets section you set up a PostGresSQL pod, dummy application and created a dynamic secret. Then, using k9s you watched the secret rotate automatically.

The Vault Secrets Operator is a first class Kubernetes operator pattern for use with HashiCorp Vault responsible for syncing Vault secrets to Kubernetes Secrets natively.

Features of the Vault Secrets Operator include support for all secrets engines and Kubernetes, AWS, JWT and AppRole authentication to Vault.

References

Vault on Red Hat Demo Platform (RHDP)

Securing logs in Confluent HashiCorp Vault