Manage Kubernetes native secrets with the Vault Secrets Operator

Challenge

Vault offers a complete solution for secrets lifecycle management, but that requires developers and operators to learn a new tool. Instead, developers want a cloud native way to access the secrets through Kubernetes and have no need to understand Vault in great depth. Vault Secrets Operator (VSO) updates Kubernetes native secrets. The user accesses Kubernetes native secrets managed on the back end by HashiCorp Vault.

Solution

A Kubernetes operator is a software extension that uses custom resources to manage applications hosted on Kubernetes.

The Vault Secrets Operator is a Kubernetes operator that syncs secrets between Vault and Kubernetes natively without requiring the users to learn details of Vault use.

The Vault secrets operator supports kv-v1 and kv-v2, TLS certificates in PKI and full range of static and dynamic secrets.

The Vault Secrets Operator syncs the secrets between Vault and the Kubernetes secrets in a specified namespace. Within that namespace, applications have access to the secrets. The secrets are still managed by Vault, but accessed through the standard way on Kubernetes.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Start interactive lab

Prerequisites

• Docker
• Helm CLI
• k9s
• Kubernetes command-line interface (CLI)
• minikube
• Recent version of the Vault binary installed. Refer to the Vault install guide, and confirm you are using a version that supports VSO: please see Supported Vault versions.
  • Vault Enterprise requires a valid license.

Install supporting tools

This tutorial was last tested 27 June 2023 on a macOS 13.3.1 using the following software versions.

$ docker version
Client:
Cloud integration: v1.0.25
Version:           20.10.16
## ...

$ helm version
version.BuildInfo{Version:"v3.12.0", GitCommit:"c9f554d75773799f72ceef38c51210f1842a1dea", GitTreeState:"clean", GoVersion:"go1.20.4"}

$ k9s version
____  __.________
|    |/ _/   __   \______
|      < \____    /  ___/
|    |  \   /    /\___ \
|____|__ \ /____//____  >
      \/            \/

Version:    0.27.3
Commit:     7c76691c389e4e7de29516932a304f7029307c6d
Date:       n/a

$ kubectl version --short
Client Version: v1.27.1
Kustomize Version: v5.0.1
Server Version: v1.26.3

$ minikube version
minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

Clone the GitHub repository

1. Clone the repository at learn-vault-secrets-operator.

   $ git clone https://github.com/hashicorp-education/learn-vault-secrets-operator
   Cloning into 'learn-vault-secrets-operator'...
   remote: Enumerating objects: 15, done.
   remote: Counting objects: 100% (15/15), done.
   remote: Compressing objects: 100% (11/11), done.
   remote: Total 15 (delta 0), reused 15 (delta 0), pack-reused 0
   Receiving objects: 100% (15/15), done.
   

2. Move into that folder.

   $ cd learn-vault-secrets-operator
   

Start minikube

minikube allows you to run a miniature Kubernetes cluster on your local machine.

1. Create a minikube cluster.

   $ minikube start
   

   The output should resemble the following:

   😄  minikube v1.30.1 on Darwin 13.4 (arm64)
   ✨  Automatically selected the docker driver
   📌  Using Docker Desktop driver with root privileges
   👍  Starting control plane node minikube in cluster minikube
   🚜  Pulling base image ...
   🔥  Creating docker container (CPUs=2, Memory=4000MB) ...
   🐳  Preparing Kubernetes v1.26.3 on Docker 23.0.2 ...
      ▪ Generating certificates and keys ...
      ▪ Booting up control plane ...
      ▪ Configuring RBAC rules ...
   🔗  Configuring bridge CNI (Container Networking Interface) ...
      ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
   🔎  Verifying Kubernetes components...
   🌟  Enabled addons: storage-provisioner, default-storageclass
   🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
   

   You have created a Kubernetes cluster running on Docker.

Install Vault cluster

Using Helm install Vault on a local instance of minikube. In Kubernetes, install Vault on it's own virtual cluster called a namespace.

1. If you have not already, add the HashiCorp repository.

   $ helm repo add hashicorp https://helm.releases.hashicorp.com
   

2. Update to the latest version of the HashiCorp Helm charts, update the repository.

   $ helm repo update
   Hang tight while we grab the latest from your chart repositories...
   ...Successfully got an update from the "hashicorp" chart repository
   ...Successfully got an update from the "open" chart repository
   ...Successfully got an update from the "bitnami" chart repository
   Update Complete. ⎈Happy Helming!⎈
   

   Details of the output might differ, the important thing is the Update Complete message.

3. Determine the latest version of Vault.

   $ helm search repo hashicorp/vault
   NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
   hashicorp/vault                         0.28.1          1.17.2          Official HashiCorp Vault Chart
   hashicorp/vault-secrets-operator        0.7.1           0.8.0           Official Vault Secrets Operator Chart
   

   Vault Secrets Operator supports the latest three versions of Vault. Please see Supported Vault versions for details.

4. Using the YAML file in the appropriate sub-folder, install Vault on your minikube cluster

   Vault CommunityVault Enterprise

   $ helm install vault hashicorp/vault -n vault --create-namespace --values vault/vault-values.yaml
   

   For Vault Enterprise create a Kubernetes secret with the Vault license.

   Start with creating a vault namespace.

   $ kubectl create ns vault
   

   Locally export an environment variable with a Vault enterprise license.

   $ export VAULT_LICENSE="..."
   

   Create a Kubernetes secret for the Vault license.

   $ kubectl create secret generic vault-license --from-literal license=$VAULT_LICENSE -n vault
   secret/vault-license created
   

   Using the YAML file found in vault-ent install Vault Enterprise.

   $ helm install vault hashicorp/vault -n vault --values vault-ent/vault-values.yaml
   

   The output should resemble the following:

   NAME: vault
   LAST DEPLOYED: Fri Mar 31 09:37:42 2023
   NAMESPACE: vault
   STATUS: deployed
   REVISION: 1
   NOTES:
   Thank you for installing HashiCorp Vault!
   
   Now that you have deployed Vault, you should look over the docs on using
   Vault with Kubernetes available here:
   
   https://www.vaultproject.io/docs/
   
   Your release is named vault. To learn more about the release, try:
   $ helm status vault
   $ helm get manifest vault
   

5. Wait until the Vault pods are Ready 1/1 and status is Running.

   $ kubectl get pods -n vault
   NAME      READY   STATUS    RESTARTS   AGE
   vault-0   1/1     Running   0          3m
   

Configure Vault

Here you connect to the Vault instance on minikube, enable and configure Kubernetes authentication, KV secrets engine, a role and policy for Kubernetes, and create a static secret.

1. Connect to the Vault instance. Until you exit you will be executing from inside the Vault instance.

   Vault CommunityVault Enterprise

   $ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
   

   $ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
   

   Create a namespace to facilitate management and isolate tenants.

   $ vault namespace create us-west-org
   Key                Value
   ---                -----
   custom_metadata    map[]
   id                 mgVUq
   path               us-west-org/
   

   Setting the VAULT_NAMESPACE environment isolates all subsequent Vault CLI commands to that namespace.

   $ export VAULT_NAMESPACE=us-west-org
   

2. Move into the tmp directory.

   $ cd tmp
   

3. Enable the Kubernetes auth method.

   $ vault auth enable -path demo-auth-mount kubernetes
   Success! Enabled kubernetes auth method at: demo-auth-mount/
   

4. Configure the auth method.

   $ vault write auth/demo-auth-mount/config \
      kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
   

   The output should resemble the following:

   Success! Data written to: auth/demo-auth-mount/config
   

5. Enable the kv v2 Secrets Engine.

   $ vault secrets enable -path=kvv2 kv-v2
   Success! Enabled the kv-v2 secrets engine at: kvv2/
   

6. Create a JSON file with a Vault policy.

   Vault CommunityVault Enterprise

   $ tee webapp.json <<EOF
   path "kvv2/data/webapp/config" {
      capabilities = ["read", "list"]
   }
   EOF
   

   After uploading the Vault policy, it allows reading of the webapp/config secret in the kvv2 secrets engine mount.

   $ tee webapp.json <<EOF
   path "kvv2/data/webapp/config" {
      capabilities = ["read", "list", "subscribe"]
      subscribe_event_types = ["kv*"]
   }
   path "sys/events/subscribe/kv*" {
      capabilities = ["read"]
   }
   EOF
   

   This policy allows instant updates of Kubernetes secrets managed by Vault Secrets Operator. Instant updates uses Event notifications to near-instantly update secrets that VSO manages.

   The example policy allows reading, listing, and subscribing to changes in the webapp/config secret in the kvv2 mount The second path allows the client to subscribe to changes to event types matching kv*. See Subscribing to event notifications for more information.

   $ vault policy write webapp webapp.json
   Success! Uploaded policy: webapp
   

7. Create a role in Vault to enable access to secrets within the kv v2 secrets engine.

   Vault CommunityVault Enterprise

   $ vault write auth/demo-auth-mount/role/role1 \
      bound_service_account_names=demo-static-app \
      bound_service_account_namespaces=app \
      policies=webapp \
      audience=vault \
      ttl=24h
   

   $ vault write auth/demo-auth-mount/role/role1 \
      bound_service_account_names=demo-static-app \
      bound_service_account_namespaces=app \
      policies=webapp \
      audience=vault \
      token_period=2m
   

   The output should resemble the following:

    Success! Data written to: auth/demo-auth-mount/role/role1
   

   Notice that the bound_service_account_namespaces is app, limiting the sync to specific namespaces.

8. Create a secret.

   $ vault kv put kvv2/webapp/config username="static-user" password="static-password"
   ===== Secret Path =====
   kvv2/data/webapp/config
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       2023-04-03T16:35:56.1103993Z
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            1
   

9. Exit the Vault instance.

   $ exit
   

Install the Vault Secrets Operator

1. Use Helm to deploy the Vault Secrets Operator.

   Vault CommunityVault Enterprise

   $ helm install vault-secrets-operator hashicorp/vault-secrets-operator -n vault-secrets-operator-system --create-namespace --values vault/vault-operator-values.yaml
   NAME: vault-secrets-operator
   LAST DEPLOYED: Fri Mar 31 10:00:29 2023
   NAMESPACE: vault-secrets-operator-system
   STATUS: deployed
   REVISION: 1
   

   $ helm install vault-secrets-operator hashicorp/vault-secrets-operator -n vault-secrets-operator-system --create-namespace --values vault-ent/vault-operator-values.yaml
   NAME: vault-secrets-operator
   LAST DEPLOYED: Fri Mar 31 10:00:29 2023
   NAMESPACE: vault-secrets-operator-system
   STATUS: deployed
   REVISION: 1
   

2. Examine the file vault-operator-values.yaml for the values that this tutorial uses for VSO:

   Vault CommunityVault Enterprise

   $ cat vault/vault-operator-values.yaml
   defaultVaultConnection:
      enabled: true
      address: "http://vault.vault.svc.cluster.local:8200"
      skipTLSVerify: false
   ...
   
   

   $ cat vault-ent/vault-operator-values.yaml
   defaultVaultConnection:
      enabled: true
      address: "http://vault.vault.svc.cluster.local:8200"
      skipTLSVerify: false
   ...
   
   

   For more details on these values see the values.yaml file for the Vault Secrets Operator Helm chart.

Deploy and sync a secret

In this section you will set up a namespace with a Kubernetes secret, and configure the Vault Secrets Operator to manage that secret.

1. Create a namespace called app on your Kubernetes cluster.

   $ kubectl create ns app
   namespace/app created
   

2. Set up Kubernetes authentication for the secret.

   Vault CommunityVault Enterprise

   $ kubectl apply -f vault/vault-auth-static.yaml
   vaultauth.secrets.hashicorp.com/static-auth created
   

   $ kubectl apply -f vault-ent/vault-auth-static.yaml
   vaultconnection.secrets.hashicorp.com/static-default created
   serviceaccount/demo-static-app created
   vaultauth.secrets.hashicorp.com/static-auth created
   

   HCP Vault Dedicated

   If you are using HCP Vault Dedicated, similar to Vault enterprise you will need to add the namespace parameter to the spec field in the vault-auth-static.yaml file.

3. Create the secret names secretkv in the app namespace.

   Vault CommunityVault Enterprise

   $ kubectl apply -f vault/static-secret.yaml
   vaultstaticsecret.secrets.hashicorp.com/vault-kv-app created
   

   $ kubectl apply -f vault-ent/static-secret.yaml
   vaultstaticsecret.secrets.hashicorp.com/vault-kv-app created
   

   If you examine the static-secret.yaml just used, look near the bottom.

   Either version you will find a field called refreshAfter. That fields determines how often to check the secret for updates. In the Vault community edition example, it set up to refresh after 30 seconds.

   In the Vault enterprise example the refreshAfter field is a fallback option, set to 1 hour. Since the instantUpdates is true, so the refreshAfter field is not used.

   ...
   syncConfig:
    instantUpdates: true
   ..

Rotate the static secret

In this section you use the k9s tool to display the secret. Then you use Vault to manually rotate the secret, and k9s to verify rotation of the Kubernetes secret.

1. Open a new terminal and start up k9s.

   $ k9s
   

2. If not already displayed, list the namespaces by typing :ns.

   

3. Use the up and down arrows to choose the app namespace and press enter.

   

4. This area is blank, so type in :secrets and press enter.

   

5. Highlight the secret named secretkv.

   

6. Display the secret by pressing the x key.

   

7. In the original terminal, connect to the Vault instance.

   $ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
   / $
   

8. Rotate the secret.

   Vault CommunityVault Enterprise

   $ vault kv put kvv2/webapp/config username="static-user2" password="static-password2"
   ===== Secret Path =====
   kvv2/data/webapp/config
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       2023-04-03T16:40:31.274411719Z
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            2
   

   Wait about 30 seconds before continuing to the next step, to allow for the secret to refresh.

   $ VAULT_NAMESPACE=us-west-org vault kv put kvv2/webapp/config username="static-user2" password="static-password2"
   ===== Secret Path =====
   kvv2/data/webapp/config
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       2023-04-03T16:40:31.274411719Z
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            2
   

   You have enabled Instant Updates and you can move to the next step.

9. Return to k9s, and escape back to the secret page and press x again to display the updated secret.

   

   The secret has changed, and now will be different than noted earlier.

10. At the first window, exit Vault.

    $ exit
    

Dynamic secrets

Manually rotating secrets is cumbersome and prone to human error. Vault dynamic secrets provides a solution for this.

Now you will create a dynamic secret with the database secrets engine. Vault manages the Dynamic secrets lifecycle by Vault and will be automatically rotated. The lifecycle management includes deleting and recreating the secrets. In this section you will use the Vault Secrets Operator to rotate the Kubernetes secret every 1 minute.

Install PostgreSQL pod

You will create a pod and install PostreSQL. PostgreSQL will later generate credentials for the database secrets engine.

1. Create a namespace for the PostgreSQL pod.

   $ kubectl create ns postgres
   

2. Add the Bitnami repository to your local Helm.

   $ helm repo add bitnami https://charts.bitnami.com/bitnami
   

3. Install PostgreSQL.

   $ helm upgrade --install postgres bitnami/postgresql --namespace postgres --set auth.audit.logConnections=true  --set auth.postgresPassword=secret-pass
   

   The output should resemble the following:

   Release "postgres" does not exist. Installing it now.
   NAME: postgres
   LAST DEPLOYED: Wed Jun  7 05:56:05 2023
   NAMESPACE: postgres
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   NOTES:
   CHART NAME: postgresql
   CHART VERSION: 12.5.6
   APP VERSION: 15.3.0
   
   ...
   
   

Setup PostgreSQL

Connect to the Vault instance, and set up a database secrets engine in Vault with a corresponding role and policy.

1. Connect to the Vault instance.

   Vault CommunityVault Enterprise

   $ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
   

   $ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
   

   Setting the VAULT_NAMESPACE environment isolates all subsequent Vault CLI commands to that namespace.

   $ export VAULT_NAMESPACE=us-west-org
   

2. Enable an instance of the Database Secrets Engine.

   $ vault secrets enable -path=demo-db database
   Success! Data written to: demo-db/config/demo-db
   

3. Configure the Database Secrets Engine.

   $ vault write demo-db/config/demo-db \
      plugin_name=postgresql-database-plugin \
      allowed_roles="dev-postgres" \
      connection_url="postgresql://{{username}}:{{password}}@postgres-postgresql.postgres.svc.cluster.local:5432/postgres?sslmode=disable" \
      username="postgres" \
      password="secret-pass"
   

   The output should resemble the following:

   Success! Data written to: demo-db/config/demo-db
   

4. Create a role for the PostgreSQL pod.

   $ vault write demo-db/roles/dev-postgres \
      db_name=demo-db \
      creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
         GRANT ALL PRIVILEGES ON DATABASE postgres TO \"{{name}}\";" \
      revocation_statements="REVOKE ALL ON DATABASE postgres FROM  \"{{name}}\";" \
      backend=demo-db \
      name=dev-postgres \
      default_ttl="1m" \
      max_ttl="1m"
   

   The output should resemble the following:

   Success! Data written to: demo-db/roles/dev-postgres
   

   Note

   Important: when you define the role in a production deployment, you must create user creation_statements, revocation_statements, renew_statements, and rotation_statements which are valid for the database you've configured. If you do not specify statements appropriate to creating, revoking, or rotating users, Vault inserts generic statements which can be unsuitable for your deployment.

   Also, you do not want default_ttl="1m" or max_ttl="1m" set in production. This is too short and in production should use a longer TTL. This tutorial is for demonstration purposes.

5. Create the demo-auth-policy-db policy.

   $ vault policy write demo-auth-policy-db - <<EOF
   path "demo-db/creds/dev-postgres" {
      capabilities = ["read"]
   }
   EOF
   

   The output should resemble the following:

   Success! Uploaded policy: demo-auth-policy-db
   

6. Disconnect to the Vault instance.

   $ exit
   

Transit encryption

You can configure Vault Secrets Operator to maintain an internal, encrypted persistent cache of client tokens.

This is helpful for being able to transparently renew leases for dynamic secrets should the operator restart. Without using the client cache, the operator would need to fetch new client tokens on restarts, and re-issue credentials for dynamic secrets, causing downtime for applications. With transit encryption configured and enabled, the client token cache is end-to-end encrypted using Transit Encryption. You cannot access the persisted tokens.

1. The Helm chart vault-operator-values.yaml already set up the Vault secrets operator for client token cache.

   $ cat vault/vault-operator-values.yaml
   ...
   controller:
      manager:
         ...
         clientCache:
            persistenceModel: direct-encrypted
            storageEncryption:
               enabled: true
               mount: demo-auth-mount
               keyName: vso-client-cache
               transitMount: demo-transit
               kubernetes:
                  role: auth-role-operator
                  serviceAccount: demo-operator
   

   Much of the remaining tutorial is setting up Vault to these specifications already present in vault-operator-values.yaml.

   The clientCache section has settings for the cache to use the direct encrypted mode.

   The storageEncryption sub-section specifies details of the transit secrets engine, and authentication used by the transit secrets engine.

   • transitMount sets the transit secrets engine mount named demo-transit.
   • mount sets the Kubernetes auth method engine named demo-auth-transit used by the demo-transit.
   • keyName specifies a key named vso-client-cache used for encrypt/decrypt operations.
   • kubernetes sub-section configures the Vault Kubernetes auth method, specifying the role and the service account.

   Refer to Vault Secrets Operator Helm chart section on storageEncryption documentation for details.

2. Connect back to the Vault instance.

   Vault CommunityVault Enterprise

   $ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
   

   $ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
   

   Setting the VAULT_NAMESPACE environment isolates all subsequent Vault CLI commands to that namespace.

   $ export VAULT_NAMESPACE=us-west-org
   

3. Enable an instance of the Transit Secrets Engine at the path demo-transit.

   $ vault secrets enable -path=demo-transit transit
   Success! Enabled the transit secrets engine at: demo-transit/
   

4. Create a encryption key.

   $ vault write -force demo-transit/keys/vso-client-cache
   Success! Data written to: demo-transit/keys/vso-client-cache
   

5. Create a policy for the operator role to access the encryption key.

   $ vault policy write demo-auth-policy-operator - <<EOF
   path "demo-transit/encrypt/vso-client-cache" {
      capabilities = ["create", "update"]
   }
   path "demo-transit/decrypt/vso-client-cache" {
      capabilities = ["create", "update"]
   }
   EOF
   

   The output should resemble the following:

   Success! Uploaded policy: demo-auth-policy-operator
   

6. Create Kubernetes auth role for the operator.

   $ vault write auth/demo-auth-mount/role/auth-role-operator \
      bound_service_account_names=vault-secrets-operator-controller-manager \
      bound_service_account_namespaces=vault-secrets-operator-system \
      token_ttl=0 \
      token_period=120 \
      token_policies=demo-auth-policy-operator \
      audience=vault
   

   The output should resemble the following:

   Success! Data written to: auth/demo-auth-mount/role/auth-role-operator
   

Setup dynamic secrets

Vault includes a number of dynamic secrets engines capable of generating temporary credentials for various applications. In this tutorial, Vault's dynamic secrets engine for PostgreSQL to generate temporary client credentials to the PostgreSQL database.

Create a role to allow access to the Kubernetes secrets engine for the demo-ns.

1. Create a new role for the dynamic secret.

   $ vault write auth/demo-auth-mount/role/auth-role \
      bound_service_account_names=demo-dynamic-app \
      bound_service_account_namespaces=demo-ns \
      token_ttl=0 \
      token_period=120 \
      token_policies=demo-auth-policy-db \
      audience=vault
   

   The output should resemble the following:

   Success! Data written to: auth/demo-auth-mount/role/auth-role
   

2. Exit the shell.

   $ exit
   

Create the application

In this section you create a namespace demo-ns that has a dynamic secret available, and an application to access it. By using this secret the application has access to PostgreSQL. Later you will use k9s to examine the variable and verify that it automatically rotates.

1. Create a new namespace.

   $ kubectl create ns demo-ns
   namespace/demo-ns created
   

2. Create the app, Vault connection, authentication, service account and corresponding secrets.

   Vault CommunityVault Enterprise

   $ kubectl apply -f dynamic-secrets/.
   deployment.apps/vso-db-demo created
   secret/vso-db-demo created
   vaultauth.secrets.hashicorp.com/default configured
   vaultauth.secrets.hashicorp.com/demo-operator created
   vaultconnection.secrets.hashicorp.com/default configured
   vaultdynamicsecret.secrets.hashicorp.com/vso-db-demo-create created
   vaultdynamicsecret.secrets.hashicorp.com/vso-db-demo created
   serviceaccount/demo-operator created
   

   $ kubectl apply -f vault-ent/dynamic-secrets/.
   deployment.apps/vso-db-demo created
   secret/vso-db-demo created
   vaultauth.secrets.hashicorp.com/default configured
   vaultauth.secrets.hashicorp.com/demo-operator created
   vaultconnection.secrets.hashicorp.com/default configured
   vaultdynamicsecret.secrets.hashicorp.com/vso-db-demo-create created
   vaultdynamicsecret.secrets.hashicorp.com/vso-db-demo created
   serviceaccount/demo-operator created
   

   Note

   If you receive warning messages resembling the following:

   *Warning: resource vaultauths/default is missing the kubectl.kubernetes.io/last-applied-configuration annotation. The kubectl apply requires this. The kubectl apply command used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.

   No preexisting information for annotation kubectl.kubernetes.io/last-applied-configuration exists, therefore this is the configuration used to compare to changes later.

Examine the dynamic secret

Using k9s, examine the dynamic secret and confirm the secrets change every minute.

1. Go back to the window with k9s and display all the available namespaces with the :ns command.

2. Find and choose the demo-ns by highlighting it and pressing return.

   

3. Display the secrets in the demo-ns pod by typing :secrets and enter.

   

4. Choose the first secret displayed and press x.

   

   The secrets should resemble the following, remember to note down the password displayed

   

5. Exit screen with escape key and wait a minimum of 60 seconds, and check the values once again with x. Notice the password has automatically changed.

   

Clean up

1. Delete the minikube cluster.

   $ minikube delete
   🔥  Deleting "minikube" in docker ...
   🔥  Deleting container "vault-secrets-operator" ...
   🔥  Removing /Users/mrken/.minikube/machines/vault-secrets-operator ...
   💀  Removed all traces of the "vault-secrets-operator" cluster.
   

Additional discussion

A Kubernetes operator is a software extension that uses custom resources to manage applications hosted on Kubernetes. The Vault Secret Operator leverages HashiCorp Vault as a complete secrets management solution.

Secrets exist within Namespaces, which are virtual clusters with a Kubernetes Cluster. The secrets operator allows you to administer the secrets through Vault, but access them as a native Kubernetes primitive.

In this tutorial you began to learn about the Vault Secrets Operator by setting up a minikube cluster with a Vault instance and a Vault Secrets Operator controller. Then you created a secret in a namespace called app. You displayed the secret in k9s, and used Vault to rotate the secret. You then validated the change in the Vault secret value also changed the Kubernetes secret.

In the dynamic secrets section you set up a PostgreSQL pod, placeholder application and created a dynamic secret. Then, using k9s you watched the secret rotate automatically.

The Vault Secrets Operator is a Kubernetes operator pattern for use with HashiCorp Vault. It is responsible for syncing Vault secrets to Kubernetes Secrets natively.

Features of the Vault Secrets Operator include support for all secrets engines and Kubernetes, AWS, JWT and AppRole authentication to Vault.

References

• Vault Secrets Operator
• Vault Secrets Operator
• Dynamic Secrets Sync demo of the Vault Secret Operator
• Vault Secrets Operator: A new method for Kubernetes integration
• Vault on Kubernetes Deployment Guide
• Kubernetes Operator pattern