Static secrets with the Vault Secrets Operator on Kubernetes

Challenge

Vault offers a complete solution for secrets management, but developers and operators all need to learn a completely new tool. They want a cloud native way to access the secrets through the Kubernetes secrets cache.

Solution

A Kubernetes Operator is a software extension that uses custom resources to manage applications hosted on Kubernetes.

The Vault secret operator is a Kubernetes Operator that syncs secrets between Vault and Kubernetes natively without having to learn details of Vault use.

Currently, Vault secret operator is available as a Beta feature, and it supports kv-v1 and kv-v2, TLS certificates in PKI and full range of static and dynamic secrets.

The Secrets Operator syncs the secrets between Vault and the Kubernetes secrets in a specified namespace. Within that namespace, applications have access to the secrets. The secrets are still managed by Vault, but accessed through the standard way on Kubernetes.

Prerequisites

• Vault version 1.12.0 or later
• Docker
• Kubernetes command-line interface (CLI)
• Kubernetes in Docker (Kind)
• k9s
• The helm CLI

This lab was written with the following versions.

Kubernetes in Docker (Kind):

$ kind version     
kind v0.17.0 go1.19.3 darwin/arm64

Kubernetes-cli:

$ kubectl version --short
Client Version: v1.26.1
Kustomize Version: v4.5.7

K9s:

$ k9s version
____  __.________       
|    |/ _/   __   \______
|      < \____    /  ___/
|    |  \   /    /\___ \ 
|____|__ \ /____//____  >
        \/            \/ 

Version:    0.27.3
Commit:     7c76691c389e4e7de29516932a304f7029307c6d
Date:       n/a

The GitHub repo

1. Clone repo at learn-vault-secrets-operator.

   $ git clone https://github.com/hashicorp-education/learn-vault-secrets-operator.git
   Cloning into 'learn-vault-secrets-operator'...
   remote: Enumerating objects: 15, done.
   remote: Counting objects: 100% (15/15), done.
   remote: Compressing objects: 100% (11/11), done.
   remote: Total 15 (delta 0), reused 15 (delta 0), pack-reused 0
   Receiving objects: 100% (15/15), done.
   

   Copy

2. Move into that folder.

   $ cd learn-vault-secrets-operator
   

   Copy

Kind of Kubernetes

1. Create a Kind cluster.

   $ kind create cluster \
       --wait=5m \
       --image kindest/node:v1.25.3 \
       --name vault-secrets-operator  \
       --config kind/config.yaml
   

   Copy

   Output should resemble the following:

   Creating cluster "vault-secrets-operator" ...
   ✓ Ensuring node image (kindest/node:v1.25.3) 🖼 
   ✓ Preparing nodes 📦  
   ✓ Writing configuration 📜 
   ✓ Starting control-plane 🕹️ 
   ✓ Installing CNI 🔌 
   ✓ Installing StorageClass 💾 
   ✓ Waiting ≤ 5m0s for control-plane = Ready ⏳ 
   • Ready after 18s 💚
   Set kubectl context to "kind-vault-secrets-operator"
   You can now use your cluster with:
   
   kubectl cluster-info --context kind-vault-secrets-operator
   
   Have a nice day! 👋 
   

   You have created a Kubernetes cluster running on Docker.

Vault Cluster

1. If you have not already, add the HashiCorp Repo.

   $ helm repo add hashicorp https://helm.releases.hashicorp.com 
   

   Copy

2. In order to have the latest version of the HashiCorp helm charts, Update the repo.

   $ helm repo update
   Hang tight while we grab the latest from your chart repositories...
   ...Successfully got an update from the "hashicorp" chart repository
   ...Successfully got an update from the "open" chart repository
   ...Successfully got an update from the "bitnami" chart repository
   Update Complete. ⎈Happy Helming!⎈
   

   Copy

   Details of the output might differ, the important things to look for is the Update Complete.

3. Determine the latest version of Vault.

   $ helm search repo hashicorp/vault
   NAME            CHART VERSION   APP VERSION     DESCRIPTION                   
   hashicorp/vault 0.23.0          1.12.1          Official HashiCorp Vault Chart
   

   Copy

   Ideally for this lab you will need Vault v1.10 or greater.

4. Using the YAML file in /vault install Vault on your Kind cluster

   $ helm install vault hashicorp/vault -n vault --create-namespace --values vault/vault-values.yaml
   

   Copy

   The Output should resemble the following:

   NAME: vault
   LAST DEPLOYED: Fri Mar 31 09:37:42 2023
   NAMESPACE: vault
   STATUS: deployed
   REVISION: 1
   NOTES:
   Thank you for installing HashiCorp Vault!
   
   Now that you have deployed Vault, you should look over the docs on using
   Vault with Kubernetes available here:
   
   https://www.vaultproject.io/docs/
   
   Your release is named vault. To learn more about the release, try:
   $ helm status vault
   $ helm get manifest vault
   

Configure Vault

1. Connect to the Vault instance.

   $ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
   

   Copy

2. Enable the Kubernetes auth method.

   $ vault auth enable kubernetes
   Success! Enabled kubernetes auth method at: kubernetes/
   

   Copy

3. Configure the auth method.

   $ vault write auth/kubernetes/config \
   kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
   

   Copy

   Output as follows:

   Success! Data written to: auth/kubernetes/config 
   

4. Enable the kv v2 Secrets Engine.

   $ vault secrets enable -path=kvv2 kv-v2
   Success! Enabled the kv-v2 secrets engine at: kvv2/
   

   Copy

5. Create a read only policy.

   $ vault policy write dev - <<EOF
   path "kvv2/*" {
     capabilities = ["read"]
   }
   EOF
   

   Copy

   Output is as follows:

   Success! Uploaded policy: dev 
   

6. Create a role in Vault to enable access to secret.

   $ vault write auth/kubernetes/role/role1 \
       bound_service_account_names=default \
       bound_service_account_namespaces=app \
       policies=dev \
       audience=vault \
       ttl=24h
   

   Copy

   Output as follows:

    Success! Data written to: auth/kubernetes/role/role1
   

   Notice that the bound_service_account_namespaces is app, limiting where the secret is synced to.

7. Create some secrets.

   $ vault kv put kvv2/webapp/config username="static-user" password="static-password"
   ===== Secret Path =====
   kvv2/data/webapp/config
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       2023-04-03T16:35:56.1103993Z
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            1
   

   Copy

8. Exit the Vault instance.

   $ exit
   

   Copy

The Vault Secrets Operator

1. Now use helm to deploy the Vault Secrets Operator.

   $ helm install vault-secrets-operator hashicorp/vault-secrets-operator --version 0.1.0-beta -n vault-secrets-operator-system --create-namespace --values vault/vault-operator-values.yaml
   NAME: vault-secrets-operator
   LAST DEPLOYED: Fri Mar 31 10:00:29 2023
   NAMESPACE: vault-secrets-operator-system
   STATUS: deployed
   REVISION: 1
   

   Copy

2. Examining the file vault/vault-operator-values.yaml:

   vault/vault-operator-values.yaml

   defaultAuthMethod:
   enabled: true
   # Vault namespace for the VaultAuthMethod CR
   namespace: "app"
   method: kubernetes
   mount: kubernetes 
   

   The same namespace, app as mentioned during the secret creation is mentioned here.

Deploy and sync a secret

1. Create a namespace called app on your Kind cluster.

   $ kubectl create ns app
   namespace/app created
   

   Copy

2. Create the secret in the app namespace.

   $ kubectl apply -f vault/static-secret.yaml
   vaultstaticsecret.secrets.hashicorp.com/vault-kv-app created
   

   Copy

Secret rotation

First, display the secret in k9s.

1. Open a new terminal and start up k9s.

   $ k9s --context kind-vault-secrets-operator
   

   Copy

2. If not already displayed, list the namespaces by typing :ns.

   

3. Choose the app namespace and press enter.

   

4. This area is blank, so type in :secrets and hit enter.

   

5. Now the secrets named secretkv is displayed, highlight it.

   

6. Display the secret by pressing the x key.

   

7. Back in your terminal, connect to the Vault instance.

   $ kubectl exec --stdin=true --tty=true vault-0 -n vault -- /bin/sh
   / $ 
   

   Copy

8. Rotate the secret.

   $ vault kv put kvv2/webapp/config username="static-user2" password="static-password2"
   ===== Secret Path =====
   kvv2/data/webapp/config
   
   ======= Metadata =======
   Key                Value
   ---                -----
   created_time       2023-04-03T16:40:31.274411719Z
   custom_metadata    <nil>
   deletion_time      n/a
   destroyed          false
   version            2
   

   Copy

9. Return to k9s, and escape back to the secret page and hit x again to display the updated secret.

   

Clean up

1. Exit the Vault instance.

   $ exit
   

   Copy

2. Now delete the Kind cluster.

   $ kind delete cluster -n vault-secrets-operator
   Deleting cluster "vault-secrets-operator" ...
   

   Copy

Additional discussion

A Kubernetes Operator is a software extension that uses custom resources to manage applications hosted on Kubernetes. The Vault Secret operator leverages the HashiCorp Vault as a complete secrets management solution.

Secrets exist within Namespaces, which are virtual clusters with a Kubernetes Cluster. The secrets operator allows you to administer the secrets through Vault, but access as a Kubernetes native structure.

Kubernetes cluster administrators interested in using the the Vault Secrets Operator to rotate Dynamic secrets are encouraged to look at the demo include with the source for the Vault Secrets Operator.

Next steps

The Vault Secrets Operator is a first class Kubernetes Operator pattern for use with HashiCorp Vault responsible for syncing Vault secrets to Kubernetes Secrets natively.

In this tutorial you set up a Kubernetes in Docker Cluster with a Vault instance, Vault Secrets Operator controller and created a secret in a namespace called app. Then you displayed the secret in k9s, and used Vault to rotate the secret. Finally, you validated the change in the Vault secret value was reflected in the Kubernetes secret.

References

• Vault Secrets Operator
• Dynamic Secrets Sync demo of the Vault Secret Operator
• Vault Secrets Operator: A new method for Kubernetes integration
• Vault on Kubernetes Deployment Guide
• Kubernetes Operator Pattern