Secure secret management using Kubernetes service accounts

Purpose of this guide

This integration pattern demonstrates how to implement Kubernetes service accounts and leverage their metadata to provide access to Vault namespaces and secrets via Vault Secrets Operator. The goal is to exemplify HashiCorp's best practices for structuring Vault namespaces and mount paths.

Target audience

• Platform team responsible for Vault
• Platform team responsible for Kubernetes

Roles and responsibilities

• Vault administrator: Manages Vault namespaces, configures Kubernetes authentication and sets access policies.
• Kubernetes administrator: Creates and manages Kubernetes service accounts, configures permissions for Vault to validate Kubernetes service accounts and deploys the Vault Secrets Operator.
• Both roles will need to collaborate to ensure that secrets are correctly synced from Vault and accessible for Kubernetes applications.

Benefits

• Scalable and secure: Leverages Kubernetes service accounts for secure access to Vault, eliminating the need for manual credential management.
• Efficiency: Automates secret management, reducing the risk of human error and ensuring that secrets are always up-to-date.

Prerequisites

• Vault Enterprise/HCP Vault Dedicated 1.16 or higher
• Kubernetes 1.21 or higher
• Administrative permissions for both Vault and Kubernetes

Example scenario

Applications and workloads frequently need credentials to access resources, but the responsibility for managing the lifecycle of these credentials - such as creating, rotating, delivering, retiring, and revoking them - should not fall on the applications themselves. Automating these processes is essential for minimizing human involvement and reducing security risks. Vault's authentication methods and secret engines are specifically designed to facilitate this automation.

However, for applications to use Vault, they need an initial set of credentials, a challenge often referred to as "secret zero" or "secure introduction." This issue can be addressed by using a trusted identity provider. Public cloud platforms, and common container orchestration systems, typically provide identity services for the workloads they host (e.g. AWS IAM roles and Kubernetes service accounts). We recommend leveraging these trusted identity systems so that the applications running in these environments do not require initial credentials, effectively eliminating the "secret zero" problem.

This guide explains a Vault namespace and mount path design that uses Kubernetes service accounts as an identity mechanism for applications to securely access Vault-stored secrets. Additionally, it provides an example of how Kubernetes service account metadata can be used to grant access to specific Vault secrets, and how these secrets can be made available to applications as native Kubernetes secrets using the Vault Secrets Operator.

Design overview

• Vault namespaces and mount paths: There are multiple sub-namespaces under the admin namespace, each serving different tenants (e.g., business units). Each sub-namespace has a dedicated K/V secrets engine.
• Kubernetes authentication: Applications authenticate against a Kubernetes authentication mount, unique for each Vault sub-namespace if tenants do not share Kubernetes clusters. Authentication is done using Kubernetes service accounts, typically with the Vault Secrets Operator or Vault Agent Injector.
• Policy management: ACL policy path templating is used to simplify policy management, allowing applications to access K/V secrets within their business segment (e.g., team-a) and application-specific paths (e.g., my-app).

Step 1: Configure Vault namespace

Task: Setup Vault namespace and K/V v2 secrets engine
Persona: Vault administrator
Description: Prepare the namespace structure and enable the K/V secrets engine.

1. Create namespaces, if these do not exist.

   vault namespace create admin
   vault namespace create -namespace="admin" tenant-1
   

2. Set up a mount point called 'secret' as K/V version 2 within the namespace 'admin/tenant-1'.

   vault secrets enable -namespace="admin/tenant-1" -path="secret" kv-v2
   

3. Create some secrets to reflect the K/V path structure.

   # Team A - App 1
   vault kv put -namespace="admin/tenant-1" \
     -mount="secret" team-a/my-app/test \
     user=hello \
     password=kubernetes
   
   # Team B - App 2
   vault kv put -namespace="admin/tenant-1" \
     -mount="secret" team-b/another-app/test \
     user=hello \
     password=nomad
   

Step 2: Configure Kubernetes authentication

Task: Create Kubernetes service account for Vault
Persona: Kubernetes administrator
Description: In this step, you will create a Kubernetes service account for Vault and assign the necessary permissions for it to call the Kubernetes TokenReview API. This is needed in order for Vault to be able to validate other Kubernetes service accounts that are authenticating to Vault. Additionally, you will create a service account token secret, which will be used in the subsequent task to configure the Kubernetes auth method in Vault.

1. Create a service account for Vault and assign the permissions to act as a Kubernetes auth delegator.

   kubectl create namespace vault-auth
   kubectl create serviceaccount vault-sa -n vault-auth
   
   kubectl create clusterrolebinding vault-auth-binding \
     -n vault-auth \
     --clusterrole=system:auth-delegator \
     --serviceaccount=vault-auth:vault-sa
   

2. Create a service account token secret that is used by Vault to authenticate to Kubernetes.

   cat <<EOF > vault-secret.yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: vault-sa-secret
     namespace: vault-auth
     annotations:
       kubernetes.io/service-account.name: vault-sa
   type: kubernetes.io/service-account-token
   EOF
   
   kubectl apply -f vault-secret.yaml
   

Task: Setup Kubernetes auth
Persona: Vault administrator, Kubernetes administrator
Description: Enable Kubernetes authentication and import metadata from the Service Account to a Vault entity alias. The flag use_annotations_as_alias_metadata=true imports the metadata from the Service Account to a Vault entity alias, which you can use when constructing the Vault policies in the next task.

1. Enable the Kubernetes authentication method in Vault.

   vault auth enable -namespace="admin/tenant-1" kubernetes
   

2. Configure the Kubernetes authentication method with the Kubernetes service account token acquired in the previous step, the Kubernetes CA certificate and Kubernetes API URL.

   export SA_TOKEN=$(kubectl get secret vault-sa-secret -n vault-auth \
     -o jsonpath="{.data.token}" | base64 --decode)
   
   export KUBERNETES_CA=$(kubectl get secret vault-sa-secret -n vault-auth \
     -o jsonpath="{.data['ca\.crt']}" | base64 --decode)
   
   export KUBERNETES_URL=$(kubectl config view --minify \
     -o jsonpath='{.clusters[0].cluster.server}')
   
   vault write -namespace="admin/tenant-1" auth/kubernetes/config \
     use_annotations_as_alias_metadata=true \
     token_reviewer_jwt="${SA_TOKEN}" \
     kubernetes_host="${KUBERNETES_URL}" \
     kubernetes_ca_cert="${KUBERNETES_CA}"
   

Task: Setup policy and role
Persona: Vault administrator
Description: Define a group policy and attach the application to the group, leveraging Kubernetes metadata. The following policy allows the application to access K/V secrets within their business segment and tied to the application name.

1. Find the accessor for the Kubernetes auth method within the 'admin/tenant-1' namespace

   vault auth list -namespace="admin/tenant-1" -format=json \
     | jq -r '.["kubernetes/"].accessor' \
     > accessor_kubernetes.txt
   

2. Define a policy that makes use of Vault ACL policy templates.

   tee my-app-policy.hcl <<EOF
   # Allows to read K/V secrets 
   path "secret/data/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.BusinessSegmentName}}/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.AppName}}/*" {
       capabilities = ["read"]
   }
   # Allows reading K/V secret versions and metadata
   path "secret/metadata/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.BusinessSegmentName}}/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.AppName}}/*" {
         capabilities = ["list", "read"]
   }
   EOF
   
   vault policy write -namespace="admin/tenant-1" my-app-policy \
     my-app-policy.hcl
   

3. Associate the policy with a Vault authentication role.

   vault write -namespace="admin/tenant-1" auth/kubernetes/role/my-app \
       bound_service_account_names=my-app \
       bound_service_account_namespaces=app-1 \
       policies=my-app-policy \
       audience=https://kubernetes.default.svc \
       ttl=1h
   

Step 3: Install Vault Secrets Operator (VSO)

Task: Install the Vault Secrets Operator
Persona: Kubernetes administrator
Description: Follow the Vault Secrets Operator installation guide for detailed instructions.

1. Set up a Kubernetes namespace for VSO.

   kubectl create ns vault-secrets-operator-system
   

2. (Optional) Retrieve the CA certificate for Vault. Create a Kubernetes secret (vault-cacert) containing this certificate. This CA certificate is used to validate the Vault server’s certificate. This step is only required if the Vault server’s certificate is not already trusted. Replace vault.example.com with the hostname of your Vault server.

   openssl s_client -showcerts -connect vault.example.com:8200 \
     </dev/null 2>/dev/null|openssl x509 -outform PEM > vault-ca.pem
   
   kubectl create secret generic vault-cacert \
       --namespace=vault-secrets-operator-system \
       --from-literal=ca.crt="$(cat vault-ca.pem)"
   

3. Create the VSO values configuration file. This file specifies the Vault address, TLS server name, and TLS verification settings. If the Vault server's certificate is already trusted by Kubernetes, you can omit the caCertSecret flag. Replace vault.example.com with the hostname of your Vault server.

   cat <<EOF | tee vault-operator-values.yaml
   defaultVaultConnection:
     enabled: true
     address: "https://vault.example.com:8200"
     caCertSecret: "vault-cacert"
     tlsServerName: "vault.example.com"
     skipTLSVerify: false
   defaultAuthMethod:
     enabled: false
   EOF
   

4. Install the Vault Secrets Operator into the Kubernetes cluster. The VSO is responsible for reading secrets stored in Vault and making them available to workloads via Kubernetes secrets. The installation is performed using Helm, specifying the configuration values file.

   helm install vault-secrets-operator hashicorp/vault-secrets-operator \
     --namespace vault-secrets-operator-system \
     --create-namespace \
     --values vault-operator-values.yaml
   

5. Verify the installation by checking the status of the pods and services.

   kubectl get pods,svc -n vault-secrets-operator-system
   

Step 4: Configure Kubernetes

Task: Increase Vault’s permissions to retrieve Kubernetes service account metadata
Persona: Kubernetes administrator
Description: Grant the Vault service account read access to other Kubernetes service accounts to retrieve metadata, which is necessary for Vault to reference this metadata within the Vault policies. The access is granted by assigning a cluster role with the ability to view information about service accounts within the Kubernetes cluster.

1. Create a Kubernetes ClusterRole with permissions to list and get ServiceAccounts.

   kubectl create clusterrole read-serviceaccounts \
     --verb="list" \
     --verb="get" \
     --resource=serviceaccounts
   

2. Create a binding that grants the vault-sa service account within the vault-auth namespace the permissions defined in the read-serviceaccounts ClusterRole.

   kubectl create clusterrolebinding read-serviceaccounts-binding \
     --clusterrole=read-serviceaccounts \
     --serviceaccount=vault-auth:vault-sa
   

3. Verify the assigned permissions.

   kubectl auth can-i get serviceaccounts \
     --as system:serviceaccount:vault-auth:vault-sa
   

Task: Create application service account
Persona: Kubernetes administrator
Description: Create a service account for the application with the necessary annotations.

1. Set up a Kubernetes namespace for the application.

   kubectl create ns app-1
   

2. Create a service account including the metadata that will be used by Vault.

   cat <<EOF | tee service-account-my-app.yml
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: my-app
     namespace: app-1
     annotations:
       vault.hashicorp.com/alias-metadata-BusinessUnitName: "tenant-1"
       vault.hashicorp.com/alias-metadata-BusinessSegmentName: "team-a"
       vault.hashicorp.com/alias-metadata-AppName: "my-app"
   EOF
   
   kubectl apply -f service-account-my-app.yml
   

Task: Test secret read access
Persona: Vault administrator and Kubernetes administrator
Description: Verify that the Service Account can access the secret in Vault.

export APP_TOKEN=$(vault write -namespace="admin/tenant-1" -field="token" \
  auth/kubernetes/login \
  role=my-app \
  jwt=$(kubectl create token -n app-1 my-app))

VAULT_TOKEN=$APP_TOKEN vault kv get \
  -namespace="admin/tenant-1" \
  -mount=secret team-a/my-app/test 

Step 5: Deploy and sync a Kubernetes secret

Task: Deploy a secret
Persona: Kubernetes administrator
Description: Deploy and configure the VaultAuth and VaultStaticSecret resources to enable syncing secrets from Vault to Kubernetes.

1. Define and apply the VaultAuth configuration by creating a YAML configuration file for the VaultAuth resource. The VaultAuth resource specifies how the VSO authenticates with Vault, including the namespace, authentication method, and the role associated with the Kubernetes Service Account.

   cat <<EOF | tee static-auth.yaml
   apiVersion: secrets.hashicorp.com/v1beta1
   kind: VaultAuth
   metadata:
     name: static-auth
     namespace: app-1
   spec:
     namespace: admin/tenant-1
     method: kubernetes
     mount: kubernetes
     kubernetes:
       role: my-app
       serviceAccount: my-app
       audiences:
         - https://kubernetes.default.svc
   EOF
   
   kubectl apply -f static-auth.yaml
   

2. Define and apply the VaultStaticSecret configuration by creating a YAML configuration file for the VaultStaticSecret resource. This resource defines the secret to be synced from Vault to Kubernetes, including the type of secret, the path, and destination details. You also set a refresh interval (refreshAfter), which controls how often the secret is checked for updates. For demonstration purposes this is set rather short to 30s, but this could also be longer depending on the needs of the consuming applications.

   cat <<EOF | tee static-secret.yaml
   apiVersion: secrets.hashicorp.com/v1beta1
   kind: VaultStaticSecret
   metadata:
     name: vault-kv-app
     namespace: app-1
   spec:
     namespace: admin/tenant-1
     type: kv-v2
     mount: secret
     path: team-a/my-app/test 
     destination:
       name: secretkv
       create: true
   
     refreshAfter: 30s
     vaultAuthRef: static-auth
   EOF
   
   kubectl apply -f static-secret.yaml
   

3. Verify the correct deployment and synchronization of the secrets from Vault to Kubernetes. This includes describing the VaultStaticSecret resource to check for errors and retrieving the details of the synced Kubernetes Secret.

   # Describe the VaultStaticSecret resource
   kubectl describe vaultstaticsecret.secrets.hashicorp.com/vault-kv-app -n app-1
   
   # Get the details of the Kubernetes Secret
   kubectl get secrets secretkv -n app-1 -o yaml
   

Task: Verify by testing the rotation
Persona: Vault administrator and Kubernetes administrator
Description: Verify that the secret can be rotated and accessed correctly.

1. Test by updating the secret stored in Vault under the specified namespace and path. By changing the secret values (e.g., username and password), you trigger the rotation process. This ensures that the latest version of the secret is available for applications that access it.

   vault kv put -namespace="admin/tenant-1" \
     -mount=secret team-a/my-app/test \
     username=moin \
     password=consul
   

2. After updating the secret, use the following command to retrieve the synced secret. This step verifies that the secret has been correctly propagated to Kubernetes and that the updated values are accessible. The command extracts the password field from the Kubernetes Secret, decodes it, and prints the result.

   kubectl get secrets secretkv -n app-1 -o json \
     | jq -r '.data.password' | base64 --decode
   

Clean up infrastructure

Optional - If needed

# Uninstall VSO via Helm
helm uninstall vault-secrets-operator -n vault-secrets-operator-system

# Delete Kubernetes namespaces
kubectl delete ns vault-secrets-operator-system
kubectl delete ns vault-auth
kubectl delete ns app-1

Summary and related resources

This guide has walked you through the integration of HashiCorp Vault with Kubernetes service accounts to securely manage secrets across your Kubernetes environment. By leveraging Kubernetes service accounts and their metadata, you can ensure that only authorized applications have access to sensitive data.

For further learning, please refer to the following documentation and tutorials:

• Vault Secrets Operator installation guide
• Kubernetes authentication method
• Policy templating
• Namespace structure