Manage Vault access with Kubernetes service accounts

Author: Jan Repnak

This integration pattern demonstrates how to implement Kubernetes service accounts and leverage their metadata to provide access to Vault namespaces and secrets via Vault Secrets Operator. The goal is to exemplify HashiCorp's best practices for structuring Vault namespaces and mount paths.

By managing Vault access with Kubernetes service accounts, you gain the following benefits:

• Scalable and secure: Leverages Kubernetes service accounts for secure access to Vault, eliminating the need for manual credential management.
• Efficient: Automates secret management, reducing the risk of human error and ensuring that secrets are always up-to-date.

Target audience

This guide references the following roles:

• Platform team responsible for Vault and Kubernetes

More specifically, this guide is intended for the following roles:

• Vault administrator manages Vault namespaces, configures Kubernetes authentication and sets access policies.
• Kubernetes administrator creates and manages Kubernetes service accounts, configures permissions for Vault to validate Kubernetes service accounts and deploys the Vault Secrets Operator.

Both roles will need to collaborate to ensure that secrets are correctly synced from Vault and accessible for Kubernetes applications.

Prerequisites

To complete this guide, you will need the following:

• Vault Enterprise/HCP Vault Dedicated 1.16 or higher
• Kubernetes 1.21 or higher
• Administrative permissions for both Vault and Kubernetes

Background and best practices

Applications and workloads frequently need credentials to access resources, but the responsibility for managing the lifecycle of these credentials - such as creating, rotating, delivering, retiring, and revoking them - should not fall on the applications themselves. Automating these processes is essential for minimizing human involvement and reducing security risks. Vault's authentication methods and secret engines are specifically designed to facilitate this automation.

However, for applications to use Vault, they need an initial set of credentials, a challenge often referred to as "secret zero" or "secure introduction." This issue can be addressed by using a trusted identity provider. Public cloud platforms, and common container orchestration systems, typically provide identity services for the workloads they host (e.g. AWS IAM roles and Kubernetes service accounts). We recommend leveraging these trusted identity systems so that the applications running in these environments do not require initial credentials, effectively eliminating the "secret zero" problem.

This guide explains a Vault namespace and mount path design that uses Kubernetes service accounts as an identity mechanism for applications to securely access Vault-stored secrets. Additionally, it provides an example of how Kubernetes service account metadata can be used to grant access to specific Vault secrets, and how these secrets can be made available to applications as native Kubernetes secrets using the Vault Secrets Operator.

Design overview

• Vault namespaces and mount paths: There are multiple sub-namespaces under the admin namespace, each serving different tenants (e.g., business units). Each sub-namespace has a dedicated K/V secrets engine.
• Kubernetes authentication: Applications authenticate against a Kubernetes authentication mount, unique for each Vault sub-namespace if tenants do not share Kubernetes clusters. Authentication is done using Kubernetes service accounts, typically with the Vault Secrets Operator or Vault Agent Injector.
• Policy management: ACL policy path templating is used to simplify policy management, allowing applications to access K/V secrets within their business segment (e.g., team-a) and application-specific paths (e.g., my-app).

Configure Vault namespace

The Vault administrator needs to create the necessary namespaces and enable the K/V secrets engine.

1. Create namespaces, if these do not exist.

   $ vault namespace create admin
   $ vault namespace create -namespace="admin" tenant-1
   

2. Set up a mount point called 'secret' as K/V version 2 within the namespace 'admin/tenant-1'.

   $ vault secrets enable -namespace="admin/tenant-1" -path="secret" kv-v2
   

3. Create some secrets to reflect the K/V path structure.

   Team A - App 1

   $ vault kv put -namespace="admin/tenant-1" \
     -mount="secret" team-a/my-app/test \
     user=hello \
     password=kubernetes
   

   Team B - App 2

   $ vault kv put -namespace="admin/tenant-1" \
     -mount="secret" team-b/another-app/test \
     user=hello \
     password=nomad
   

Configure Kubernetes authentication

The Kubernetes administrator needs to create a Kubernetes service account for Vault

In this step, you will create a Kubernetes service account for Vault and assign the necessary permissions for it to call the Kubernetes TokenReview API. This is needed in order for Vault to be able to validate other Kubernetes service accounts that are authenticating to Vault. Additionally, you will create a service account token secret, which will be used in the subsequent task to configure the Kubernetes auth method in Vault.

1. Create a service account for Vault.

   $ kubectl create namespace vault-auth
   $ kubectl create serviceaccount vault-sa -n vault-auth
   

2. Assign the permissions to act as a Kubernetes auth delegator.

   $ kubectl create clusterrolebinding vault-auth-binding \
     -n vault-auth \
     --clusterrole=system:auth-delegator \
     --serviceaccount=vault-auth:vault-sa
   

3. Define a service account token secret that is used by Vault to authenticate to Kubernetes.

   $ cat <<EOF > vault-secret.yaml
   apiVersion: v1
   kind: Secret
   metadata:
     name: vault-sa-secret
     namespace: vault-auth
     annotations:
       kubernetes.io/service-account.name: vault-sa
   type: kubernetes.io/service-account-token
   EOF
   

   Create the secret.

   $ kubectl apply -f vault-secret.yaml
   

The Vault and Kubernetes administrators will need to collaborate to ensure that the service account token secret is correctly created and used.

Enable Kubernetes authentication and import metadata from the Service Account to a Vault entity alias. The flag use_annotations_as_alias_metadata=true imports the metadata from the Service Account to a Vault entity alias, which you can use when constructing the Vault policies in the next task.

1. Enable the Kubernetes authentication method in Vault.

   $ vault auth enable -namespace="admin/tenant-1" kubernetes
   

2. Configure the Kubernetes authentication method with the Kubernetes service account token acquired in the previous step, the Kubernetes CA certificate and Kubernetes API URL.

   First, export the values as environment variables.

   $ export SA_TOKEN=$(kubectl get secret vault-sa-secret -n vault-auth \
     -o jsonpath="{.data.token}" | base64 --decode)
   
   $ export KUBERNETES_CA=$(kubectl get secret vault-sa-secret -n vault-auth \
     -o jsonpath="{.data['ca\.crt']}" | base64 --decode)
   
   $ export KUBERNETES_URL=$(kubectl config view --minify \
     -o jsonpath='{.clusters[0].cluster.server}')
   

   Then, configure the Kubernetes authentication method.

   $ vault write -namespace="admin/tenant-1" auth/kubernetes/config \
     use_annotations_as_alias_metadata=true \
     token_reviewer_jwt="${SA_TOKEN}" \
     kubernetes_host="${KUBERNETES_URL}" \
     kubernetes_ca_cert="${KUBERNETES_CA}"
   

The Vault administrator needs to set up the policy and role. Define a group policy and attach the application to the group, leveraging Kubernetes metadata. The following policy allows the application to access K/V secrets within their business segment and tied to the application name.

1. Find the accessor for the Kubernetes auth method within the 'admin/tenant-1' namespace

   $ vault auth list -namespace="admin/tenant-1" -format=json \
     | jq -r '.["kubernetes/"].accessor' \
     > accessor_kubernetes.txt
   

2. Define a policy that makes use of Vault ACL policy templates.

   $ tee my-app-policy.hcl <<EOF
   # Allows to read K/V secrets 
   path "secret/data/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.BusinessSegmentName}}/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.AppName}}/*" {
       capabilities = ["read"]
   }
   # Allows reading K/V secret versions and metadata
   path "secret/metadata/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.BusinessSegmentName}}/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.AppName}}/*" {
         capabilities = ["list", "read"]
   }
   EOF
   

   Then, create the policy.

   $ vault policy write -namespace="admin/tenant-1" my-app-policy \
     my-app-policy.hcl
   

3. Associate the policy with a Vault authentication role.

   $ vault write -namespace="admin/tenant-1" auth/kubernetes/role/my-app \
       bound_service_account_names=my-app \
       bound_service_account_namespaces=app-1 \
       policies=my-app-policy \
       audience=https://kubernetes.default.svc \
       ttl=1h
   

Install Vault Secrets Operator (VSO)

The Kubernetes administrator needs to install the Vault Secrets Operator (VSO). Follow the Vault Secrets Operator installation guide for detailed instructions.

1. Set up a Kubernetes namespace for VSO.

   $ kubectl create ns vault-secrets-operator-system
   

2. (Optional) Retrieve the CA certificate for Vault. Create a Kubernetes secret (vault-cacert) containing this certificate. This CA certificate is used to validate the Vault server's certificate. This step is only required if the Vault server's certificate is not already trusted. Replace vault.example.com with the hostname of your Vault server.

   $ openssl s_client -showcerts -connect vault.example.com:8200 \
     </dev/null 2>/dev/null|openssl x509 -outform PEM > vault-ca.pem
   
   $ kubectl create secret generic vault-cacert \
       --namespace=vault-secrets-operator-system \
       --from-literal=ca.crt="$(cat vault-ca.pem)"
   

3. Create the VSO values configuration file. This file specifies the Vault address, TLS server name, and TLS verification settings. If the Vault server's certificate is already trusted by Kubernetes, you can omit the caCertSecret flag. Replace vault.example.com with the hostname of your Vault server.

   $ cat <<EOF | tee vault-operator-values.yaml
     defaultVaultConnection:
       enabled: true
       address: "https://vault.example.com:8200"
       caCertSecret: "vault-cacert"
       tlsServerName: "vault.example.com"
       skipTLSVerify: false
     defaultAuthMethod:
       enabled: false
     EOF
   

4. Install the Vault Secrets Operator into the Kubernetes cluster. The VSO is responsible for reading secrets stored in Vault and making them available to workloads via Kubernetes secrets. The installation is performed using Helm, specifying the configuration values file.

   $ helm install vault-secrets-operator hashicorp/vault-secrets-operator \
     --namespace vault-secrets-operator-system \
     --create-namespace \
     --values vault-operator-values.yaml
   

5. Verify the installation by checking the status of the pods and services.

   $ kubectl get pods,svc -n vault-secrets-operator-system
   

Configure Kubernetes

The Kubernetes administrator needs to increase Vault's permissions to retrieve Kubernetes service account metadata.

Grant the Vault service account read access to other Kubernetes service accounts to retrieve metadata, which is necessary for Vault to reference this metadata within the Vault policies. The access is granted by assigning a cluster role with the ability to view information about service accounts within the Kubernetes cluster.

1. Create a Kubernetes ClusterRole with permissions to list and get ServiceAccounts.

   $ kubectl create clusterrole read-serviceaccounts \
     --verb="list" \
     --verb="get" \
     --resource=serviceaccounts
   

2. Create a binding that grants the vault-sa service account within the vault-auth namespace the permissions defined in the read-serviceaccounts ClusterRole.

   $ kubectl create clusterrolebinding read-serviceaccounts-binding \
     --clusterrole=read-serviceaccounts \
     --serviceaccount=vault-auth:vault-sa
   

3. Verify the assigned permissions.

   $ kubectl auth can-i get serviceaccounts \
     --as system:serviceaccount:vault-auth:vault-sa
   

The Kubernetes administrator needs to create a service account for the application with the necessary annotations.

1. Set up a Kubernetes namespace for the application.

   $ kubectl create ns app-1
   

2. Create a service account including the metadata that will be used by Vault.

   $ cat <<EOF | tee service-account-my-app.yml
     apiVersion: v1
     kind: ServiceAccount
     metadata:
       name: my-app
       namespace: app-1
       annotations:
         vault.hashicorp.com/alias-metadata-BusinessUnitName: "tenant-1"
         vault.hashicorp.com/alias-metadata-BusinessSegmentName: "team-a"
         vault.hashicorp.com/alias-metadata-AppName: "my-app"
     EOF
   
   $ kubectl apply -f service-account-my-app.yml
   

The Vault and Kubernetes administrators need to test the read access to the secret by verifying that the service account can access the secret in Vault.

$ export APP_TOKEN=$(vault write -namespace="admin/tenant-1" -field="token" \
  auth/kubernetes/login \
  role=my-app \
  jwt=$(kubectl create token -n app-1 my-app))

VAULT_TOKEN=$APP_TOKEN vault kv get \
  -namespace="admin/tenant-1" \
  -mount=secret team-a/my-app/test 

Deploy and sync a Kubernetes secret

The Kubernetes administrator needs to deploy and configure the VaultAuth and VaultStaticSecret resources to enable syncing secrets from Vault to Kubernetes.

1. Define and apply the VaultAuth configuration by creating a YAML configuration file for the VaultAuth resource. The VaultAuth resource specifies how the VSO authenticates with Vault, including the namespace, authentication method, and the role associated with the Kubernetes Service Account.

   $ cat <<EOF | tee static-auth.yaml
     apiVersion: secrets.hashicorp.com/v1beta1
     kind: VaultAuth
     metadata:
       name: static-auth
       namespace: app-1
     spec:
       namespace: admin/tenant-1
       method: kubernetes
       mount: kubernetes
       kubernetes:
         role: my-app
         serviceAccount: my-app
         audiences:
           - https://kubernetes.default.svc
     EOF
   
   $ kubectl apply -f static-auth.yaml
   

2. Define and apply the VaultStaticSecret configuration by creating a YAML configuration file for the VaultStaticSecret resource. This resource defines the secret to be synced from Vault to Kubernetes, including the type of secret, the path, and destination details. You also set a refresh interval (refreshAfter), which controls how often the secret is checked for updates. For demonstration purposes this is set rather short to 30s, but this could also be longer depending on the needs of the consuming applications.

   $ cat <<EOF | tee static-secret.yaml
     apiVersion: secrets.hashicorp.com/v1beta1
     kind: VaultStaticSecret
     metadata:
       name: vault-kv-app
       namespace: app-1
     spec:
       namespace: admin/tenant-1
       type: kv-v2
       mount: secret
       path: team-a/my-app/test 
       destination:
         name: secretkv
         create: true
   
       refreshAfter: 30s
       vaultAuthRef: static-auth
     EOF
   
   $ kubectl apply -f static-secret.yaml
   

3. Verify the correct deployment and synchronization of the secrets from Vault to Kubernetes. First, describe the VaultStaticSecret resource to check for errors.

   Describe the VaultStaticSecret resource

   $ kubectl describe vaultstaticsecret.secrets.hashicorp.com/vault-kv-app -n app-1
   

   Next, retrieve the details of the synced Kubernetes secret.

   Get the details of the Kubernetes secret

   $ kubectl get secrets secretkv -n app-1 -o yaml
   

The Vault and Kubernetes administrators need to verify that the secret can be rotated and accessed correctly.

1. Test by updating the secret stored in Vault under the specified namespace and path. By changing the secret values (e.g., username and password), you trigger the rotation process. This ensures that the latest version of the secret is available for applications that access it.

   $ vault kv put -namespace="admin/tenant-1" \
     -mount=secret team-a/my-app/test \
     username=moin \
     password=consul
   

2. After updating the secret, use the following command to retrieve the synced secret. This step verifies that the secret has been correctly propagated to Kubernetes and that the updated values are accessible. The command extracts the password field from the Kubernetes Secret, decodes it, and prints the result.

   $ kubectl get secrets secretkv -n app-1 -o json \
     | jq -r '.data.password' | base64 --decode
   

(Optional) Clean up resources

Once you are done with this guide, you can clean up the resources.

First, uninstall VSO via Helm.

$ helm uninstall vault-secrets-operator -n vault-secrets-operator-system

Delete Kubernetes namespaces.

$ kubectl delete ns vault-secrets-operator-system
$ kubectl delete ns vault-auth
$ kubectl delete ns app-1

Conclusion

This guide has walked you through the integration of HashiCorp Vault with Kubernetes service accounts to securely manage secrets across your Kubernetes environment. By leveraging Kubernetes service accounts and their metadata, you can ensure that only authorized applications have access to sensitive data.

For further learning, refer to the following documentation and tutorials:

• Vault Secrets Operator installation guide
• Kubernetes authentication method
• Policy templating
• Namespace structure