Manage Vault access with Kubernetes service accounts

15min
|
HCP
Vault

Author: Jan Repnak

This integration pattern demonstrates how to implement Kubernetes service accounts and leverage their metadata to provide access to Vault namespaces and secrets through the Vault Secrets Operator. The goal is to exemplify HashiCorp's best practices for structuring Vault namespaces and mount paths.

By managing Vault access with Kubernetes service accounts, you gain the following benefits:

Scalable and secure: Leverages Kubernetes service accounts for secure access to Vault, eliminating the need for manual credential management.
Efficient: Automates secret management, reducing the risk of human error and ensuring that secrets are always up-to-date.

Target audience

This guide references the following roles:

Platform team responsible for Vault and Kubernetes

More specifically, this guide targets the following roles:

Vault administrator manages Vault namespaces, configures Kubernetes authentication and sets access policies.
Kubernetes administrator creates and manages Kubernetes service accounts, configures permissions for Vault to validate Kubernetes service accounts and deploys the Vault Secrets Operator.

Both roles need to collaborate to ensure that secrets are correctly synced from Vault and accessible for Kubernetes applications.

Prerequisites

To complete this guide, you need the following:

Vault Enterprise/HCP Vault Dedicated 1.16 or higher
Kubernetes 1.21 or higher
Administrative permissions for both Vault and Kubernetes

Background and best practices

Applications and workloads frequently need credentials to access resources, but the responsibility for managing the lifecycle of these credentials - such as creating, rotating, delivering, retiring, and revoking them - must not fall on the applications themselves. Automating these processes is essential for minimizing human involvement and reducing security risks. Vault's authentication methods and secret engines are specifically designed to facilitate this automation.

However, for applications to use Vault, they need an initial set of credentials, a challenge often referred to as "secret zero" or "secure introduction." You can address this issue by using a trusted identity provider. Public cloud platforms, and common container orchestration systems, typically provide identity services for the workloads they host (for example, AWS IAM roles and Kubernetes service accounts). We recommend leveraging these trusted identity systems so that the applications running in these environments do not require initial credentials, effectively eliminating the "secret zero" problem.

This guide explains a Vault namespace and mount path design that uses Kubernetes service accounts to authenticate applications. It demonstrates how to use service account metadata to authorize access to specific Vault secrets. The guide also shows how the Vault Secrets Operator allows applications to consume these secrets as native Kubernetes secrets or mounted CSI volumes.

Design overview

Vault namespaces and mount paths: There are multiple sub-namespaces under the admin namespace, each serving different tenants (for example, business units). Each sub-namespace has a dedicated K/V secrets engine.
Kubernetes authentication: Applications authenticate against a Kubernetes authentication mount, unique for each Vault sub-namespace if tenants do not share Kubernetes clusters. Applications use Kubernetes service accounts to authenticate, typically with the Vault Secrets Operator or Vault Agent Injector.
Policy management: ACL policy path templating simplifies policy management, allowing applications to access K/V secrets within their business segment (for example, team-a) and application-specific paths (for example, app-name).

Vault namespace design

Configure Vault namespace

The Vault administrator needs to create the necessary namespaces and enable the K/V secrets engine.

Create namespaces, if these do not exist.

$ vault namespace create admin
$ vault namespace create -namespace="admin" tenant-1

Set up a mount point called 'secret' as K/V version 2 within the admin/tenant-1 namespace.
```
$ vault secrets enable -namespace="admin/tenant-1" -path="secret" kv-v2
```

Create some secrets to reflect the K/V path structure.

Team A - App 1

$ vault kv put -namespace="admin/tenant-1" \
  -mount="secret" team-a/my-app/test \
  user=hello \
  password=kubernetes

Team B - App 2

$ vault kv put -namespace="admin/tenant-1" \
  -mount="secret" team-b/another-app/test \
  user=hello \
  password=nomad

Configure Kubernetes authentication

The Kubernetes administrator needs to create a Kubernetes service account for Vault and assign the necessary permissions for it to call the Kubernetes TokenReview API. Vault requires these permissions to validate other Kubernetes service accounts authenticating to Vault. You also create a service account token secret, which the subsequent task uses to configure the Kubernetes auth method in Vault.

Note

This guide uses a permanent Kubernetes service account token for Vault, meaning it does not expire. However, other methods are available for Vault to validate Kubernetes service accounts. For example, you could use a short-lived token provided inside the pod if Vault is also deployed in Kubernetes. Vault can also use the client's token if it is granted the system:auth-delegator cluster role in Kubernetes. For more information on these methods, refer to the Kubernetes auth documentation.

Create a service account for Vault.

$ kubectl create namespace vault-auth
$ kubectl create serviceaccount vault-sa -n vault-auth

Assign the permissions to act as a Kubernetes auth delegator.

$ kubectl create clusterrolebinding vault-auth-binding \
  -n vault-auth \
  --clusterrole=system:auth-delegator \
  --serviceaccount=vault-auth:vault-sa

Define a service account token secret that Vault uses to authenticate to Kubernetes.

$ cat <<EOF > vault-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: vault-sa-secret
  namespace: vault-auth
  annotations:
    kubernetes.io/service-account.name: vault-sa
type: kubernetes.io/service-account-token
EOF

Create the secret.

$ kubectl apply -f vault-secret.yaml

The Vault and Kubernetes administrators need to collaborate to ensure that the service account token secret is correctly created and used.

Enable Kubernetes authentication and import metadata from the Service Account to a Vault entity alias. The flag use_annotations_as_alias_metadata=true imports the metadata from the Service Account to a Vault entity alias, which you can use when constructing the Vault policies in the next task.

Note

Using Kubernetes service account metadata within Vault is only supported with the Kubernetes auth method, not the JWT/OIDC auth method.

Enable the Kubernetes authentication method in Vault.

$ vault auth enable -namespace="admin/tenant-1" kubernetes

Configure the Kubernetes authentication method with the Kubernetes service account token acquired in the previous step, the Kubernetes CA certificate and Kubernetes API URL.

First, export the values as environment variables.

$ export SA_TOKEN=$(kubectl get secret vault-sa-secret -n vault-auth \
  -o jsonpath="{.data.token}" | base64 --decode)

$ export KUBERNETES_CA=$(kubectl get secret vault-sa-secret -n vault-auth \
  -o jsonpath="{.data['ca\.crt']}" | base64 --decode)

$ export KUBERNETES_URL=$(kubectl config view --minify \
  -o jsonpath='{.clusters[0].cluster.server}')

Then, configure the Kubernetes authentication method.

$ vault write -namespace="admin/tenant-1" auth/kubernetes/config \
  use_annotations_as_alias_metadata=true \
  token_reviewer_jwt="${SA_TOKEN}" \
  kubernetes_host="${KUBERNETES_URL}" \
  kubernetes_ca_cert="${KUBERNETES_CA}"

The Vault administrator needs to set up the policy and role. Define a group policy and attach the application to the group, leveraging Kubernetes metadata. The following policy allows the application to access K/V secrets within their business segment and tied to the application name.

Find the accessor for the Kubernetes auth method within the admin/tenant-1 namespace

$ vault auth list -namespace="admin/tenant-1" -format=json \
  | jq -r '.["kubernetes/"].accessor' \
  > accessor_kubernetes.txt

Define a policy that makes use of Vault ACL policy templates.

$ tee my-app-policy.hcl <<EOF
# Allows to read K/V secrets 
path "secret/data/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.BusinessSegmentName}}/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.AppName}}/*" {
    capabilities = ["read"]
}
# Allows reading K/V secret versions and metadata
path "secret/metadata/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.BusinessSegmentName}}/{{identity.entity.aliases.$(cat accessor_kubernetes.txt).metadata.AppName}}/*" {
      capabilities = ["list", "read"]
}
# Allows the CSI driver to verify the Vault Enterprise license
path "sys/license/status" {
    capabilities = ["read"]
}
EOF

Then, create the policy.

$ vault policy write -namespace="admin/tenant-1" my-app-policy \
  my-app-policy.hcl

Associate the policy with a Vault authentication role.

$ vault write -namespace="admin/tenant-1" auth/kubernetes/role/my-app \
    bound_service_account_names=my-app \
    bound_service_account_namespaces=app-1 \
    policies=my-app-policy \
    audience=https://kubernetes.default.svc \
    ttl=1h

Install Vault Secrets Operator (VSO)

The Kubernetes administrator needs to install the Vault Secrets Operator (VSO). Follow the Vault Secrets Operator installation guide for detailed instructions.

Set up a Kubernetes namespace for VSO.

$ kubectl create ns vault-secrets-operator-system

(Optional) Retrieve the CA certificate for Vault. Create a Kubernetes secret (vault-cacert) containing this certificate. Vault uses this CA certificate to validate the Vault server's certificate. This step is only required if the Vault server's certificate is not already trusted. Replace vault.example.com with the hostname of your Vault server.
```
$ openssl s_client -showcerts -connect vault.example.com:8200 \
  </dev/null 2>/dev/null|openssl x509 -outform PEM > vault-ca.pem
```
```
$ kubectl create secret generic vault-cacert \
    --namespace=vault-secrets-operator-system \
    --from-literal=ca.crt="$(cat vault-ca.pem)"
```

(Optional) Create RBAC permissions for the Vault Secrets Operator CSI service account to access the vault-cacert secret.

$ cat <<EOF | tee vso-permissions.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: vault-secrets-operator-csi-role
  namespace: vault-secrets-operator-system
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get"]
    resourceNames:
      - vault-cacert
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: vault-secrets-operator-csi-rolebinding
  namespace: vault-secrets-operator-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: vault-secrets-operator-csi-role
subjects:
  - kind: ServiceAccount
    name: vault-secrets-operator-csi
    namespace: vault-secrets-operator-system
EOF

(Optional) Apply the RBAC permissions.

$ kubectl apply -f vso-permissions.yaml

(OpenShift Only) OpenShift requires the privileged Security Context Constraint (SCC) for the Vault Secrets Operator CSI service account. This permission enables the operator to manage CSI volumes and mount secrets into application pods.
```
$ oc adm policy add-scc-to-user privileged \
  -z vault-secrets-operator-csi \
  -n vault-secrets-operator-system
```
Create the VSO values configuration file. This file specifies the Vault address, TLS server name, and TLS verification settings. If the Vault server's certificate is already trusted by Kubernetes, you can omit the caCertSecret flag. Replace vault.example.com with the hostname of your Vault server.
Include the csi flag set to enabled: true to activate the Vault Secrets Operator's CSI driver feature.
```
$ cat <<EOF | tee vault-operator-values.yaml
defaultVaultConnection:
  enabled: true
  address: "https://vault.example.com:8200"
  caCertSecret: "vault-cacert"
  tlsServerName: "vault.example.com"
  skipTLSVerify: false
defaultAuthMethod:
  enabled: false
csi:
  enabled: true
EOF
```
Install the Vault Secrets Operator into the Kubernetes cluster. The VSO reads secrets stored in Vault and makes them available to workloads as Kubernetes secrets. Helm performs the installation using the configuration values file.
```
$ helm install vault-secrets-operator hashicorp/vault-secrets-operator \
  --namespace vault-secrets-operator-system \
  --create-namespace \
  --values vault-operator-values.yaml
```
Verify the installation by checking the status of the pods and services.
```
$ kubectl get pods,svc -n vault-secrets-operator-system
```
(OpenShift Only) Next, label the csi.vso.hashicorp.com CSIDriver resource with the security.openshift.io/csi-ephemeral-volume-profile=restricted key. This instructs OpenShift to trust the Vault Secrets Operator CSI driver for use in restricted environments and to allow pods using the restricted security profile to attach its ephemeral volumes.
```
$ oc label csidriver csi.vso.hashicorp.com \
  security.openshift.io/csi-ephemeral-volume-profile=restricted --overwrite
$ oc get csidriver csi.vso.hashicorp.com
```

Configure Kubernetes

The Kubernetes administrator needs to increase Vault's permissions to retrieve Kubernetes service account metadata.

Grant the Vault service account read access to other Kubernetes service accounts to retrieve metadata, which is necessary for Vault to reference this metadata within the Vault policies. Kubernetes grants the access by assigning a cluster role with the ability to view information about service accounts within the Kubernetes cluster.

Create a Kubernetes ClusterRole with permissions to list and get ServiceAccounts.

$ kubectl create clusterrole read-serviceaccounts \
  --verb="list" \
  --verb="get" \
  --resource=serviceaccounts

Create a binding that grants the vault-sa service account within the vault-auth namespace the permissions defined in the read-serviceaccounts ClusterRole.
```
$ kubectl create clusterrolebinding read-serviceaccounts-binding \
  --clusterrole=read-serviceaccounts \
  --serviceaccount=vault-auth:vault-sa
```

Verify the assigned permissions.

$ kubectl auth can-i get serviceaccounts \
  --as system:serviceaccount:vault-auth:vault-sa

The Kubernetes administrator needs to create a service account for the application with the necessary annotations.

Set up a Kubernetes namespace for the application.
```
$ kubectl create ns app-1
```

Create a service account including the metadata that Vault uses.

$ cat <<EOF | tee service-account-my-app.yml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: my-app
  namespace: app-1
  annotations:
    vault.hashicorp.com/alias-metadata-BusinessUnitName: "tenant-1"
    vault.hashicorp.com/alias-metadata-BusinessSegmentName: "team-a"
    vault.hashicorp.com/alias-metadata-AppName: "my-app"
EOF

$ kubectl apply -f service-account-my-app.yml

To confirm the service account has the correct permissions, the Vault and Kubernetes administrators can test read access by authenticating with the service account's token and attempting to read the secret.

$ export APP_TOKEN=$(vault write -namespace="admin/tenant-1" -field="token" \
  auth/kubernetes/login \
  role=my-app \
  jwt=$(kubectl create token -n app-1 my-app --audience=https://kubernetes.default.svc))

$ VAULT_TOKEN=$APP_TOKEN vault kv get \
  -namespace="admin/tenant-1" \
  -mount=secret team-a/my-app/test

Option 1: Deploy and sync a Kubernetes secret

The Kubernetes administrator needs to deploy and configure the VaultAuth and VaultStaticSecret resources to enable syncing secrets from Vault to Kubernetes.

Define and apply the VaultAuth configuration by creating a YAML configuration file for the VaultAuth resource. The VaultAuth resource specifies how the VSO authenticates with Vault, including the namespace, authentication method, and the role associated with the Kubernetes Service Account.

$ cat <<EOF | tee static-auth.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: static-auth
  namespace: app-1
spec:
  namespace: admin/tenant-1
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: my-app
    serviceAccount: my-app
    audiences:
      - https://kubernetes.default.svc
EOF

$ kubectl apply -f static-auth.yaml

Define and apply the VaultStaticSecret configuration by creating a YAML configuration file for the VaultStaticSecret resource. This resource defines the secret to sync from Vault to Kubernetes, including the type of secret, the path, and destination details. You also set a refresh interval (refreshAfter), which controls how often VSO checks the secret for updates. For demonstration purposes this value is rather short at 30 s, but you can increase it depending on the needs of the consuming applications.
```
$ cat <<EOF | tee static-secret.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vault-kv-app
  namespace: app-1
spec:
  namespace: admin/tenant-1
  type: kv-v2
  mount: secret
  path: team-a/my-app/test 
  destination:
    name: secretkv
    create: true
  refreshAfter: 30s
  vaultAuthRef: static-auth
EOF
```
```
$ kubectl apply -f static-secret.yaml
```
Verify the correct deployment and synchronization of the secrets from Vault to Kubernetes. First, describe the VaultStaticSecret resource to check for errors.
Describe the VaultStaticSecret resource
```
$ kubectl describe vaultstaticsecret.secrets.hashicorp.com/vault-kv-app -n app-1
```
Next, retrieve the details of the synced Kubernetes secret.
```
$ kubectl get secrets secretkv -n app-1 -o yaml
```

The Vault and Kubernetes administrators need to verify that the secret rotates and remains accessible correctly.

Test by updating the secret stored in Vault under the specified namespace and path. By changing the secret values (for example, username and password), you trigger the rotation process. This ensures that the latest version of the secret is available for applications that access it.
```
$ vault kv put -namespace="admin/tenant-1" \
  -mount=secret team-a/my-app/test \
  username=moin \
  password=consul
```
After updating the secret, use the following command to retrieve the synced secret. This step verifies that the secret has been correctly propagated to Kubernetes and that the updated values are accessible. The command extracts the password field from the Kubernetes Secret, decodes it, and prints the result.
```
$ kubectl get secrets secretkv -n app-1 -o json \
  | jq -r '.data.password' | base64 --decode
```

Option 2: Deploy and mount a secret using CSI

The Kubernetes administrator needs to deploy and configure the VaultAuth and CSISecrets resources to enable syncing secrets from Vault to Kubernetes using the CSI driver.

$ cat <<EOF | tee static-auth.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata:
  name: static-auth
  namespace: app-1
spec:
  namespace: admin/tenant-1
  method: kubernetes
  mount: kubernetes
  kubernetes:
    role: my-app
    serviceAccount: my-app
    audiences:
      - https://kubernetes.default.svc
EOF

$ kubectl apply -f static-auth.yaml

Define and apply the CSISecrets configuration by creating a YAML configuration file for the VaultStaticSecret resource. The CSISecrets resource defines the secrets to sync from Vault, including the mount path, secret path, access control patterns for service accounts/namespaces/pods, and container state sync configuration.

$ cat <<EOF | tee csi-secret.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: CSISecrets
metadata:
  name: vault-kv-app
  namespace: app-1
spec: 
  vaultAuthRef:
    name: static-auth
  secrets: 
    vaultStaticSecrets:
    - mount: secret
      path: team-a/my-app/test
      type: kv-v2
  accessControl:
    serviceAccountPattern: "my-app"
    namespacePatterns:
    - "app-1"
    podNamePatterns:
    - "^my-app-"
  syncConfig:
    containerState: 
      namePattern: "^(app|sidecar)$"
EOF

$ kubectl apply -f csi-secret.yaml

Verify the CSISecrets resource deployment.

$ kubectl get CSISecrets vault-kv-app -n app-1
$ kubectl describe CSISecrets vault-kv-app -n app-1

Create a deployment manifest for a test application. The my-app pods mount the CSI volume and continuously read the synchronized username and password values.

$ cat <<EOF | tee csi-deploy.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: app-1
  labels:
    app.kubernetes.io/component: my-app
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: my-app
  replicas: 2
  template:
    metadata:
      labels:
        app.kubernetes.io/component: my-app
    spec:
      serviceAccountName: my-app
      containers:
      - name: app-1
        image: busybox:latest
        command: ['/bin/sh', '-c', '--']
        args: ['while true; do cat /var/run/csi-secrets/static_secret_0_username && cat /var/run/csi-secrets/static_secret_0_password; echo; sleep 10; done;']
        volumeMounts:
        - name: csi-secrets
          mountPath: /var/run/csi-secrets
      volumes:
      - name: csi-secrets
        csi:
          driver: csi.vso.hashicorp.com
          volumeAttributes:
            csiSecretsName: vault-kv-app
            csiSecretsNamespace: app-1
EOF

Deploy the application and stream the logs to verify the secret mounted.

$ kubectl apply -f csi-deploy.yaml -n app-1
$ kubectl logs deploy/my-app -n app-1 --follow

Optionally clean up resources

Once you finish this guide, you can clean up the resources.

First, uninstall VSO using Helm.

$ helm uninstall vault-secrets-operator -n vault-secrets-operator-system

Delete Kubernetes namespaces.

$ kubectl delete ns vault-secrets-operator-system
$ kubectl delete ns vault-auth
$ kubectl delete ns app-1

Conclusion

This guide has walked you through the integration of HashiCorp Vault with Kubernetes service accounts to securely manage secrets across your Kubernetes environment. By leveraging Kubernetes service accounts and their metadata, you can ensure that only authorized applications have access to sensitive data.

For further learning, refer to the following documentation and tutorials:

Manage Vault with Terraform

Migrate to Vault Integrated Storage