Simplify Vault policy management for multiple Kubernetes workloads

21min
|
Vault

HashiCorp Vault provides centralized secrets management at scale, supporting several different applications and workflows.

Challenge

An organization may have hundreds of applications spread out across multiple clusters. Creating a unique policy per application is an easy way to provide initial access. However, without careful planning, it may be difficult to effectively manage and audit application policies over time.

If you are not familiar with Vault ACL policies, follow the access controls with Vault policies tutorial.

Solution

In the Simplify management of Vault policies with ACL policy templating tutorial, you learned how to use templates in a Vault ACL policy for users authenticating to Vault.

In this tutorial, you will use Vault's policy path templating features to create a single ACL policy that allows different Kubernetes workloads to access unique secret paths in a secure and auditable way. This will reduce the total number of policies that you need to manage.

Prerequisites

Vault installed.
jq for consuming Vault JSON formatted output and capturing specific fields.
Kubectl installed.
Helm installed.
minikube installed.
Docker installed.

Set up the lab

Start minikube

minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters locally inside Virtual Machines (VM) on your system.

Start a Kubernetes cluster.

$ minikube start --driver=docker
😄  minikube v1.12.1 on Darwin 10.15.6
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating docker container (CPUs=2, Memory=1991MB) ...
🐳  Preparing Kubernetes v1.18.3 on Docker 19.03.2 ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube"

The initialization process takes several minutes as it retrieves any necessary dependencies and executes various container images.

Verify the status of the minikube cluster.
```
$ minikube status

minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
```
The host, kubelet, and apiserver report that they are running. kubectl, a command line interface (CLI) for running commands against the Kubernetes cluster, is also configured to communicate with this recently started cluster.

Install the Vault Helm chart

The Vault Helm chart installs a Vault pod, along with a vault service and a vault service account.

Add the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

Update all the repositories to ensure helm is aware of the latest versions.

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈Happy Helming!⎈

Install the latest version of the Vault server running in development mode.
```
$ helm install vault hashicorp/vault --set "server.dev.enabled=true"
NAME: vault
## ...
```
The Vault pod deploys in the default namespace.
Display all the pods within the default namespace.
```
$ kubectl get pods
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          80s
vault-agent-injector-5945fb98b5-tpglz   1/1     Running   0          80s
```
The vault-0 pod runs a Vault server in development mode.
Development mode
Running a Vault server in development is automatically initialized and unsealed. This is ideal in a learning environment but not recommended for a production environment.
Wait until the vault-0 pod is running and ready (1/1).

Configure Kubernetes auth method

Start an interactive shell session on the vault-0 pod.
```
$ kubectl exec -it vault-0 -- /bin/sh
/ $
```
Your system prompt changes to a new prompt / $. Commands issued at this prompt run on the vault-0 container.
Based on the path naming convention, set the CLUSTER_NAME environment value to minikube, the default cluster name when using minikube.
```
$ export CLUSTER_NAME="minikube"
```

Enable the Kubernetes authentication method.

$ vault auth enable -path=$CLUSTER_NAME kubernetes
Success! Enabled kubernetes auth method at: minikube/

Configure the Kubernetes authentication method to use the location of the Kubernetes host.
```
$ vault write auth/$CLUSTER_NAME/config \
   kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443"
```
Example output:
```
Success! Data written to: auth/minikube/config
```
The environment variable KUBERNETES_PORT_443_TCP_ADDR references the internal network address of the Kubernetes host.

Enable the audit log.

$ vault audit enable file file_path=/tmp/audit_log.json

Exit the vault-0 pod.
```
$ exit
```

Deploy the example application

Retrieve the web application and additional configuration by cloning the hashicorp-education/learn-vault-policy-path-templates repository from GitHub.
```
$ git clone https://github.com/hashicorp-education/learn-vault-policy-path-templates
```
Change into the learn-vault-policy-path-templates directory.
```
$ cd learn-vault-policy-path-templates
```
Working directory
This tutorial assumes that you run the remainder of commands within this directory.

Establish naming conventions

Establish naming standards that allow applications to login and read secrets from consistent and predictable paths. Each application in an organization will have a unique identifier such as an application ID and be part of a logical group such as a line of business (LOB). You can use these elements to create the naming convention.

Example

The following table helps plan the various relevant paths within Vault. Build out documentation or a representative use-case in your organization. For this tutorial, the application grouping uses Kubernetes namespaces, where each namespace represents a line of business (LOB). You will also create a Kubernetes service account per application which is a common best practice.

You will choose the LOB name retail, and app100, app200 and app300 as application IDs for the examples in this guide.

Vault item	Example for Kubernetes applications
Auth method mount path	The default path is kubernetes. The recommended practice is making it specific to a cluster name, since each cluster has a different API endpoint. - Naming convention: `<cluster-name>` - Examples: `minikube`, `gke-useast1` etc.
Auth method role name	Make the role name something predictable for the application to login. - Naming convention: `<namespace>-<app-name>` - Example: `retail-app100`
Login path	The login endpoint for Kubernetes auth method will be: - API endpoint: `auth/<auth-mount>/login` Following this tutorial, these will translate to: - Naming convention: `auth/<cluster-name>/login/` - Example: `auth/minikube/login`
Secret engine mount path	You can mount secret engines at custom paths. In this tutorial, you will mount one secret engine per Kubernetes Namespace: - Naming convention: `secrets/<namespace>` - Example: `secret/data/retail`
Secret access path	Each application will access secrets at the following path: - Naming convention: `secrets/<namespace>/<app-name>` - Example: `secret/data/retail/app100`

The actual paths in your organization will vary depending on how the organization is structured.

Write a templated policy

Upon login, Vault auto-populates certain metadata that you can use as part of templated ACL policies. You can use these templating parameters by including custom metadata as part of the secret access path.

The available templating parameters table shows various dynamic substitutions for a templated ACL policy. These parameters allow you to construct a policy that aligns with the naming conventions from earlier in the tutorial. The <metadata key> dynamic element is the focus for this scenario. For the Kubernetes auth method, the available metadata keys are in the login endpoint API sample response.

Recall from the table in the establish naming conventions section that the secret access path is: secrets/<namespace>/<app-name>. Below is a mapping between the desired path elements and metadata keys.

<namespace>: identity.entity.aliases.<mount accessor>.metadata.service_account_namespace. This is the Kubernetes Namespace.
<app-name>: identity.entity.aliases.<mount accessor>.metadata.service_account_name. This is the Kubernetes Service Account name for the application.

To access the metadata for an auth method requires its unique mount accessor. Display the mount accessors for all auth methods.

$ kubectl exec --namespace default vault-0 -- vault auth list

Example output:

Path           Type          Accessor                    Description
----           ----          --------                    -----------
minikube/      kubernetes    auth_kubernetes_f9be05d8    n/a
token/         token         auth_token_8b8425b0         token based credentials

The Accessor column displays the kubernetes auth method's mount accessor.

Create a variable to store the mount accessor for the kubernetes auth method.

$ export MOUNT_ACCESSOR=$(kubectl exec --namespace default vault-0 -- vault auth list -format=json | jq -r '."minikube/".accessor')

Create an ACL policy, named minikube-kv-read, that grants read and list capabilities to secrets at the path secret/<service_account_namespace>/<service_account_name>.
```
$ kubectl exec -i --namespace default vault-0 -- \
  vault policy write minikube-kv-read - << EOF
    path "secret/data/{{identity.entity.aliases.$MOUNT_ACCESSOR.metadata.service_account_namespace}}/{{identity.entity.aliases.$MOUNT_ACCESSOR.metadata.service_account_name}}" {
        capabilities=["read","list"]
    }
EOF
```
This templatized policy uses identity.entity.aliases.$MOUNT_ACCESSOR.metadata.service_account_namespace and identity.entity.aliases.$MOUNT_ACCESSOR.metadata.service_account_name to create a policy that is flexible based on the Kubernetes namespace and service account.
Refer to the complete list of available templating parameters for more information.
Display the minikube-kv-read policy.
```
$ kubectl exec --namespace default vault-0 -- vault policy read minikube-kv-read
```
The created policy displays the templatized path with the kubernetes/ auth method's mount accessor in the path.

Onboard applications

In this step, you will deploy a demo application named basic-example and ensure that it can read the intended secret using the templated policy. You will repeat the process once to see how multiple applications can access unique paths using the same ACL policy.

Create Role and write secret

Create a Kubernetes Service Account and a Vault login role. The role allows us to enforce the application Service Account name and Namespace, and associate the templated ACL policy. You will also store a secret in the KV secrets engine for this application.

Create an APP_NAME environment variable.
```
$ export APP_NAME=app100
```
Create an APP_NAMESPACE environment variable.
```
$ export APP_NAMESPACE=retail
```
Create an CLUSTER_NAME environment variable.
```
$ export CLUSTER_NAME=minikube
```
Create a Kubernetes namespace called, "retail".
```
$ kubectl create ns $APP_NAMESPACE
```

Create application Service Account.

$ kubectl create sa $APP_NAME -n $APP_NAMESPACE

Each application requires a unique Kubernetes auth method role path based on the application's service account and namespace. Create a Kubernetes auth role named $APP_NAMESPACE-$APP_NAME.

$ kubectl exec --namespace default -it vault-0 --  \
    vault write auth/$CLUSTER_NAME/role/$APP_NAMESPACE-$APP_NAME \
       bound_service_account_names=$APP_NAME \
       bound_service_account_namespaces=$APP_NAMESPACE \
       policies=$CLUSTER_NAME-kv-read \
       period=120s

Example output:

Success! Data written to: auth/minikube/role/retail-app100

Write a secret for the secret/data/<APP_NAMESPACE>/<APP_NAME> path.

$ kubectl exec --namespace default -it vault-0 -- \
    vault kv put secret/$APP_NAMESPACE/$APP_NAME \
      app=$APP_NAME \
      username=demo \
      password=test

Example output:

====== Secret Path ======
secret/data/retail/app100

======= Metadata =======
Key                Value
---                -----
created_time       2023-10-12T18:47:54.761554676Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Deploy application

You will now create a Kubernetes Deployment object. As a first step, you will make a series of substitutions using the sed command to create a Deployment manifest YAML file from a template.

The deployment YAML file contains a SECRET_PATH environment variable with the value: secret/data/retail/app123. The application will try to read secrets from this path.
The VAULT_ADDR environment variable reflects the DNS name where application pods can reach Vault. If you deploy Vault in the same Kubernetes cluster, issue the command kubectl get svc to view the vault service name. Assuming that the service name is vault, and the namespace is default, then the FQDN will be vault.default.svc.cluster.local (refer to the Kubernetes DNS records documentation).

Create the app deployment YAML file with a set of substitutions.

$ cat deployment-template.yaml \
    | sed -e s/"my-deployment"/"$APP_NAME-deployment"/ \
          -e s/"my-namespace"/$APP_NAMESPACE/ \
          -e s/"my-app-name"/"$APP_NAME"/g \
          -e s/"my-app-sa"/$APP_NAME/ \
          -e s/"login-path"/"auth\/$CLUSTER_NAME\/login"/ \
          -e s/"secret-path"/"secret\/data\/$APP_NAMESPACE\/$APP_NAME"/ \
          -e s/"my-role"/"$APP_NAMESPACE-$APP_NAME"/ \
    > deployment-$APP_NAMESPACE-$APP_NAME.yaml

Display the deployment file to ensure all the fields are correct.

$ cat deployment-$APP_NAMESPACE-$APP_NAME.yaml

Example output:

apiVersion: apps/v1
kind: Deployment
metadata:
name: app100-deployment
namespace: retail
spec:
replicas: 1
selector:
   matchLabels:
      app: app100
template:
   metadata:
      labels:
      app: app100
   spec:
      serviceAccountName: app100
      containers:
      - name: app
         image: "kawsark/vault-example-init:0.0.9"
         imagePullPolicy: IfNotPresent
         env:
            - name: VAULT_ADDR
            value: "http://vault.default.svc.cluster.local:8200"
            - name: VAULT_ROLE
            value: retail-app100
            - name: SECRET_PATH
            value: secret/data/retail/app100
            - name: VAULT_LOGIN_PATH
            value: auth/minikube/login

Create the deployment.

$ kubectl create -f deployment-$APP_NAMESPACE-$APP_NAME.yaml

Verify that the pod is running.

$ kubectl get pods -n $APP_NAMESPACE

NAME                                READY   STATUS    RESTARTS   AGE
app100-deployment-74dcb4c76-gqxhz   1/1     Running   0          39s

Wait until the pod, prefixed with the $APP_NAME, is running and ready (1/1).

Display logs

Create a variable to store the unique name of the pod.

$ export POD_NAME=$(kubectl get pods -l app=$APP_NAME \
    -o jsonpath='{.items[0].metadata.name}' --namespace=$APP_NAMESPACE)

Display the logs of the pod.

$ kubectl logs $POD_NAME --namespace $APP_NAMESPACE

023/10/12 18:53:39 ==> WARNING: Don't ever write secrets to logs.
2023/10/12 18:53:39 ==>          This is for demonstration only.
2023/10/12 18:53:39 hvs.CAESIGqKUpWZFQPuRW76Fb6prcXQ_fmN-SC7VPPc2Hz1UuGHGh4KHGh2cy5BQXl3RmI3RlZtTldzWlFXU25UbGVLbFQ
2023/10/12 18:53:39 secret secret/data/retail/app100 -> &{2c89566f-1d2e-6348-5bbd-17b1f2597d83  0 false map[data:map[app:app100 password:test username:demo] metadata:map[created_time:2023-10-12T18:48:02.955848346Z custom_metadata:<nil> deletion_time: destroyed:false version:2]] [] <nil> <nil>}
2023/10/12 18:53:39 Starting renewal loop
2023/10/12 18:53:39 Successfully renewed: &api.RenewOutput{RenewedAt:time.Time{wall:0x11372678, ext:63832733619, loc:(*time.Location)(nil)}, Secret:(*api.Secret)(0xc000058f00)}
2023/10/12 18:53:39 secret secret/data/retail/app100 -> &{6a6ad400-bf10-cd88-9705-55bb41481ef6  0 false map[data:map[app:app100 password:test username:demo] metadata:map[created_time:2023-10-12T18:48:02.955848346Z custom_metadata:<nil> deletion_time: destroyed:false version:2]] [] <nil> <nil>}

The log displays the secret path and secret that it request from Vault.

Warning

Do not write secrets to logs in production. This is for demonstration only. This application will read its service account JWT and use it to authenticate with Vault. It will then log the secret and keep the token renewed. Below is a snippet from the application log showing a successful secret read from the secret access path.

The Vault audit log should show that Vault applied the minikube-kv-read ACL policy upon login and dynamically allowed access to the secret path: secrets/data/retail/app100.

$ kubectl exec -it vault-0 -- cat /tmp/audit_log.json | jq .
{
  ...snip...
  
  "auth": {
    "client_token": "hmac-sha256:97aaf5c989f...",
    
    ...snip...
    
    "token_policies": [
      "default", "minikube-kv-read" ],
    "metadata": {
      "role": "retail-app100",
      "service_account_name": "app100",
      "service_account_namespace": "retail",
  
      ...snip...
  
    "operation": "read", "mount_type": "kv",
  
    ...snip...
  
    "path": "secrets/data/retail/app100",
  },
  "response": {
    "mount_type": "kv",

    ...snip...
    
    "data": {
      "app": "hmac-sha256:7af4ccfd...",
      "password": "hmac-sha256:da5ff1936...",
      "username": "hmac-sha256:ff948a5ec..." }

Onboard additional application

You will use this method to onboard another application called app200 in the same namespace. Since the secret path scheme will remain the same, secret/data/<namespace>/<app-name>, you will not need to create another policy.

Create Role and write secret

Issue the commands below to create an application role, associate the policy and store a secret.

Update the APP_NAME environment variable value.
```
$ export APP_NAME=app200
```

Create an application Service Account.

$ kubectl create sa $APP_NAME -n $APP_NAMESPACE

Create role and associate kv-read policy.

$ kubectl exec --namespace default -it vault-0 --  \
    vault write auth/$CLUSTER_NAME/role/$APP_NAMESPACE-$APP_NAME \
       bound_service_account_names=$APP_NAME \
       bound_service_account_namespaces=$APP_NAMESPACE \
       policies=$CLUSTER_NAME-kv-read \
       period=120s

Write a secret for the secret/data/<APP_NAMESPACE>/<APP_NAME> path.

$ kubectl exec --namespace default -it vault-0 -- \
    vault kv put secret/$APP_NAMESPACE/$APP_NAME \
      app=$APP_NAME \
      username=demo \
      password=test

Deploy application

Create the Deployment manifest file, which sets the SECRET_PATH with the value secret/data/retail/app200.

Create the app deployment YAML file with a set of substitutions.

$ cat deployment-template.yaml \
    | sed -e s/"my-deployment"/"$APP_NAME-deployment"/ \
          -e s/"my-namespace"/$APP_NAMESPACE/ \
          -e s/"my-app-name"/"$APP_NAME"/g \
          -e s/"my-app-sa"/$APP_NAME/ \
          -e s/"login-path"/"auth\/$CLUSTER_NAME\/login"/ \
          -e s/"secret-path"/"secret\/data\/$APP_NAMESPACE\/$APP_NAME"/ \
          -e s/"my-role"/"$APP_NAMESPACE-$APP_NAME"/ \
    > deployment-$APP_NAMESPACE-$APP_NAME.yaml

Create the application and display pods.

$ kubectl create -f deployment-$APP_NAMESPACE-$APP_NAME.yaml
deployment.apps/app200-deployment created

Verify that the pod is running.

$ kubectl get pods -n $APP_NAMESPACE

NAME                                 READY   STATUS    RESTARTS   AGE
app100-deployment-74dcb4c76-gqxhz    1/1     Running   0          12m
app200-deployment-7c876b7597-p75ns   1/1     Running   0          14s

Display logs

Display the application logs and check for a successful login and secrets retrieval from secret/data/retail/app200.

$ export POD_NAME=$(kubectl get pods -l app=$APP_NAME \
    -o jsonpath='{.items[0].metadata.name}' --namespace=$APP_NAMESPACE)

Read application logs.
```
$ kubectl logs $POD_NAME -n $APP_NAMESPACE
```
If you change the SECRET_PATH to secret/data/retail/app100 in the deployment-retail-app200.yaml, then the secrets retrieval will fail for app200. Below is a snippet from the application logs when the app200 was deployed with the SECRET_PATH: "secret/data/retail/app100".
```
URL: GET http://vault.default.svc.cluster.local:8200/v1/secret/data/retail/app200
Code: 403. Errors:

* 1 error occurred:
        * permission denied
```

The above failure confirms that the templated policy is working as intended. This is because the ACL policy only allows access to a path ending with the application's own service account name which is app200.

Check metadata populated by Vault

This is an optional step to view the entity and entity-alias objects that Vault created automatically upon login. It will also allow you to review the entity alias metadata populated by Vault. You can view these items using the UI or the CLI as described in the following steps.

Note

At least one successful login using the Kubernetes authentication method is needed before proceeding with these steps.

If you deployed both app100 and app200, you should see two entity objects. Each entity maps to an authentication alias named as the Kubernetes service account ID (represented as service_account_uid under alias metadata).

You can expose the Vault UI with port-forwarding.

$ kubectl port-forward vault-0 8200:8200
Forwarding from 127.0.0.1:8200 -> 8200
Forwarding from [::1]:8200 -> 8200
##...

Login to Vault UI with address http://127.0.0.1:8200, and enter root as password.
Select Access, and then Entities.
Select an entity and then select the Aliases tab.
Select the alias, and then Metadata tab to see auto-populated metadata keys.

Save the list of all entity alias objects, then display each object as part of a loop. This example shows how to display the first 10 objects.

Save the entity alias list in a file, entity_alias_list.json.

$ kubectl exec --namespace default -it vault-0 -- \
     vault list -format=json identity/entity-alias/id | jq . -r > entity_alias_list.json

View the entity_alias_list.json file to list the retrieved entity IDs.

$ cat entity_alias_list.json

Example output:

[
  "15906b75-6910-147a-960c-9058effb3e63",
  "2a0c107d-31d4-fa7d-b66e-630d4599373f"
]

Retrieve the first entity alias's metadata.

$ kubectl exec --namespace default -it vault-0 -- \
    vault read identity/entity-alias/id/$(cat entity_alias_list.json | jq -r ".[0]")

Example output:

Key                          Value
---                          -----
canonical_id                 28f22e87-d7b3-9b8f-daae-045aaf13374f
creation_time                2021-02-19T08:53:27.6657874Z
id                           15906b75-6910-147a-960c-9058effb3e63
last_update_time             2021-02-19T08:53:27.6657874Z
merged_from_canonical_ids    <nil>
metadata                     map[service_account_name:app200 service_account_namespace:retail service_account_secret_name:app200-token-9fwpg service_account_uid:d5b94793-f64f-416c-a842-d6bbaa420551]
mount_accessor               auth_kubernetes_f9be05d8
mount_path                   auth/minikube/
mount_type                   kubernetes
name                         d5b94793-f64f-416c-a842-d6bbaa420551
namespace_id                 root

The metadata shows a map of key value pairs.

Vault automatically created the entity and entity alias objects, and then added the Kubernetes namespace and Service Account names as the metadata. You used this information to create a templated ACL policy that applied to multiple applications. The uniqueness of the metadata is what allows us to create one ACL policy that can apply to multiple applications.

Include custom metadata

Thus far the secret access path included auto-populated metadata: the Kubernetes service account name and namespace. However, you may want to include custom information as part of the secret access path that is not automatically populated by Vault.

To illustrate this scenario, consider a scenario where the secret access path needs to have a geographical name in it.

Naming convention: secret/data/<namespace>/<geography>/<app-name>
Examples:

In this scenario, each geography has the same application deployed and the secrets stored in Vault need to contain some localized information. The <namespace> and <app-name> elements will be auto-populated by Vault. You need to add the <geography> portion as a custom metadata. To add custom metadata, you need to create the Vault entity and entity-alias objects before logging in.

Create application service account, entity and entity-alias

Create the Kubernetes service account called app300, then create the Vault entity and entity alias objects. You will add two metadata fields: geography and owner. The owner field is optional and used for tracking purposes - it is not used as part of the secret access path.

Update the APP_NAME environment variable value.
```
$ export APP_NAME=app300
```

Create application Service Account.

$ kubectl create sa $APP_NAME -n $APP_NAMESPACE

Create an entity with metadata information.

$ kubectl exec --namespace default -it vault-0 -- \
     vault write identity/entity name=$APP_NAME \
        metadata='geography=useast1' \
        metadata='owner=foo'

Store the generated entity ID.

$ export ENTITY_ID=$(kubectl exec --namespace default -it vault-0 -- \
     vault read identity/entity/name/$APP_NAME -format=json \
     | jq -r '.data.id')

Obtain the service account name UID.

$ export SERVICE_ACCOUNT_UID=$(kubectl get sa $APP_NAME -n retail \
     -o jsonpath="{.metadata.uid}")

Create the alias to associate the entity and alias. You set the MOUNT_ACCESSOR environment variable in write a templated policy.

$ kubectl exec --namespace default -it vault-0 -- \
    vault write identity/entity-alias name=$SERVICE_ACCOUNT_UID \
       canonical_id=$ENTITY_ID \
       mount_accessor=$MOUNT_ACCESSOR

Example output:

Key             Value
---             -----
canonical_id    d31535dd-22db-ea1a-f187-a9a9fae2884b
id              ee47946c-008c-b15d-fd2a-c529b403bfdc

Check the UI to see the new Vault entity by selecting Access > Entities.
Select the app300 entity and choose the Metadata tab to view the custom metadata that was just entered.

View the entity and metadata using the CLI.

$ kubectl exec --namespace default -it vault-0 -- \
    vault read identity/entity/name/app300

Example output:

Key                    Value
---                    -----
aliases                [map[canonical_id:d31535dd-22db-ea1a-f187-a9a9fae2884b creation_time:2021-02-19T10:20:34.4567655Z id:ee47946c-008c-b15d-fd2a-c529b403bfdc last_update_time:2021-02-19T10:20:34.4567655Z merged_from_canonical_ids:<nil> metadata:<nil> mount_accessor:auth_kubernetes_f9be05d8 mount_path:auth/minikube/ mount_type:kubernetes name:65652c6a-5352-4711-b37a-e1b4c64cbc7b]]
creation_time          2021-02-19T10:09:59.1194815Z
direct_group_ids       []
disabled               false
group_ids              []
id                     d31535dd-22db-ea1a-f187-a9a9fae2884b
inherited_group_ids    []
last_update_time       2021-02-19T10:20:34.4567531Z
merged_entity_ids      <nil>
metadata               map[geography:useast1 owner:foo]
name                   app300
namespace_id           root
policies               <nil>

Create policy with custom metadata

Create a new templated policy which includes the custom metadata: geography. The additional templating element {{identity.entity.metadata.geography}} is used to populate the <geography> portion of the secret access path.

Write the templated ACL policy.

$ kubectl exec -i --namespace default vault-0 -- \
  vault policy write $CLUSTER_NAME-geo-kv-read - << EOF
 path "secret/data/{{identity.entity.aliases.$MOUNT_ACCESSOR.metadata.service_account_namespace}}/{{identity.entity.metadata.geography}}/{{identity.entity.aliases.$MOUNT_ACCESSOR.metadata.service_account_name}}" {
       capabilities=["read","list"]
}
EOF

Read the new policy to verify the configuration.

$ kubectl exec -i --namespace default vault-0 -- \
  vault policy read $CLUSTER_NAME-geo-kv-read

The remaining steps to create the application role and deploy the application should be familiar from before.

Create Role and write secret

Update the APP_NAME environment variable value to app300.
```
$ export APP_NAME=app300
```

Create a role and attach the kv-read policy.

$ kubectl exec --namespace default -it vault-0 --  \
    vault write auth/$CLUSTER_NAME/role/$APP_NAMESPACE-$APP_NAME \
       bound_service_account_names=$APP_NAME \
       bound_service_account_namespaces=$APP_NAMESPACE \
       policies=$CLUSTER_NAME-geo-kv-read \
       period=120s

Write a secret for the secrets/data/<namespace>/<geo>/<APP_NAME> path.

$ kubectl exec --namespace default -it vault-0 -- \
    vault kv put secret/$APP_NAMESPACE/useast1/$APP_NAME \
      app=$APP_NAME \
      username=demo \
      password=test

Example output:

========== Secret Path ==========
secret/data/retail/useast1/app300

======= Metadata =======
Key                Value
---                -----
created_time       2024-11-08T03:34:41.721358004Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Deploy application

After executing the substitution commands, the resulting deployment YAML manifest should indicate SECRET_PATH as secret/data/retail/useast1/app300.

Create the app deployment YAML file with a set of substitutions.

$ cat deployment-template.yaml \
    | sed -e s/"my-deployment"/"$APP_NAME-deployment"/ \
          -e s/"my-namespace"/$APP_NAMESPACE/ \
          -e s/"my-app-name"/"$APP_NAME"/g \
          -e s/"my-app-sa"/$APP_NAME/ \
          -e s/"login-path"/"auth\/$CLUSTER_NAME\/login"/ \
          -e s/"secret-path"/"secret\/data\/$APP_NAMESPACE\/useast1\/$APP_NAME"/ \
          -e s/"my-role"/"$APP_NAMESPACE-$APP_NAME"/ \
    > deployment-$APP_NAMESPACE-$APP_NAME.yaml

Create the deployment.

$ kubectl create -f deployment-$APP_NAMESPACE-$APP_NAME.yaml

Verify that the pod is running.

$ kubectl get pods -n $APP_NAMESPACE

NAME                                 READY   STATUS    RESTARTS   AGE
app100-deployment-74dcb4c76-gqxhz    1/1     Running   0          110m
app200-deployment-7c876b7597-p75ns   1/1     Running   0          97m
app300-deployment-7cb9dc57f8-668lb   1/1     Running   0          8s

Display logs

Display the application logs and check for a successful login and secrets retrieval from secret/data/retail/useast1/app300.

$ export POD_NAME=$(kubectl get pods -l app=$APP_NAME \
    -o jsonpath='{.items[0].metadata.name}' --namespace=$APP_NAMESPACE)

Read the application logs.

$ kubectl logs $POD_NAME -n $APP_NAMESPACE

Example output:

==> WARNING: Don't ever write secrets to logs.
==>          This is for demonstration only.
s.JRWjqG50ZzuQdihxpC51kFvl
secret secret/data/retail/useast1/app300 -> &{5a6b79be-719b-913b-5b38-867acacd0456  0 false map[data:map[app:app300 password:test username:demo] metadata:map[created_time:2021-02-19T11:31:12.8160839Z deletion_time: destroyed:false version:1]] [] <nil> <nil>}
Starting renewal loop
Successfully renewed: &api.RenewOutput{RenewedAt:time.Time{wall:0xa1d4520, ext:63749331458, loc:(*time.Location)(nil)}, Secret:(*api.Secret)(0xc000292900)}
secret secret/data/retail/useast1/app300 -> &{1b2e9fab-b266-c2c0-b988-6c1840cf2a2f  0 false map[data:map[app:app300 password:test username:demo] metadata:map[created_time:2021-02-19T11:31:12.8160839Z deletion_time: destroyed:false version:1]] [] <nil> <nil>}

To deploy the application in another geography, the approximate steps will be as follows:

Create the Kubernetes application Service Account
Create the Entity (with metadata) and Entity-Alias
Create the Vault Role and write a secret
Deploy the application

Summary

In this tutorial, you reviewed how to use a single templated ACL policy for multiple applications. The main advantage of this approach over writing a policy per application is to simplify onboarding steps. Fewer policies also result in easier policy management and auditing. This workflow enforces standard conventions on where applications can read and write secrets from within Vault, allowing for better organization and anomaly detection.

While you have provided a Kubernetes-based example, other authentication methods will also have associated metadata that can be useful in creating templated policies. Similarly, while you accessed the static key/value secrets engine, the secret access path also works with dynamic secrets engines where the effective path corresponds to an application role.

Help and reference

Auth method documentation
Kubernetes auth method documentation
Identity documentation
ACL policies tutorial
Simplify ACL management tutorial

Manage Kubernetes service tokens

Next Collection

Integrate with HCP Vault

This tutorial also appears in:

9 tutorials

Policies
Policies provide a declarative way to grant or forbid access to certain paths and operations in Vault. Learn how to write policies to meet your organization's needs.
- Vault