HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream

Static secrets management

Note

This page covers the operator setup and management of static secrets. For how users consume static secrets, see the User Guide: Static secrets.

Overview

With the initial cluster configuration completed and auth methods enabled, you are now ready to enable your first use case. While Vault has many capabilities around cryptography and secrets management, the initial use-case is often the manual creation of static secrets and secure storage and retrieval.

Vault's KV secrets engine stores arbitrary static secrets. Secrets can be anything from passwords to database connection strings to API keys. Vault stores these securely, and the nature of the static secret is unimportant. What is important to consider is the workflow around the lifecycle of the secrets, namely who or what has permission to create, read, update, and delete any particular secret.

Vault handles this through the use of authentication and policy.

  • Authentication(opens in new tab): Any interaction with Vault must be first authenticated. Authentication verifies the identity of a Vault client (e.g. user, machine, or application) to interact with Vault. It does not define any permissions on what that entity can look at inside Vault.
  • Policy(opens in new tab) - Policies are associated with Vault tokens and define both what secrets the authenticated client can interact with and in what ways. For example: - userA(opens in new tab) can read secret1 and not read secret2. - userB(opens in new tab) can update secret2 but not update secret1 - Any user in groupC can read secret3

This section discusses the following.

  • Static secrets and what you should consider when laying out your secrets schema.
  • Policies and what you should consider when creating them.
  • Secret consumption and what you should consider when planning this strategy.

Static secrets management

At this point, you should have one human auth method (OIDC or LDAP) and one machine auth method (AWS or AppRole) enabled, but no secrets in Vault and no policies set up to enforce access permissions of an authenticated entity.

Before you populate Vault with any static secrets, it is important to understand how Vault stores static secrets and plan how you will organize them.

Key/Value (KV) secrets engine

The KV secrets engine is a generic key-value store used to store arbitrary secrets within the configured physical storage for Vault. Secrets written to Vault are encrypted and then written to backend storage. Therefore, the backend storage mechanism never sees the unencrypted value and doesn't have the means necessary to decrypt it without Vault.

The KV secrets engine comes in two versions: version 1(opens in new tab) and version 2(opens in new tab). Version 2 provides secret versioning and utilizes a different API compared to version 1. KV v1 is more performant than v2 since there are fewer storage calls due to the lack of additional metadata or history being stored. Migration from v1 to v2 is possible but requires that the secret engine be taken offline during the migration. The process could potentially take a long time depending on the amount of data stored. In general, the additional features provided by KV v2 offset the additional write and storage overhead. Therefore, KV v2 is recommended by default unless you have specific performance requirements. Working with KV v1 is out of scope for this document.

Working with the KV secrets engine is straightforward. You can interact with the secrets engine through the UI, CLI, or API. Vault static secrets are laid out like a virtual filesystem. The path where you enable the KV secrets engine acts as the root of the file system. Everything after the root is a key-value pair to write to the secrets engine. For example, the commands below first enables the KV v2 secrets engine at the path kvv2/, then writes a new key-value secret to the path myapp, with a key of foo and value of bar. Finally, it reads the secret using the vault kv get command.

$ vault secrets enable -version=2 -path="kvv2" \
-description="Demo K/V v2" kv
 
Success! Enabled the kv secrets engine at: kvv2/
 
$ vault kv put -mount=kvv2 myapp foo=bar
 
= Secret Path =
kvv2/data/myapp
 
======= Metadata =======
Key                Value
---                -----
created_time       2023-09-18T20:32:54.482700528Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
 
$ vault kv get -mount=kvv2 myapp
 
= Secret Path =
kvv2/data/myapp
 
======= Metadata =======
Key                Value
---                -----
created_time       2023-09-18T20:32:54.482700528Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
 
=== Data ===
Key    Value
---    -----
foo    bar

Mount and path structure

Before writing any data to the KV secrets engine, it is important to consider how to organize the data under the secrets engine mount. The KV structure is important because it helps to simplify and reduce the number of Vault policies by grouping similar data under the same path.

Antipatterns for KV path structure

Some common antipatterns when designing a KV path structure are detailed below.

Creating paths based on environments

For example secret/my-app/dev/*, secret/my-app/uat/*, secret/my-app/prod/*. We do not recommend creating paths based on environments. Customers wanting to have Vault in different environments should run a separate Vault instance in each environment. For instance, if you have a dev, prod, and preprod environment, you should have a Vault instance in dev, prod, and preprod as well. If this is not feasible due to cost and overhead, you should at a minimum separate prod and non-prod. This allows Vault admins to:

  1. Have a process for letting devs add secrets in lower environments
  2. Maintain the same path structure across all environments, which minimizes need for application code changes.

Note

You should consider the increased load on your production cluster from non-prod clients, which might require you to provision hardware with better compute and storage performance.

Creating paths based on teams

We also do not recommend creating paths based on teams. This antipattern tightly couples application code to organizational topology, which may prove confusing in the event of team restructuring. We recommend instead to design paths based on the application concern. Generically, structure your paths as such: <app-name>/<service>/<component>/secret. For example, in a webstore app, you might have a billing service and an identity service, and within them you have secrets for specific components:

webapp/billing/checkout/<secret1> webapp/identity/password-reset/<secret1>

Recommended KV mount and path structure

We recommend that you mount a single KV v2 secrets engine with sub-paths per application concern. For example, in a webstore app with a 3-tier architecture (presentation, application, data), the mount structure would look like:

Mount structure

This mount structure provides several benefits:

  • It reduces the potential of hitting mount table limits.
  • It reduces operational complexity by centralizing all static secrets under a single mount

Note

There is a limit to the number of secret engines you can mount. For integrated storage, the maximum number of secret engine mount points is approximately 14000. See Vault limits and maximums for more information.

Configure the KV secrets engine

In this section, you will enable your first KV secrets engine and test out the API. Similar to all other auth methods and secrets engines, you must enable the KV secrets engine before you can use it. Login to your Vault cluster as an admin and use the command below to enable the KV v2 secrets engine using the CLI on a custom path under the tenant namespace.

$ export VAULT_NAMESPACE=<tenant namespace>
$ vault secrets enable -version=2 -path=secret kv
 
Success! Enabled the kv secrets engine at: secret/

Working with KV v2

Here we will briefly explore the functionalities of the KV v2 secrets engine. For a more thorough understanding, please review the tutorials and documentation below.

Writing data

  $ vault kv put -mount=secret my-secret foo=a bar=b
 
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:57:04.098790466Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1

Reading data

  $ vault kv get -mount=secret my-secret
 
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:57:04.098790466Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1
 
  === Data ===
  Key    Value
  ---    -----
  bar    b
  foo    a

Write another version of secret

  $ vault kv put -mount=secret my-secret foo=aa bar=bb
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:58:31.636294687Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            2

Reading the same secret now will return the latest version

  $ vault kv get -mount=secret my-secret
 
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:58:31.636294687Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            2
 
  === Data ===
  Key    Value
  ---    -----
  bar    bb
  foo    aa

Read a previous version by specifying the -version flag

  $ vault kv get -mount=secret -version=1 my-secret
 
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:57:04.098790466Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1
 
  === Data ===
  Key    Value
  ---    -----
  bar    b
  foo    a

Delete the latest version of a secret

  $ vault kv delete -mount=secret my-secret
 
  Success! Data deleted (if it existed) at: secret/data/my-secret

Undelete a specific version of a secret

  $ vault kv undelete -mount=secret -versions=2 my-secret
 
  Success! Data written to: secret/undelete/my-secret
 
  $ vault kv get -mount=secret my-secret
 
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:58:31.636294687Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            2
 
  === Data ===
  Key    Value
  ---    -----
  bar    bb
  foo    aa

Permanently delete a specific version of a secret

  $ vault kv destroy -mount=secret -versions=2 my-secret
 
  Success! Data written to: secret/destroy/my-secret

Vault policies

At this point, you should have one human auth method (OIDC or LDAP) and one machine auth method (AWS or AppRole) enabled. You have also enabled the KV v2 secrets engine with some secrets populated. The next step is to configure policies to manage access to these secrets for authenticated clients.

Policies provide a declarative way to grant or forbid access to certain paths and operations in Vault. When an identity authenticates to Vault it receives a token, and all policies associated with that identity are attached to that token.

Vault static secrets are organized and accessed using paths, and it is the role of ACL policies to describe the permissions on these paths. For example:

  1. A secret object is created at secrets/billing-service.
  2. A policy named billing-service-read is created where read capabilities are defined against the above path.
  3. The identity group billing-service-dev is mapped to the billing-service-read policy in the OIDC auth method.
  4. As a member of the billing-service-dev group, Bob authenticates with Vault using OIDC, and the token Bob receives has the billing-service-read policy attached.
  5. Bob is allowed to read the secret at secrets/billing-service.

A thorough exploration of Vault policies can be found in the following documents.

There are some best practices that you should keep in mind when writing Vault policies.

Principle of least privilege

Err on the side of caution if a consumer is uncertain or lacks clarity regarding their access and policy requirements. If a client realizes the need for access to a secret at a later point, it can be identified and addressed accordingly. In contrast, granting access to an unnecessary secret can be challenging to detect and carries potential risks.

Role-based policies

It is often helpful to create several core policies based on functional roles when onboarding users to Vault, such as the following.

  • Vault cluster administrator: full access to Vault.
  • Vault operator: responsible for general Vault operations such as configuring auth methods, secrets engines, and policies.
  • Security team: consume audit logs, review and approve policies changes.
  • Developer: read-only access to specific paths required by their applications.
  • Application owners: write access to specific paths for their applications.

In addition to role-based policies for Vault users, organizations often have numerous policies tailored for application integrations. Each application may have distinct requirements that warrants the creation of individual policies. Moreover, multiple policies may be crafted for various stages in the software development lifecycle (e.g., development, quality assurance, staging, production) for each application. Furthermore, automation workflows, such as CI/CD pipelines or other application build tools, may also require specific policies.

Policy templates

Being as restrictive as possible in your policies can lead to a large number of policies to manage. To simplify this process, we recommend the use of ACL Policy Path Templating(opens in new tab).

Policy templating allows for a much smaller set of policies to be used, but still provide the fine grained access controls needed.

Version control

We recommend that you put your policies in a version control system such as GitHub and have a peer-review process for both security and audit purposes. Change management and standardization becomes much easier when policies are centrally managed in a code repository. Vault operators or application teams can submit changes using pull requests and Vault administrators or security teams can approve or deny policy changes.

We recommend using Terraform to codify the configuration of Vault, including policy management. Refer to the Codify Management of Vault Enterprise(opens in new tab) tutorial which demonstrates policy deployment in multiple namespaces.

Configure policies for KV Secrets

In this section, you will create policies for a hypothetical application to allow the application

  • Owners to create and update static secrets.
  • Developers to read static secrets for their application.
  • To read its required static secrets.

Before you begin, you should have

  • Configured either the OIDC or LDAP auth method for human access.
  • Configured either the AWS or AppRole auth method for machine access.
  • Enabled the KV v2 secrets engine.

Step 1: Write a secret

Login with an admin user and write a secret for the hypothetical application (billing-service).

$ vault kv put -mount=secret billing-service foo=bar
 
== Secret Path ==
secret/data/billing-service
 
======= Metadata =======
Key                Value
---                -----
created_time       2023-09-21T18:27:29.267845823Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Step 2: Create policy for application owners

$ vault policy write billing-service-owner -<<EOF
path "secret/data/billing-service" {
  capabilities = ["create", "read", "update", "delete"]
}
EOF
Success! Uploaded policy: billing-service-owner

Step 3: Create policy for the application and its developers

$ vault policy write billing-service-dev -<<EOF
 
path "secret/data/billing-service" {
  capabilities = ["read"]
}
EOF
 
Success! Uploaded policy: billing-service-dev

Step 4: Associate policy for human auth method

For OIDC auth:

Similar to the configuration for Vault admins in setting up the OIDC auth method, you can use the commands below to create an external group called billing-service-owner for the application owners and associate it to the billing-service-owner policy. This external group is tied to an OIDC group called billing-service-owner by the group alias.

$ GROUP_ID=$(vault write -format=json identity/group \
  name="billing-service-owner" \
  type="external" \
  policies="billing-service-owner" | jq -r ".data.id")
 
$ MOUNT_ACCESSOR=$(vault read -field=accessor sys/mounts/auth/oidc)
 
$ vault write identity/group-alias \
  name="billing-service-owner" \
  mount_accessor=$MOUNT_ACCESSOR \
  canonical_id=$GROUP_ID

Create another external group called billing-service-dev for the application developers.

$ GROUP_ID=$(vault write -format=json identity/group \
  name="billing-service-dev" \
  type="external" \
  policies="billing-service-dev" | jq -r ".data.id")
 
$ MOUNT_ACCESSOR=$(vault read -field=accessor sys/mounts/auth/oidc)
 
$ vault write identity/group-alias \
  name="billing-service-dev" \
  mount_accessor=$MOUNT_ACCESSOR \
  canonical_id=$GROUP_ID

For LDAP auth:

Map the LDAP group called billing-service-owner for the application owners to the billing-service-owner policy.

$ vault write auth/ldap/groups/billing-service-owner policies=billing-service-owner

Map the LDAP group called billing-service-dev for the application developers to the billing-service-dev policy.

$ vault write auth/ldap/groups/billing-service-dev policies=billing-service-dev

Step 5: Associate policy for machine auth method

For AWS auth:

Similar to the configuration when you set up a demo role for the AWS auth method, use the command below to create a new role called billing-service-dev for the application and associate it to the billing-service-dev policy.

$ vault write auth/aws/role/billing-service-dev auth_type=iam \
bound_iam_principal_arn=arn:aws:iam::123456789012:role/aws-ec2role-for-vault-autthmethod \
policies=billing-service-dev token_ttl=8 max_ttl=8
 
Success! Data written to: auth/aws/role/billing-service-dev

For AppRole auth:

Create a role called billing-service-dev for the application and map it to the billing-service-dev policy.

$ vault write auth/approle/role/billing-service-dev \
secret_id_num_uses=1 \
secret_id_ttl=15m \
token_policies="billing-service-dev" \
token_ttl=1h \
token_max_ttl=1h

Note

The application is responsible for renewing the Vault token once it authenticates using the Secret ID. Recall that limiting the SecretID TTL and its number of uses means that a new SecretID will need to be delivered to the application once the application's Vault token reaches its maximum TTL and can no longer be renewed.

Step 6: Validate access

For OIDC auth:

Login with a user belonging to the billing-service-owner group. You should see that the token is attached to the billing-service-owner identity policy.

  $ export VAULT_NAMESPACE=<tenant namespace>
  $ vault login -method=oidc
 
  Waiting for OIDC authentication to complete...
  Success! You are now authenticated. The token information displayed below
  is already stored in the token helper. You do NOT need to run "vault login"
  again. Future Vault requests will automatically use this token.
 
  Key                    Value
  ---                    -----
  token                  hvs.token
  token_accessor         84HTrye0clS28Elc13ZDgiXo.mFAaM
  token_duration         1h
  token_renewable        true
  token_policies         ["default"]
  identity_policies      ["billing-service-owner"]
  policies               ["default" "billing-service-owner"]
  token_meta_email       alice@domain.com
  token_meta_role        default
  token_meta_username    alice

Create a new secret.

  vault kv put -mount=secret billing-service pizza=pepperoni
 
  == Secret Path ==
  secret/data/billing-service
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-21T20:18:44.084663057Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            2

Perform step 1 and 2 to validate access for application developers. You should only be able to read the secret under the path secret/billing-service.

For LDAP auth

Login with a user belonging to the billing-service-owner group. You should see that the token is attached to the billing-service-owner identity policy.

  vault login -method=ldap username=nwong
 
  Password (will be hidden):
  Success! You are now authenticated. The token information displayed below
  is already stored in the token helper. You do NOT need to run "vault login"
  again. Future Vault requests will automatically use this token.
 
  Key                    Value
  ---                    -----
  token                  hvs.token
  token_accessor         EyhczVcVIT8BUuGU1r5GIJ16.JeR4H
  token_duration         1h
  token_renewable        true
  token_policies         ["default" "billing-service-owner"]
  identity_policies      []
  policies               ["default" "billing-sevice-owner"]
  token_meta_username    nwong

Create a new secret.

  vault kv put -mount=secret billing-service pizza=pepperoni
 
  == Secret Path ==
  secret/data/billing-service
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-21T20:18:44.084663057Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            2

Perform step 1 and 2 to validate access for application developers. You should only be able to read the secret under the path secret/billing-service.

For AWS auth

Login from an EC2 instance attached to an instance profile that is bound to the IAM principal you configured for the billing-service-dev role. You should see that the token is attached to the billing-service-dev policy.

  $ vault login -method=aws role=billing-service-dev
 
  Success! You are now authenticated. The token information displayed below
  is already stored in the token helper. You do NOT need to run "vault login"
  again. Future Vault requests will automatically use this token.
 
  Key                          Value
  ---                          -----
  token                        hvs.token
  token_accessor               cGAOT1f11td0tDCgR4Ff1Xw8.Ojft7
  token_duration               8s
  token_renewable              true
  token_policies               ["default" "billing-service-dev"]
  identity_policies            []
  policies                     ["default" "billing-service-dev"]
  token_meta_client_arn        arn:aws:sts::123456789012:assumed-role/aws-ec2role-for-vault-authmethod/i-02ddcf5aed1121703
  token_meta_client_user_id    AROATYM2SX6XH5UEKGF7G
  token_meta_role_id           53f43687-2f55-c5e9-3570-c51b1cb69763
  token_meta_account_id        123456789012
  token_meta_auth_type         iam
  token_meta_canonical_arn     arn:aws:iam::123456789012:role/aws-ec2role-for-vault-authmethod

Verify that you can read the secret under the secret/billing-service.

    $ vault kv get -mount=secret billing-service
 
    == Secret Path ==
    secret/data/billing-service
 
    ======= Metadata =======
    Key                Value
    ---                -----
    created_time       2023-09-21T21:38:43.532185535Z
    custom_metadata    <nil>
    deletion_time      n/a
    destroyed          false
    version            1
 
    === Data ===
    Key    Value
    ---    -----
    foo    bar

For AppRole auth:

Read the RoleID using the command below.

    $ ROLE_ID=$(vault read -field=role_id auth/approle/role/billing-service-dev/role-id)

Next, generate a SecretID for the role.

  $ SECRET_ID=$(vault write -force -field=secret_id auth/approle/role/billing-service-dev/secret-id)

Login using the billing-service-dev role.

  $ vault write auth/approle/login role_id=$ROLE_ID secret_id=$SECRET_ID
 
  Key                     Value
  ---                     -----
  token                   hvs.token
  token_accessor          JhFFIcLlqdjmM6Rq15v0DkGW
  token_duration          768h
  token_renewable         true
  token_policies          ["default" "billing-service-dev"]
  identity_policies       []
  policies                ["default" "billing-service-dev"]
  token_meta_role_name    billing-service-dev

Verify that you can read the secret under the secret/billing-service.

  $ vault kv get -mount=secret billing-service
 
  == Secret Path ==
  secret/data/billing-service
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-21T21:38:43.532185535Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1
 
  === Data ===
  Key    Value
  ---    -----
  foo    bar
Edit this page on GitHub