Static secrets

Overview

With the initial cluster configuration completed and auth methods enabled, you are now ready to enable your first use case. While Vault has many capabilities around cryptography and secrets management, the initial use-case is often the manual creation of static secrets and secure storage and retrieval.

Vault's KV secrets engine stores arbitrary static secrets. Secrets can be anything from passwords to database connection strings to API keys. Vault stores these securely, and the nature of the static secret is unimportant. What is important to consider is the workflow around the lifecycle of the secrets, namely who or what has permission to create, read, update, and delete any particular secret.

Vault handles this through the use of authentication and policy.

Authentication(opens in new tab): Any interaction with Vault must be first authenticated. Authentication verifies the identity of a Vault client (e.g. user, machine, or application) to interact with Vault. It does not define any permissions on what that entity can look at inside Vault.
Policy(opens in new tab) - Policies are associated with Vault tokens and define both what secrets the authenticated client can interact with and in what ways. For example: - userA(opens in new tab) can read secret1 and not read secret2. - userB(opens in new tab) can update secret2 but not update secret1 - Any user in groupC can read secret3

This section discusses the following.

Static secrets and what you should consider when laying out your secrets schema.
Policies and what you should consider when creating them.
Secret consumption and what you should consider when planning this strategy.

Static secrets management

At this point, you should have one human auth method (OIDC or LDAP) and one machine auth method (AWS or AppRole) enabled, but no secrets in Vault and no policies set up to enforce access permissions of an authenticated entity.

Before you populate Vault with any static secrets, it is important to understand how Vault stores static secrets and plan how you will organize them.

Key/Value (KV) secrets engine

The KV secrets engine is a generic key-value store used to store arbitrary secrets within the configured physical storage for Vault. Secrets written to Vault are encrypted and then written to backend storage. Therefore, the backend storage mechanism never sees the unencrypted value and doesn't have the means necessary to decrypt it without Vault.

The KV secrets engine comes in two versions: version 1(opens in new tab) and version 2(opens in new tab). Version 2 provides secret versioning and utilizes a different API compared to version 1. KV v1 is more performant than v2 since there are fewer storage calls due to the lack of additional metadata or history being stored. Migration from v1 to v2 is possible but requires that the secret engine be taken offline during the migration. The process could potentially take a long time depending on the amount of data stored. In general, the additional features provided by KV v2 offset the additional write and storage overhead. Therefore, KV v2 is recommended by default unless you have specific performance requirements. Working with KV v1 is out of scope for this document.

Working with the KV secrets engine is straightforward. You can interact with the secrets engine through the UI, CLI, or API. Vault static secrets are laid out like a virtual filesystem. The path where you enable the KV secrets engine acts as the root of the file system. Everything after the root is a key-value pair to write to the secrets engine. For example, the commands below first enables the KV v2 secrets engine at the path kvv2/, then writes a new key-value secret to the path myapp, with a key of foo and value of bar. Finally, it reads the secret using the vault kv get command.

$ vault secrets enable -version=2 -path="kvv2" \
-description="Demo K/V v2" kv
 
Success! Enabled the kv secrets engine at: kvv2/
 
$ vault kv put -mount=kvv2 myapp foo=bar
 
= Secret Path =
kvv2/data/myapp
 
======= Metadata =======
Key                Value
---                -----
created_time       2023-09-18T20:32:54.482700528Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
 
$ vault kv get -mount=kvv2 myapp
 
= Secret Path =
kvv2/data/myapp
 
======= Metadata =======
Key                Value
---                -----
created_time       2023-09-18T20:32:54.482700528Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1
 
=== Data ===
Key    Value
---    -----
foo    bar

Mount and path structure

Before writing any data to the KV secrets engine, it is important to consider how to organize the data under the secrets engine mount. The KV structure is important because it helps to simplify and reduce the number of Vault policies by grouping similar data under the same path.

Antipatterns for KV path structure

Some common antipatterns when designing a KV path structure are detailed below.

Creating paths based on environments

For example secret/my-app/dev/*, secret/my-app/uat/*, secret/my-app/prod/*. We do not recommend creating paths based on environments. Customers wanting to have Vault in different environments should run a separate Vault instance in each environment. For instance, if you have a dev, prod, and preprod environment, you should have a Vault instance in dev, prod, and preprod as well. If this is not feasible due to cost and overhead, you should at a minimum separate prod and non-prod. This allows Vault admins to:

Have a process for letting devs add secrets in lower environments
Maintain the same path structure across all environments, which minimizes need for application code changes.

Note

You should consider the increased load on your production cluster from non-prod clients, which might require you to provision hardware with better compute and storage performance.

Creating paths based on teams

We also do not recommend creating paths based on teams. This antipattern tightly couples application code to organizational topology, which may prove confusing in the event of team restructuring. We recommend instead to design paths based on the application concern. Generically, structure your paths as such: <app-name>/<service>/<component>/secret. For example, in a webstore app, you might have a billing service and an identity service, and within them you have secrets for specific components:

webapp/billing/checkout/<secret1> webapp/identity/password-reset/<secret1>

Recommended KV mount and path structure

We recommend that you mount a single KV v2 secrets engine with sub-paths per application concern. For example, in a webstore app with a 3-tier architecture (presentation, application, data), the mount structure would look like:

Mount structure

This mount structure provides several benefits:

It reduces the potential of hitting mount table limits.
It reduces operational complexity by centralizing all static secrets under a single mount

Note

There is a limit to the number of secret engines you can mount. For integrated storage, the maximum number of secret engine mount points is approximately 14000. See Vault limits and maximums for more information.

Configure the KV secrets engine

In this section, you will enable your first KV secrets engine and test out the API. Similar to all other auth methods and secrets engines, you must enable the KV secrets engine before you can use it. Login to your Vault cluster as an admin and use the command below to enable the KV v2 secrets engine using the CLI on a custom path under the tenant namespace.

$ export VAULT_NAMESPACE=<tenant namespace>
$ vault secrets enable -version=2 -path=secret kv
 
Success! Enabled the kv secrets engine at: secret/

Working with KV v2

Here we will briefly explore the functionalities of the KV v2 secrets engine. For a more thorough understanding, please review the tutorials and documentation below.

KV secrets engine - version 2(opens in new tab)
Versioned Key/value secrets engine(opens in new tab)

Writing data

  $ vault kv put -mount=secret my-secret foo=a bar=b
 
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:57:04.098790466Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1

Reading data

  $ vault kv get -mount=secret my-secret
 
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:57:04.098790466Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1
 
  === Data ===
  Key    Value
  ---    -----
  bar    b
  foo    a

Write another version of secret

  $ vault kv put -mount=secret my-secret foo=aa bar=bb
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:58:31.636294687Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            2

Reading the same secret now will return the latest version

  $ vault kv get -mount=secret my-secret
 
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:58:31.636294687Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            2
 
  === Data ===
  Key    Value
  ---    -----
  bar    bb
  foo    aa

Read a previous version by specifying the -version flag

  $ vault kv get -mount=secret -version=1 my-secret
 
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:57:04.098790466Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1
 
  === Data ===
  Key    Value
  ---    -----
  bar    b
  foo    a

Delete the latest version of a secret

  $ vault kv delete -mount=secret my-secret
 
  Success! Data deleted (if it existed) at: secret/data/my-secret

Undelete a specific version of a secret

  $ vault kv undelete -mount=secret -versions=2 my-secret
 
  Success! Data written to: secret/undelete/my-secret
 
  $ vault kv get -mount=secret my-secret
 
  ==== Secret Path ====
  secret/data/my-secret
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-20T18:58:31.636294687Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            2
 
  === Data ===
  Key    Value
  ---    -----
  bar    bb
  foo    aa

Permanently delete a specific version of a secret

  $ vault kv destroy -mount=secret -versions=2 my-secret
 
  Success! Data written to: secret/destroy/my-secret

Vault policies

At this point, you should have one human auth method (OIDC or LDAP) and one machine auth method (AWS or AppRole) enabled. You have also enabled the KV v2 secrets engine with some secrets populated. The next step is to configure policies to manage access to these secrets for authenticated clients.

Policies provide a declarative way to grant or forbid access to certain paths and operations in Vault. When an identity authenticates to Vault it receives a token, and all policies associated with that identity are attached to that token.

Vault static secrets are organized and accessed using paths, and it is the role of ACL policies to describe the permissions on these paths. For example:

A secret object is created at secrets/billing-service.
A policy named billing-service-read is created where read capabilities are defined against the above path.
The identity group billing-service-dev is mapped to the billing-service-read policy in the OIDC auth method.
As a member of the billing-service-dev group, Bob authenticates with Vault using OIDC, and the token Bob receives has the billing-service-read policy attached.
Bob is allowed to read the secret at secrets/billing-service.

A thorough exploration of Vault policies can be found in the following documents.

Policies - concepts(opens in new tab)
Getting started guide to policy writing(opens in new tab)

There are some best practices that you should keep in mind when writing Vault policies.

Principle of least privilege

Err on the side of caution if a consumer is uncertain or lacks clarity regarding their access and policy requirements. If a client realizes the need for access to a secret at a later point, it can be identified and addressed accordingly. In contrast, granting access to an unnecessary secret can be challenging to detect and carries potential risks.

Role-based policies

It is often helpful to create several core policies based on functional roles when onboarding users to Vault, such as the following.

Vault cluster administrator: full access to Vault.
Vault operator: responsible for general Vault operations such as configuring auth methods, secrets engines, and policies.
Security team: consume audit logs, review and approve policies changes.
Developer: read-only access to specific paths required by their applications.
Application owners: write access to specific paths for their applications.

In addition to role-based policies for Vault users, organizations often have numerous policies tailored for application integrations. Each application may have distinct requirements that warrants the creation of individual policies. Moreover, multiple policies may be crafted for various stages in the software development lifecycle (e.g., development, quality assurance, staging, production) for each application. Furthermore, automation workflows, such as CI/CD pipelines or other application build tools, may also require specific policies.

Policy templates

Being as restrictive as possible in your policies can lead to a large number of policies to manage. To simplify this process, we recommend the use of ACL Policy Path Templating(opens in new tab).

Policy templating allows for a much smaller set of policies to be used, but still provide the fine grained access controls needed.

Version control

We recommend that you put your policies in a version control system such as GitHub and have a peer-review process for both security and audit purposes. Change management and standardization becomes much easier when policies are centrally managed in a code repository. Vault operators or application teams can submit changes using pull requests and Vault administrators or security teams can approve or deny policy changes.

We recommend using Terraform to codify the configuration of Vault, including policy management. Refer to the Codify Management of Vault Enterprise(opens in new tab) tutorial which demonstrates policy deployment in multiple namespaces.

Configure policies for KV Secrets

In this section, you will create policies for a hypothetical application to allow the application

Owners to create and update static secrets.
Developers to read static secrets for their application.
To read its required static secrets.

Before you begin, you should have

Configured either the OIDC or LDAP auth method for human access.
Configured either the AWS or AppRole auth method for machine access.
Enabled the KV v2 secrets engine.

Step 1: Write a secret

$ vault kv put -mount=secret billing-service foo=bar
 
== Secret Path ==
secret/data/billing-service
 
======= Metadata =======
Key                Value
---                -----
created_time       2023-09-21T18:27:29.267845823Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Step 2: Create policy for application owners

$ vault policy write billing-service-owner -<<EOF
path "secret/data/billing-service" {
  capabilities = ["create", "read", "update", "delete"]
}
EOF
Success! Uploaded policy: billing-service-owner

Step 3: Create policy for the application and its developers

$ vault policy write billing-service-dev -<<EOF
 
path "secret/data/billing-service" {
  capabilities = ["read"]
}
EOF
 
Success! Uploaded policy: billing-service-dev

Step 4: Associate policy for human auth method

For OIDC auth:

Similar to the configuration for Vault admins in setting up the OIDC auth method, you can use the commands below to create an external group called billing-service-owner for the application owners and associate it to the billing-service-owner policy. This external group is tied to an OIDC group called billing-service-owner by the group alias.

$ GROUP_ID=$(vault write -format=json identity/group \
  name="billing-service-owner" \
  type="external" \
  policies="billing-service-owner" | jq -r ".data.id")
 
$ MOUNT_ACCESSOR=$(vault read -field=accessor sys/mounts/auth/oidc)
 
$ vault write identity/group-alias \
  name="billing-service-owner" \
  mount_accessor=$MOUNT_ACCESSOR \
  canonical_id=$GROUP_ID

Create another external group called billing-service-dev for the application developers.

$ GROUP_ID=$(vault write -format=json identity/group \
  name="billing-service-dev" \
  type="external" \
  policies="billing-service-dev" | jq -r ".data.id")
 
$ MOUNT_ACCESSOR=$(vault read -field=accessor sys/mounts/auth/oidc)
 
$ vault write identity/group-alias \
  name="billing-service-dev" \
  mount_accessor=$MOUNT_ACCESSOR \
  canonical_id=$GROUP_ID

For LDAP auth:

Map the LDAP group called billing-service-owner for the application owners to the billing-service-owner policy.

$ vault write auth/ldap/groups/billing-service-owner policies=billing-service-owner

Map the LDAP group called billing-service-dev for the application developers to the billing-service-dev policy.

$ vault write auth/ldap/groups/billing-service-dev policies=billing-service-dev

Step 5: Associate policy for machine auth method

For AWS auth:

Similar to the configuration when you set up a demo role for the AWS auth method, use the command below to create a new role called billing-service-dev for the application and associate it to the billing-service-dev policy.

$ vault write auth/aws/role/billing-service-dev auth_type=iam \
bound_iam_principal_arn=arn:aws:iam::123456789012:role/aws-ec2role-for-vault-autthmethod \
policies=billing-service-dev token_ttl=8 max_ttl=8
 
Success! Data written to: auth/aws/role/billing-service-dev

For AppRole auth:

Create a role called billing-service-dev for the application and map it to the billing-service-dev policy.

$ vault write auth/approle/role/billing-service-dev \
secret_id_num_uses=1 \
secret_id_ttl=15m \
token_policies="billing-service-dev" \
token_ttl=1h \
token_max_ttl=1h

Note

The application is responsible for renewing the Vault token once it authenticates using the Secret ID. Recall that limiting the SecretID TTL and its number of uses means that a new SecretID will need to be delivered to the application once the application's Vault token reaches its maximum TTL and can no longer be renewed.

Step 6: Validate access

For OIDC auth:

Login with a user belonging to the billing-service-owner group. You should see that the token is attached to the billing-service-owner identity policy.

  $ export VAULT_NAMESPACE=<tenant namespace>
  $ vault login -method=oidc
 
  Waiting for OIDC authentication to complete...
  Success! You are now authenticated. The token information displayed below
  is already stored in the token helper. You do NOT need to run "vault login"
  again. Future Vault requests will automatically use this token.
 
  Key                    Value
  ---                    -----
  token                  hvs.token
  token_accessor         84HTrye0clS28Elc13ZDgiXo.mFAaM
  token_duration         1h
  token_renewable        true
  token_policies         ["default"]
  identity_policies      ["billing-service-owner"]
  policies               ["default" "billing-service-owner"]
  token_meta_email       alice@domain.com
  token_meta_role        default
  token_meta_username    alice

Create a new secret.

  vault kv put -mount=secret billing-service pizza=pepperoni
 
  == Secret Path ==
  secret/data/billing-service
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-21T20:18:44.084663057Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            2

Perform step 1 and 2 to validate access for application developers. You should only be able to read the secret under the path secret/billing-service.

For LDAP auth

Login with a user belonging to the billing-service-owner group. You should see that the token is attached to the billing-service-owner identity policy.

  vault login -method=ldap username=nwong
 
  Password (will be hidden):
  Success! You are now authenticated. The token information displayed below
  is already stored in the token helper. You do NOT need to run "vault login"
  again. Future Vault requests will automatically use this token.
 
  Key                    Value
  ---                    -----
  token                  hvs.token
  token_accessor         EyhczVcVIT8BUuGU1r5GIJ16.JeR4H
  token_duration         1h
  token_renewable        true
  token_policies         ["default" "billing-service-owner"]
  identity_policies      []
  policies               ["default" "billing-sevice-owner"]
  token_meta_username    nwong

Create a new secret.

  vault kv put -mount=secret billing-service pizza=pepperoni
 
  == Secret Path ==
  secret/data/billing-service
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-21T20:18:44.084663057Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            2

Perform step 1 and 2 to validate access for application developers. You should only be able to read the secret under the path secret/billing-service.

For AWS auth

Login from an EC2 instance attached to an instance profile that is bound to the IAM principal you configured for the billing-service-dev role. You should see that the token is attached to the billing-service-dev policy.

  $ vault login -method=aws role=billing-service-dev
 
  Success! You are now authenticated. The token information displayed below
  is already stored in the token helper. You do NOT need to run "vault login"
  again. Future Vault requests will automatically use this token.
 
  Key                          Value
  ---                          -----
  token                        hvs.token
  token_accessor               cGAOT1f11td0tDCgR4Ff1Xw8.Ojft7
  token_duration               8s
  token_renewable              true
  token_policies               ["default" "billing-service-dev"]
  identity_policies            []
  policies                     ["default" "billing-service-dev"]
  token_meta_client_arn        arn:aws:sts::123456789012:assumed-role/aws-ec2role-for-vault-authmethod/i-02ddcf5aed1121703
  token_meta_client_user_id    AROATYM2SX6XH5UEKGF7G
  token_meta_role_id           53f43687-2f55-c5e9-3570-c51b1cb69763
  token_meta_account_id        123456789012
  token_meta_auth_type         iam
  token_meta_canonical_arn     arn:aws:iam::123456789012:role/aws-ec2role-for-vault-authmethod

Verify that you can read the secret under the secret/billing-service.

    $ vault kv get -mount=secret billing-service
 
    == Secret Path ==
    secret/data/billing-service
 
    ======= Metadata =======
    Key                Value
    ---                -----
    created_time       2023-09-21T21:38:43.532185535Z
    custom_metadata    <nil>
    deletion_time      n/a
    destroyed          false
    version            1
 
    === Data ===
    Key    Value
    ---    -----
    foo    bar

For AppRole auth:

Read the RoleID using the command below.

    $ ROLE_ID=$(vault read -field=role_id auth/approle/role/billing-service-dev/role-id)

Next, generate a SecretID for the role.

  $ SECRET_ID=$(vault write -force -field=secret_id auth/approle/role/billing-service-dev/secret-id)

  $ vault write auth/approle/login role_id=$ROLE_ID secret_id=$SECRET_ID
 
  Key                     Value
  ---                     -----
  token                   hvs.token
  token_accessor          JhFFIcLlqdjmM6Rq15v0DkGW
  token_duration          768h
  token_renewable         true
  token_policies          ["default" "billing-service-dev"]
  identity_policies       []
  policies                ["default" "billing-service-dev"]
  token_meta_role_name    billing-service-dev

Verify that you can read the secret under the secret/billing-service.

  $ vault kv get -mount=secret billing-service
 
  == Secret Path ==
  secret/data/billing-service
 
  ======= Metadata =======
  Key                Value
  ---                -----
  created_time       2023-09-21T21:38:43.532185535Z
  custom_metadata    <nil>
  deletion_time      n/a
  destroyed          false
  version            1
 
  === Data ===
  Key    Value
  ---    -----
  foo    bar

Consumption of secrets

At this point, you should have one human auth method (OIDC or LDAP) and one machine auth method (AWS or AppRole) enabled, allowing both humans and machines to authenticate to Vault. You have also populated some static secrets in the KV v2 secrets engine and created policies to enforce permissions on which identities can access specific secrets. The next step is to consider how to deliver these secrets from Vault to the application.

Secret consumption patterns

There are two patterns for your applications to fetch secrets from Vault: direct API integration or via Vault Agent.

Direct API integration requires that your application make API calls to authenticate with Vault and access secrets. To do this, you could either write custom integration code or depend on a third party client library(opens in new tab). Regardless of which option you choose, it requires that your application be Vault-aware, meaning that you need to write and maintain application code to enable your application to communicate with Vault. For some applications, this is not a problem and may actually be the preferred method. For example, if you have only a handful of applications or you want to keep strict, customized control over the way each application interacts with Vault, you might be fine with the added overhead of maintaining that code and making sure a member of each application's team is trained to understand Vault and that code.

In other situations — typically in large enterprises — updating each application's code base could be a monumental task or simply a non-starter for several reasons.

If your organization has hundreds or thousands of applications, you may not have the time, resources, or expertise to update and maintain Vault integration code in every application.
Your organization may not permit the teams deploying some applications to add the Vault integration code, or any code. For example, certain legacy applications may be too brittle to allow the addition of Vault integration code.
The applications (secrets consumers) and the systems (secrets producers) are, in many cases, managed by different teams. This makes coordinating the maintenance of Vault integration code into a clean workflow very difficult.
Some teams deploy third-party applications that are not owned by the organization, and therefore it is not possible to add Vault integration code.

For those situations, we recommend a much more scalable and simpler experience — using the Vault Agent or Vault Proxy(opens in new tab). Vault Agent can obtain secrets and provide them to applications, and Vault Proxy can act as a proxy between Vault and the application, optionally simplifying the authentication process and caching requests.

Note

Vault Proxy is a new feature introduced in Vault 1.14. Vault Agent has the capability to act as an API Proxy for Vault but this feature will be deprecated in future release. See this table comparing the capabilities between Vault Agent and Vault Proxy.

Vault Agent

We briefly introduced Vault Agent earlier in this document when we discussed how Vault solves the secret zero problem. Vault agent is a client daemon that uses the same Vault binary as the Vault server, except that it is run with the vault agent command. Here we will explore the features offered by the Vault Agent.

Auto-Auth - Automatically authenticate to Vault and manage the token renewal process.
Templating - Allows rendering of user-supplied templates by Vault Agent, using the token generated by the auto-auth step.
Caching - Allows client-side caching of responses containing newly created tokens and responses containing leased secrets generated off of these newly created tokens. The agent also manages the renewals of the cached tokens and leases.
Process Supervisor Mode - Runs a child process with Vault secrets injected as environment variables.
Windows Service(opens in new tab) - Allows running the Vault Agent as a Windows service.

Auto-auth

Vault is built on zero trust principles, so applications that need to fetch secrets from Vault are required to make authenticated requests by providing a Vault token. Vault Agent can make the authentication process for obtaining a Vault token easy and transparent to applications.

When Vault Agent starts up, it automatically authenticates(opens in new tab) with Vault and receives a Vault token via a process known as auto-auth(opens in new tab). The token can then be written to disk to any number of file sinks - a file sink is a local file or a file mapped via some other process (NFS, Gluster, CIFs, etc.). In addition to file permissions, the token can be protected through Vault's response-wrapping mechanism and encryption. Applications can then use this token to request secrets from Vault.

Vault Agent also manages the lifecycle of the Vault token that is received from auto-auth. This includes timely renewal of the tokens and re-authentication if a token can no longer be renewed. The sink is updated with the new token value whenever re-authentication occurs.

Vault Agent auto auth

The client can simply retrieve the token from the sink and connect to Vault using the token. This simplifies client integration since the Vault agent handles the login and token refresh logic.

To use this feature, run the Vault binary in agent mode (vault agent -config=<config file path>) on the client. The agent configuration file must specify the auth method and sink(opens in new tab) locations that dictate where the token should be written on the filesystem.

Templating

Vault Agent supports the ability to render static and dynamic(opens in new tab) secrets from Vault by using templates(opens in new tab) that describe how and where to render those secrets on the local system. Applications or service owners define these templates using Consul Template markup(opens in new tab) and then the secrets are consumed easily by the applications, eliminating the need for additional formatting and parsing. A common use case for templating is formatting credentials from Vault's Database Secrets Engine(opens in new tab) to be rendered as database connection strings for applications to consume directly. An example of this approach is demonstrated in the tutorial here(opens in new tab).

After the Vault Agent starts up and authenticates with Vault, it can be set up to retrieve secrets needed for an application. It writes these secrets to disk through the templates defined in the agent configuration, which the application can then use as needed. Review the Vault agent templates tutorial(opens in new tab).

Vault Agent templating

Caching

When Vault is controlling access to all of your secrets, it is very important to ensure that Vault's performance scales as consumption grows, so it can handle the corresponding load. The Vault Agent supports the ability to proxy and cache requests to Vault as well as responses from Vault. This helps reduce the number of requests sent to Vault servers at one time, reducing the peak load on Vault servers.

Vault Agent Caching

Reference Materials:

Vault agent caching documentation(opens in new tab)
Vault agent caching tutorial(opens in new tab)

Process supervisor mode

Vault Agent's process supervisor mode allows secrets to be injected into a process via environment variables using Consul Template markup(opens in new tab).

It is common for applications to load certain configurations from environment variables. When the application starts up, it reads environment variables and uses them to initialize connections to various databases, message queues, or other services. Often, these initialization parameters contain secrets such as database credentials, certificates, and API keys.

In this mode, Vault Agent will start the application as a child process with the environment variables populated using the configured secrets. The Vault Agent will act as a process supervisor to the application and will restart it when the configured static secrets change.

Reference Materials:

Vault agent's process supervisor mode documentation(opens in new tab)
Vault Agent - secrets as environment variables tutorial(opens in new tab)

Windows service

Vault Agent can be run as a Windows service(opens in new tab). In order to do this, Vault Agent must be registered with the Windows Service Control Manager. After Vault Agent is registered, it can be started like any other Windows service.

Vault proxy

The Vault Proxy offers an alternative integration method for Vault clients. It is designed to simplify the integration workflow for clients that need to make direct API calls to Vault.

The Vault Proxy acts as an API proxy for Vault. Any requests sent to the Vault Proxy will be forwarded to the Vault server, and these requests can be configured to optionally allow or force the automatic use of the auto-auth token, simplifying the authentication process. It also allows client-side caching of responses to reduce the load to the Vault server.

Unlike Vault Agent templating, clients must be able to integrate with Vault's APIs in order to use Vault Proxy. See this table(opens in new tab) for a comparison of the capabilities between Vault Agent and Vault Proxy.

Reference Materials:

Vault Proxy documentation(opens in new tab)

The Vault Agent advantage

The Vault Agent simplifies the zero trust adoption journey by reducing the effort and time needed by your application teams to integrate with Vault. Applications can easily integrate with the agent, which handles authentication into Vault along with the fetching, rendering, and lifecycle management of application secrets and the caching of these secrets to reduce the load on Vault.

The diagram below shows that as tasks such as fetching the initial secrets, secrets renewal, secrets rendering, templating, and caching become more Vault agnostic from the perspective of your application, they become easier to manage at scale.

Vault Agent Advantage

Vault Agent best practices

Deployment pattern

For each application that needs access to Vault, we recommend deploying one agent per application. This means that each application has its own login and identity with Vault, which aligns with the practice of least privilege. When using machine identity authentication (e.g. AWS, GCP, Azure), you should deploy a Vault agent for each host. If multiple applications share a host, use separate agent instances for each application with an auth method like AppRole that is not tied to the machine's identity.

Telemetry

The agent supports and collects various runtime metrics about its performance, the auto-auth and the cache status. We recommend that you enable telemetry and send these metrics to a monitoring platform. See the list of available metrics here(opens in new tab).

Integration pattern

How you actually integrate Vault Agent into your application deployment workflow will vary based on several factors. Ask yourself the following questions.

What is the lifecycle of my application? Is it more ephemeral or long-lived?
What are the lifecycles of my authentication tokens? Are they long-lived and do they require continual renewal to show liveliness of a service or do you want to enforce periodic re-authentications?
Do I have a group of applications running on my host which each need their own token? Can I use a native authentication capability (e.g. AWS IAM, K8s, Azure MSI, Google Cloud IAM, etc.)?

The answers to these questions will help you decide if Vault Agent should run as a daemon or as a prerequisite of a service configuration. Take for example the following Systemd service definition for running Nomad:

[Unit]
Description=Nomad Agent
Requires=consul-online.target
After=consul-online.target
[Service]
KillMode=process
KillSignal=SIGINT
Environment=VAULT_ADDR=http://active.vault.service.consul:8200
Environment=VAULT_SKIP_VERIFY=true
ExecStartPre=/usr/local/bin/vault agent -config /etc/vault-agent.d/vault-agent.hcl
ExecStart=/usr/bin/nomad-vault.sh
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

Notice the ExecStartPre directive that runs Vault Agent before the desired service starts. Here, we want to run the Vault agent as a prerequisite before Nomad starts up. To prevent Vault agent from running as a daemon, you can enable the exit_after_auth(opens in new tab) flag to instruct the agent to exit after a single successful authentication and all templates rendered.

The service startup script expects a Vault token value set as shown in the /usr/bin/nomad-vault.sh startup script:

#!/usr/bin/env bash
if [ -f /mnt/ramdisk/token ]; then
exec env VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /mnt/ramdisk/token)) \
/usr/local/bin/nomad agent \
-config=/etc/nomad.d \
-vault-tls-skip-verify=true
else
echo "Nomad service failed due to missing Vault token"
exit 1
fi

Applications which expect Vault tokens typically look for a VAULT_TOKEN environment variable. Here, you are using Vault Agent auto-auth to get a token and write it out to a RAM disk and as part of the Nomad startup script. You read the response-wrapped token from the RAM disk, and save it to your VAULT_TOKEN environment variable before starting Nomad.

Similarly, you can use Vault Agent templating to generate application configuration files as part of your application startup script. This would enable your application to be completely agnostic from Vault. For example, the template below renders the application settings for a .NET Core application. It contains most of the default configuration for an ASP.NET Core application but creates a ConnectionString based on the database username and password from Vault using the database secrets engine.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "AllowedHosts": "*",
  "ConnectionStrings": {
    {{- with secret "projects-api/database/creds/projects-api-role" }}
    "Database": "Server=.;Database=HashiCorp;user id={{ .Data.username }};password={{ .Data.password }}"
    {{- end }}
  }
}

Vault Agent will update the application settings each time Vault rotates the database username and password. ASP.NET Core allows live reload of the application each time the application settings file changes. This makes your database credentials to be short-lived and significantly reduces its vulnerability. If your application does not support live reload of application configurations, you can utilize the exec(opens in new tab) command to execute a command to restart the application each time the template changes. For more details on how to use the Vault agent to generate application configurations, see the Vault Agent with .NET Core(opens in new tab) tutorial.

Configure the Vault Agent

In this section, you will configure the Vault Agent to consume secrets from the KV v2 secrets engine.

The content in this section assumes that you have completed the configurations in the previous sections. Before you begin, you should have:

Configured either the OIDC or LDAP auth method for human access.
Configured either the AWS or AppRole auth method for machine access.
Enabled the KV v2 secrets engine and populated it with some secrets.
Configured policies to allow applications to read its required static secrets.

Vault Agent and AWS auth

Before you begin, ensure that you have configured the AWS auth method and tested that you can log in successfully from an EC2 instance.

Step 1: Create the Vault Agent configuration file

Log in to your EC2 instance and create a Vault Agent configuration called vault-agent-template.hcl using the example below.

The auto_auth(opens in new tab) block uses the aws auth method enabled at the auth/aws path on the Vault server. The namespace value specifies where your aws auth method is mounted, replace it with the name of your tenant namespace. The Vault Agent will use the billing-service-dev role to authenticate.

The template(opens in new tab) block configures Vault agent templating. The content parameter embeds the content of a consul-template(opens in new tab) directly into the configuration file rather than supplying the source path to the template file. The destination parameter tells the agent where the output will be written to. See a configuration example here(opens in new tab) where the source path is provided for the template file.

Finally, the vault(opens in new tab) block configures how the Agent connects to the remote Vault server.

pid_file = "./pidfile"
 
auto_auth {
  method "aws" {
    namespace = "<tenant namespace>"
    mount_path = "auth/aws"
 
    config = {
      type = "iam"
      role = "billing-service-dev"
    }
  }
}
 
template {
  contents = "{{ with secret \"secret/billing-service\" }}{{ .Data.data.foo }}{{ end }}"
  destination = "/home/vault-user/render-content.txt"
}
 
vault {
  address = "https://dev.vault.example.com:8200"
  ca_cert = "/opt/vault/tls/ca.pem"
}

Step 2: Start the Vault Agent

Execute the following command to start the agent.

$ vault agent -config=./vault-agent-template.hcl

You should see the following.

2023-10-17T19:31:51.080Z [INFO]  agent.exec.server: starting exec server
2023-10-17T19:31:51.081Z [INFO]  agent.exec.server: no env templates or exec config, exiting
2023-10-17T19:31:51.082Z [INFO]  agent.auth.handler: starting auth handler
2023-10-17T19:31:51.082Z [INFO]  agent.auth.handler: authenticating
2023-10-17T19:31:51.084Z [INFO]  agent.sink.server: starting sink server
2023-10-17T19:31:51.085Z [INFO]  agent.template.server: starting template server
2023-10-17T19:31:51.085Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-10-17T19:31:51.086Z [INFO] (runner) creating watcher
2023-10-17T19:31:51.138Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2023-10-17T19:31:51.139Z [INFO]  agent.auth.handler: starting renewal process
2023-10-17T19:31:51.140Z [INFO]  agent.template.server: template server received new token
2023-10-17T19:31:51.140Z [INFO] (runner) stopping
2023-10-17T19:31:51.140Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-10-17T19:31:51.140Z [INFO] (runner) creating watcher
2023-10-17T19:31:51.143Z [INFO] (runner) starting
2023-10-17T19:31:51.152Z [INFO]  agent.auth.handler: renewed auth token
2023-10-17T19:31:51.175Z [INFO] (runner) rendered "(dynamic)" => "/home/ssm-user/render-content.txt"
^C==> Vault Agent shutdown triggered
2023-10-17T19:31:58.456Z [INFO]  agent.exec.server: exec server stopped
2023-10-17T19:31:58.456Z [INFO] (runner) stopping
2023-10-17T19:31:58.456Z [INFO]  agent.template.server: template server stopped
2023-10-17T19:31:58.456Z [INFO]  agent.auth.handler: shutdown triggered, stopping lifetime watcher
2023-10-17T19:31:58.456Z [INFO]  agent.auth.handler: auth handler stopped
2023-10-17T19:31:58.456Z [INFO]  agent.sink.server: sink server stopped
2023-10-17T19:31:58.456Z [INFO]  agent: sinks finished, exiting

Now navigate to the directory where you have configured the agent to output the templated secrets. You should see that the content of the file contains the value of the secret.

$ cat render-content.txt
bar

Step 3: Render secrets as environment variables

Another way to provide secrets to your application is to inject them as environment variables. Vault Agent's process supervisor mode allows Vault secrets to be injected into a process via environment variables using Consul Template markup.

Create an agent configuration called vault-agent-envar.hcl using the example below.

pid_file = "./pidfile"
 
auto_auth {
  method "aws" {
    namespace = "<tenant namespace>"
    mount_path = "auth/aws"
 
    config = {
      type = "iam"
      role = "billing-service-dev"
    }
  }
}
 
template_config {
   static_secret_render_interval = "1m"
   exit_on_retry_failure         = true
}
 
vault {
  address = "https://dev.vault.example.com:8200"
  ca_cert = "/opt/vault/tls/ca.pem"
}
 
env_template "MY_SECRET" {
   contents             = "{{ with secret \"secret/data/billing-service\" }}{{ .Data.data.foo }}{{ end }}"
   error_on_missing_key = true
}
 
exec {
   command                   = ["./kv-demo.sh"]
   restart_on_secret_changes = "always"
   restart_stop_signal       = "SIGTERM"
}

The template_config block has configuration entries that affect all templates. The static_secret_render_interval(opens in new tab) entry configures how often Vault Agent Template should render non-lease secrets such as KV v2. Here we are telling the Agent to re-render the template every minute. The exit_on_retry_failure(opens in new tab) entry configures Vault Agent to exit after it has exhausted its number of template retry attempts due to failures. The number of retries can be configured using the retry stanza(opens in new tab), but the default number of retries is 12.

The env_template(opens in new tab) block maps the template specified in the contents to the environment variable name in the title of the block. In the example, we are injecting an environment variable called MY_SECRET with the value of the secret foo at secret/data/myapp. The error_on_missing_key configuration tells the Agent to exit with an error if the secret key being accessed does not exist.

The exec(opens in new tab) block configures how the child process is started. The command(opens in new tab) entry specifies the command to start the child process. restart_on_secret_changes(opens in new tab) controls whether the agent will restart the child process on secret changes. For example, if the static secret is updated (on static_secret_render_interval(opens in new tab)), then the agent will restart the child process. The restart_stop_signal(opens in new tab) is the signal that the agent will send to the child process when a secret has been updated and the process needs to be restarted. The process has 30 seconds after this signal is sent until SIGKILL is sent to force the child process to stop.

Create a simple shell script called kv-demo.sh which will display the value of the environment variable named MY_SECRET.

#!/bin/sh
 
echo
echo "==== DEMO APP ===="
echo "MY_SECRET    = ${MY_SECRET}"
 
sleep 100

Step 4: Start the Vault Agent with new configuration

Start the agent by executing the following command.

$ vault agent -config=./vault-agent-envar.hcl

You should see the following.

2023-10-17T19:35:45.423Z [INFO]  agent.exec.server: starting exec server
2023-10-17T19:35:45.423Z [INFO] (runner) creating new runner (dry: true, once: false)
2023-10-17T19:35:45.424Z [INFO] (runner) creating watcher
2023-10-17T19:35:45.424Z [INFO]  agent.auth.handler: starting auth handler
2023-10-17T19:35:45.424Z [INFO]  agent.auth.handler: authenticating
2023-10-17T19:35:45.424Z [INFO]  agent.sink.server: starting sink server
2023-10-17T19:35:45.424Z [INFO]  agent.template.server: starting template server
2023-10-17T19:35:45.424Z [INFO]  agent.template.server: no templates found
2023-10-17T19:35:45.469Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2023-10-17T19:35:45.469Z [INFO]  agent.auth.handler: starting renewal process
2023-10-17T19:35:45.470Z [INFO]  agent.exec.server: exec server received new token
2023-10-17T19:35:45.470Z [INFO] (runner) stopping
2023-10-17T19:35:45.470Z [INFO] (runner) creating new runner (dry: true, once: false)
2023-10-17T19:35:45.470Z [INFO] (runner) creating watcher
2023-10-17T19:35:45.470Z [INFO] (runner) starting
2023-10-17T19:35:45.480Z [INFO]  agent.auth.handler: renewed auth token
2023-10-17T19:35:45.494Z [INFO] (runner) rendered "(dynamic)" => "MY_SECRET"
2023-10-17T19:35:47.495Z [INFO]  agent.exec.server: [INFO] (child) spawning: ./kv-demo.sh
 
==== DEMO APP ====
MY_SECRET    = bar

Now change the value of the static secret in Vault. The agent will re-render the secret every minute, and if the value of the secret changes, it will restart the child process and the updated secret value will be shown.

2023-09-26T19:32:36.727Z [INFO] (runner) rendered "(dynamic)" => "MY_SECRET"
2023-09-26T19:32:38.727Z [INFO]  agent.exec.server: stopping process: process_id=118363
2023-09-26T19:32:38.728Z [INFO]  agent.exec.server: [INFO] (child) stopping process
2023-09-26T19:32:38.728Z [INFO]  agent.exec.server: [INFO] (child) spawning: ./kv-demo.sh
 
==== DEMO APP ====
MY_SECRET    = bazzzz

Vault Agent and AppRole auth

Before you begin, ensure that you have configured the AppRole auth method and tested that you can login successfully.

Step 1: Create the Vault Agent configuration file

Login to the VM where the application is deployed and create a Vault Agent configuration called vault-agent-template.hcl using the example below.

The auto_auth(opens in new tab) block uses the approle auth method enabled at the auth/approle path on the Vault server. The namespace value specifies where your approle auth method is mounted, replace it with the name of your tenant namespace. The role_id_file_path(opens in new tab) is the path to the file with the Role ID and the secret_id_file_path(opens in new tab) is the path to the file with the Secret ID. By default, the Vault agent will delete the Secret ID file after it is read. The secret_id_response_wrapping_path(opens in new tab) tells the Agent that the Secret ID is response-wrapped and should be verified against the response wrapping path set here. Using a response-wrapped Secret ID is recommended as discussed in the AppRole Best Practice(opens in new tab) section.

The template(opens in new tab) block configures Vault agent templating. The source parameter specifies the source path to the template file. The destination parameter tells the agent where the output will be written to.

Finally, the vault(opens in new tab) block configures how the Agent connects to the remote Vault server.

pid_file = "./pidfile"
 
auto_auth {
 method {
   type = "approle"
   namespace = "demo"
   config = {
     role_id_file_path = "/home/vault/app1_role_id"
     secret_id_file_path = "/home/vault/app1_secret_id"
     secret_id_response_wrapping_path = "auth/approle/role/billing-service-dev/secret-id"
   }
 }
}
 
template {
 source = "/home/vault/kv.tpl"
 destination = "/home/vault/kv.html"
}
 
vault {
 address = "https://dev.vault.example.com:8200"
 ca_cert = "/opt/vault/tls/ca.pem"
}

Step 2: Create the template file

Create the template file referenced in the agent configuration. The example below creates an HTML file by fetching the secret foo from the path secret/billing-service. You can find more information on the consul-templating language here(opens in new tab).

tee /home/vault/kv.tpl <<EOF
{{ with secret "secret/billing-service" }}
<html>
<head>
</head>
  <body>
    <li><strong>foo</strong> &#58; {{ .Data.data.foo }}</li>
</body>
</html>
{{ end }}
EOF

Step 3: Start the Vault Agent

Start the agent by executing the following command.

$ vault agent -config=./vault-agent-template.hcl

You should see the following.

2023-10-17T20:51:51.119Z [INFO]  agent.exec.server: starting exec server
2023-10-17T20:51:51.119Z [INFO]  agent.exec.server: no env templates or exec config, exiting
2023-10-17T20:51:51.119Z [INFO]  agent.auth.handler: starting auth handler
2023-10-17T20:51:51.119Z [INFO]  agent.auth.handler: authenticating
2023-10-17T20:51:51.120Z [INFO]  agent.sink.server: starting sink server
2023-10-17T20:51:51.120Z [INFO]  agent.template.server: starting template server
2023-10-17T20:51:51.120Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-10-17T20:51:51.122Z [INFO] (runner) creating watcher
2023-10-17T20:51:51.189Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2023-10-17T20:51:51.189Z [INFO]  agent.auth.handler: starting renewal process
2023-10-17T20:51:51.194Z [INFO]  agent.template.server: template server received new token
2023-10-17T20:51:51.194Z [INFO] (runner) stopping
2023-10-17T20:51:51.198Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-10-17T20:51:51.199Z [INFO] (runner) creating watcher
2023-10-17T20:51:51.203Z [INFO] (runner) starting
2023-10-17T20:51:51.211Z [INFO]  agent.auth.handler: renewed auth token
2023-10-17T20:51:51.237Z [INFO] (runner) rendered "/vault-agent/kv.tpl" => "/vault-agent/kv.html"

Now navigate to the directory where you have configured the agent to output the templated secrets. You should see that the content of the file contains the value of the secret.

$ cat kv.html
 
<html>
<head>
</head>
 <body>
   <li><strong>foo</strong> &#58; bar</li>
</body>
</html>

Authentication for applications

Secrets Sync