Static secrets

Note

Static secrets are configured and managed by platform operators. See the Administration Guide: Static secrets management.

Consumption of secrets

At this point, you should have one human auth method (OIDC or LDAP) and one machine auth method (AWS or AppRole) enabled, allowing both humans and machines to authenticate to Vault. You have also populated some static secrets in the KV v2 secrets engine and created policies to enforce permissions on which identities can access specific secrets. The next step is to consider how to deliver these secrets from Vault to the application.

Secret consumption patterns

There are two patterns for your applications to fetch secrets from Vault: direct API integration or via Vault Agent.

Direct API integration requires that your application make API calls to authenticate with Vault and access secrets. To do this, you could either write custom integration code or depend on a third party client library(opens in new tab). Regardless of which option you choose, it requires that your application be Vault-aware, meaning that you need to write and maintain application code to enable your application to communicate with Vault. For some applications, this is not a problem and may actually be the preferred method. For example, if you have only a handful of applications or you want to keep strict, customized control over the way each application interacts with Vault, you might be fine with the added overhead of maintaining that code and making sure a member of each application's team is trained to understand Vault and that code.

In other situations — typically in large enterprises — updating each application's code base could be a monumental task or simply a non-starter for several reasons.

If your organization has hundreds or thousands of applications, you may not have the time, resources, or expertise to update and maintain Vault integration code in every application.
Your organization may not permit the teams deploying some applications to add the Vault integration code, or any code. For example, certain legacy applications may be too brittle to allow the addition of Vault integration code.
The applications (secrets consumers) and the systems (secrets producers) are, in many cases, managed by different teams. This makes coordinating the maintenance of Vault integration code into a clean workflow very difficult.
Some teams deploy third-party applications that are not owned by the organization, and therefore it is not possible to add Vault integration code.

For those situations, we recommend a much more scalable and simpler experience — using the Vault Agent or Vault Proxy(opens in new tab). Vault Agent can obtain secrets and provide them to applications, and Vault Proxy can act as a proxy between Vault and the application, optionally simplifying the authentication process and caching requests.

Note

Vault Proxy is a new feature introduced in Vault 1.14. Vault Agent has the capability to act as an API Proxy for Vault but this feature will be deprecated in future release. See this table comparing the capabilities between Vault Agent and Vault Proxy.

Vault Agent

We briefly introduced Vault Agent earlier in this document when we discussed how Vault solves the secret zero problem. Vault agent is a client daemon that uses the same Vault binary as the Vault server, except that it is run with the vault agent command. Here we will explore the features offered by the Vault Agent.

Auto-Auth - Automatically authenticate to Vault and manage the token renewal process.
Templating - Allows rendering of user-supplied templates by Vault Agent, using the token generated by the auto-auth step.
Caching - Allows client-side caching of responses containing newly created tokens and responses containing leased secrets generated off of these newly created tokens. The agent also manages the renewals of the cached tokens and leases.
Process Supervisor Mode - Runs a child process with Vault secrets injected as environment variables.
Windows Service(opens in new tab) - Allows running the Vault Agent as a Windows service.

Auto-auth

Vault is built on zero trust principles, so applications that need to fetch secrets from Vault are required to make authenticated requests by providing a Vault token. Vault Agent can make the authentication process for obtaining a Vault token easy and transparent to applications.

When Vault Agent starts up, it automatically authenticates(opens in new tab) with Vault and receives a Vault token via a process known as auto-auth(opens in new tab). The token can then be written to disk to any number of file sinks - a file sink is a local file or a file mapped via some other process (NFS, Gluster, CIFs, etc.). In addition to file permissions, the token can be protected through Vault's response-wrapping mechanism and encryption. Applications can then use this token to request secrets from Vault.

Vault Agent also manages the lifecycle of the Vault token that is received from auto-auth. This includes timely renewal of the tokens and re-authentication if a token can no longer be renewed. The sink is updated with the new token value whenever re-authentication occurs.

Vault Agent auto auth

The client can simply retrieve the token from the sink and connect to Vault using the token. This simplifies client integration since the Vault agent handles the login and token refresh logic.

To use this feature, run the Vault binary in agent mode (vault agent -config=<config file path>) on the client. The agent configuration file must specify the auth method and sink(opens in new tab) locations that dictate where the token should be written on the filesystem.

Templating

Vault Agent supports the ability to render static and dynamic(opens in new tab) secrets from Vault by using templates(opens in new tab) that describe how and where to render those secrets on the local system. Applications or service owners define these templates using Consul Template markup(opens in new tab) and then the secrets are consumed easily by the applications, eliminating the need for additional formatting and parsing. A common use case for templating is formatting credentials from Vault's Database Secrets Engine(opens in new tab) to be rendered as database connection strings for applications to consume directly. An example of this approach is demonstrated in the tutorial here(opens in new tab).

After the Vault Agent starts up and authenticates with Vault, it can be set up to retrieve secrets needed for an application. It writes these secrets to disk through the templates defined in the agent configuration, which the application can then use as needed. Review the Vault agent templates tutorial(opens in new tab).

Vault Agent templating

Caching

When Vault is controlling access to all of your secrets, it is very important to ensure that Vault's performance scales as consumption grows, so it can handle the corresponding load. The Vault Agent supports the ability to proxy and cache requests to Vault as well as responses from Vault. This helps reduce the number of requests sent to Vault servers at one time, reducing the peak load on Vault servers.

Vault Agent Caching

Reference Materials:

Vault agent caching documentation(opens in new tab)
Vault agent caching tutorial(opens in new tab)

Process supervisor mode

Vault Agent's process supervisor mode allows secrets to be injected into a process via environment variables using Consul Template markup(opens in new tab).

It is common for applications to load certain configurations from environment variables. When the application starts up, it reads environment variables and uses them to initialize connections to various databases, message queues, or other services. Often, these initialization parameters contain secrets such as database credentials, certificates, and API keys.

In this mode, Vault Agent will start the application as a child process with the environment variables populated using the configured secrets. The Vault Agent will act as a process supervisor to the application and will restart it when the configured static secrets change.

Reference Materials:

Vault agent's process supervisor mode documentation(opens in new tab)
Vault Agent - secrets as environment variables tutorial(opens in new tab)

Windows service

Vault Agent can be run as a Windows service(opens in new tab). In order to do this, Vault Agent must be registered with the Windows Service Control Manager. After Vault Agent is registered, it can be started like any other Windows service.

Vault proxy

The Vault Proxy offers an alternative integration method for Vault clients. It is designed to simplify the integration workflow for clients that need to make direct API calls to Vault.

The Vault Proxy acts as an API proxy for Vault. Any requests sent to the Vault Proxy will be forwarded to the Vault server, and these requests can be configured to optionally allow or force the automatic use of the auto-auth token, simplifying the authentication process. It also allows client-side caching of responses to reduce the load to the Vault server.

Unlike Vault Agent templating, clients must be able to integrate with Vault's APIs in order to use Vault Proxy. See this table(opens in new tab) for a comparison of the capabilities between Vault Agent and Vault Proxy.

Reference Materials:

Vault Proxy documentation(opens in new tab)

The Vault Agent advantage

The Vault Agent simplifies the zero trust adoption journey by reducing the effort and time needed by your application teams to integrate with Vault. Applications can easily integrate with the agent, which handles authentication into Vault along with the fetching, rendering, and lifecycle management of application secrets and the caching of these secrets to reduce the load on Vault.

The diagram below shows that as tasks such as fetching the initial secrets, secrets renewal, secrets rendering, templating, and caching become more Vault agnostic from the perspective of your application, they become easier to manage at scale.

Vault Agent Advantage

Vault Agent best practices

Deployment pattern

For each application that needs access to Vault, we recommend deploying one agent per application. This means that each application has its own login and identity with Vault, which aligns with the practice of least privilege. When using machine identity authentication (e.g. AWS, GCP, Azure), you should deploy a Vault agent for each host. If multiple applications share a host, use separate agent instances for each application with an auth method like AppRole that is not tied to the machine's identity.

Telemetry

The agent supports and collects various runtime metrics about its performance, the auto-auth and the cache status. We recommend that you enable telemetry and send these metrics to a monitoring platform. See the list of available metrics here(opens in new tab).

Integration pattern

How you actually integrate Vault Agent into your application deployment workflow will vary based on several factors. Ask yourself the following questions.

What is the lifecycle of my application? Is it more ephemeral or long-lived?
What are the lifecycles of my authentication tokens? Are they long-lived and do they require continual renewal to show liveliness of a service or do you want to enforce periodic re-authentications?
Do I have a group of applications running on my host which each need their own token? Can I use a native authentication capability (e.g. AWS IAM, K8s, Azure MSI, Google Cloud IAM, etc.)?

The answers to these questions will help you decide if Vault Agent should run as a daemon or as a prerequisite of a service configuration. Take for example the following Systemd service definition for running Nomad:

[Unit]
Description=Nomad Agent
Requires=consul-online.target
After=consul-online.target
[Service]
KillMode=process
KillSignal=SIGINT
Environment=VAULT_ADDR=http://active.vault.service.consul:8200
Environment=VAULT_SKIP_VERIFY=true
ExecStartPre=/usr/local/bin/vault agent -config /etc/vault-agent.d/vault-agent.hcl
ExecStart=/usr/bin/nomad-vault.sh
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target

Notice the ExecStartPre directive that runs Vault Agent before the desired service starts. Here, we want to run the Vault agent as a prerequisite before Nomad starts up. To prevent Vault agent from running as a daemon, you can enable the exit_after_auth(opens in new tab) flag to instruct the agent to exit after a single successful authentication and all templates rendered.

The service startup script expects a Vault token value set as shown in the /usr/bin/nomad-vault.sh startup script:

#!/usr/bin/env bash
if [ -f /mnt/ramdisk/token ]; then
exec env VAULT_TOKEN=$(vault unwrap -field=token $(jq -r '.token' /mnt/ramdisk/token)) \
/usr/local/bin/nomad agent \
-config=/etc/nomad.d \
-vault-tls-skip-verify=true
else
echo "Nomad service failed due to missing Vault token"
exit 1
fi

Applications which expect Vault tokens typically look for a VAULT_TOKEN environment variable. Here, you are using Vault Agent auto-auth to get a token and write it out to a RAM disk and as part of the Nomad startup script. You read the response-wrapped token from the RAM disk, and save it to your VAULT_TOKEN environment variable before starting Nomad.

Similarly, you can use Vault Agent templating to generate application configuration files as part of your application startup script. This would enable your application to be completely agnostic from Vault. For example, the template below renders the application settings for a .NET Core application. It contains most of the default configuration for an ASP.NET Core application but creates a ConnectionString based on the database username and password from Vault using the database secrets engine.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "AllowedHosts": "*",
  "ConnectionStrings": {
    {{- with secret "projects-api/database/creds/projects-api-role" }}
    "Database": "Server=.;Database=HashiCorp;user id={{ .Data.username }};password={{ .Data.password }}"
    {{- end }}
  }
}

Vault Agent will update the application settings each time Vault rotates the database username and password. ASP.NET Core allows live reload of the application each time the application settings file changes. This makes your database credentials to be short-lived and significantly reduces its vulnerability. If your application does not support live reload of application configurations, you can utilize the exec(opens in new tab) command to execute a command to restart the application each time the template changes. For more details on how to use the Vault agent to generate application configurations, see the Vault Agent with .NET Core(opens in new tab) tutorial.

Configure the Vault Agent

In this section, you will configure the Vault Agent to consume secrets from the KV v2 secrets engine.

The content in this section assumes that you have completed the configurations in the previous sections. Before you begin, you should have:

Configured either the OIDC or LDAP auth method for human access.
Configured either the AWS or AppRole auth method for machine access.
Enabled the KV v2 secrets engine and populated it with some secrets.
Configured policies to allow applications to read its required static secrets.

Vault Agent and AWS auth

Before you begin, ensure that you have configured the AWS auth method and tested that you can log in successfully from an EC2 instance.

Step 1: Create the Vault Agent configuration file

Log in to your EC2 instance and create a Vault Agent configuration called vault-agent-template.hcl using the example below.

The auto_auth(opens in new tab) block uses the aws auth method enabled at the auth/aws path on the Vault server. The namespace value specifies where your aws auth method is mounted, replace it with the name of your tenant namespace. The Vault Agent will use the billing-service-dev role to authenticate.

The template(opens in new tab) block configures Vault agent templating. The content parameter embeds the content of a consul-template(opens in new tab) directly into the configuration file rather than supplying the source path to the template file. The destination parameter tells the agent where the output will be written to. See a configuration example here(opens in new tab) where the source path is provided for the template file.

Finally, the vault(opens in new tab) block configures how the Agent connects to the remote Vault server.

pid_file = "./pidfile"
 
auto_auth {
  method "aws" {
    namespace = "<tenant namespace>"
    mount_path = "auth/aws"
 
    config = {
      type = "iam"
      role = "billing-service-dev"
    }
  }
}
 
template {
  contents = "{{ with secret \"secret/billing-service\" }}{{ .Data.data.foo }}{{ end }}"
  destination = "/home/vault-user/render-content.txt"
}
 
vault {
  address = "https://dev.vault.example.com:8200"
  ca_cert = "/opt/vault/tls/ca.pem"
}

Step 2: Start the Vault Agent

Execute the following command to start the agent.

$ vault agent -config=./vault-agent-template.hcl

You should see the following.

2023-10-17T19:31:51.080Z [INFO]  agent.exec.server: starting exec server
2023-10-17T19:31:51.081Z [INFO]  agent.exec.server: no env templates or exec config, exiting
2023-10-17T19:31:51.082Z [INFO]  agent.auth.handler: starting auth handler
2023-10-17T19:31:51.082Z [INFO]  agent.auth.handler: authenticating
2023-10-17T19:31:51.084Z [INFO]  agent.sink.server: starting sink server
2023-10-17T19:31:51.085Z [INFO]  agent.template.server: starting template server
2023-10-17T19:31:51.085Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-10-17T19:31:51.086Z [INFO] (runner) creating watcher
2023-10-17T19:31:51.138Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2023-10-17T19:31:51.139Z [INFO]  agent.auth.handler: starting renewal process
2023-10-17T19:31:51.140Z [INFO]  agent.template.server: template server received new token
2023-10-17T19:31:51.140Z [INFO] (runner) stopping
2023-10-17T19:31:51.140Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-10-17T19:31:51.140Z [INFO] (runner) creating watcher
2023-10-17T19:31:51.143Z [INFO] (runner) starting
2023-10-17T19:31:51.152Z [INFO]  agent.auth.handler: renewed auth token
2023-10-17T19:31:51.175Z [INFO] (runner) rendered "(dynamic)" => "/home/ssm-user/render-content.txt"
^C==> Vault Agent shutdown triggered
2023-10-17T19:31:58.456Z [INFO]  agent.exec.server: exec server stopped
2023-10-17T19:31:58.456Z [INFO] (runner) stopping
2023-10-17T19:31:58.456Z [INFO]  agent.template.server: template server stopped
2023-10-17T19:31:58.456Z [INFO]  agent.auth.handler: shutdown triggered, stopping lifetime watcher
2023-10-17T19:31:58.456Z [INFO]  agent.auth.handler: auth handler stopped
2023-10-17T19:31:58.456Z [INFO]  agent.sink.server: sink server stopped
2023-10-17T19:31:58.456Z [INFO]  agent: sinks finished, exiting

Now navigate to the directory where you have configured the agent to output the templated secrets. You should see that the content of the file contains the value of the secret.

$ cat render-content.txt
bar

Step 3: Render secrets as environment variables

Another way to provide secrets to your application is to inject them as environment variables. Vault Agent's process supervisor mode allows Vault secrets to be injected into a process via environment variables using Consul Template markup.

Create an agent configuration called vault-agent-envar.hcl using the example below.

pid_file = "./pidfile"
 
auto_auth {
  method "aws" {
    namespace = "<tenant namespace>"
    mount_path = "auth/aws"
 
    config = {
      type = "iam"
      role = "billing-service-dev"
    }
  }
}
 
template_config {
   static_secret_render_interval = "1m"
   exit_on_retry_failure         = true
}
 
vault {
  address = "https://dev.vault.example.com:8200"
  ca_cert = "/opt/vault/tls/ca.pem"
}
 
env_template "MY_SECRET" {
   contents             = "{{ with secret \"secret/data/billing-service\" }}{{ .Data.data.foo }}{{ end }}"
   error_on_missing_key = true
}
 
exec {
   command                   = ["./kv-demo.sh"]
   restart_on_secret_changes = "always"
   restart_stop_signal       = "SIGTERM"
}

The template_config block has configuration entries that affect all templates. The static_secret_render_interval(opens in new tab) entry configures how often Vault Agent Template should render non-lease secrets such as KV v2. Here we are telling the Agent to re-render the template every minute. The exit_on_retry_failure(opens in new tab) entry configures Vault Agent to exit after it has exhausted its number of template retry attempts due to failures. The number of retries can be configured using the retry stanza(opens in new tab), but the default number of retries is 12.

The env_template(opens in new tab) block maps the template specified in the contents to the environment variable name in the title of the block. In the example, we are injecting an environment variable called MY_SECRET with the value of the secret foo at secret/data/myapp. The error_on_missing_key configuration tells the Agent to exit with an error if the secret key being accessed does not exist.

The exec(opens in new tab) block configures how the child process is started. The command(opens in new tab) entry specifies the command to start the child process. restart_on_secret_changes(opens in new tab) controls whether the agent will restart the child process on secret changes. For example, if the static secret is updated (on static_secret_render_interval(opens in new tab)), then the agent will restart the child process. The restart_stop_signal(opens in new tab) is the signal that the agent will send to the child process when a secret has been updated and the process needs to be restarted. The process has 30 seconds after this signal is sent until SIGKILL is sent to force the child process to stop.

Create a simple shell script called kv-demo.sh which will display the value of the environment variable named MY_SECRET.

#!/bin/sh
 
echo
echo "==== DEMO APP ===="
echo "MY_SECRET    = ${MY_SECRET}"
 
sleep 100

Step 4: Start the Vault Agent with new configuration

Start the agent by executing the following command.

$ vault agent -config=./vault-agent-envar.hcl

You should see the following.

2023-10-17T19:35:45.423Z [INFO]  agent.exec.server: starting exec server
2023-10-17T19:35:45.423Z [INFO] (runner) creating new runner (dry: true, once: false)
2023-10-17T19:35:45.424Z [INFO] (runner) creating watcher
2023-10-17T19:35:45.424Z [INFO]  agent.auth.handler: starting auth handler
2023-10-17T19:35:45.424Z [INFO]  agent.auth.handler: authenticating
2023-10-17T19:35:45.424Z [INFO]  agent.sink.server: starting sink server
2023-10-17T19:35:45.424Z [INFO]  agent.template.server: starting template server
2023-10-17T19:35:45.424Z [INFO]  agent.template.server: no templates found
2023-10-17T19:35:45.469Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2023-10-17T19:35:45.469Z [INFO]  agent.auth.handler: starting renewal process
2023-10-17T19:35:45.470Z [INFO]  agent.exec.server: exec server received new token
2023-10-17T19:35:45.470Z [INFO] (runner) stopping
2023-10-17T19:35:45.470Z [INFO] (runner) creating new runner (dry: true, once: false)
2023-10-17T19:35:45.470Z [INFO] (runner) creating watcher
2023-10-17T19:35:45.470Z [INFO] (runner) starting
2023-10-17T19:35:45.480Z [INFO]  agent.auth.handler: renewed auth token
2023-10-17T19:35:45.494Z [INFO] (runner) rendered "(dynamic)" => "MY_SECRET"
2023-10-17T19:35:47.495Z [INFO]  agent.exec.server: [INFO] (child) spawning: ./kv-demo.sh
 
==== DEMO APP ====
MY_SECRET    = bar

Now change the value of the static secret in Vault. The agent will re-render the secret every minute, and if the value of the secret changes, it will restart the child process and the updated secret value will be shown.

2023-09-26T19:32:36.727Z [INFO] (runner) rendered "(dynamic)" => "MY_SECRET"
2023-09-26T19:32:38.727Z [INFO]  agent.exec.server: stopping process: process_id=118363
2023-09-26T19:32:38.728Z [INFO]  agent.exec.server: [INFO] (child) stopping process
2023-09-26T19:32:38.728Z [INFO]  agent.exec.server: [INFO] (child) spawning: ./kv-demo.sh
 
==== DEMO APP ====
MY_SECRET    = bazzzz

Vault Agent and AppRole auth

Before you begin, ensure that you have configured the AppRole auth method and tested that you can login successfully.

Step 1: Create the Vault Agent configuration file

Login to the VM where the application is deployed and create a Vault Agent configuration called vault-agent-template.hcl using the example below.

The auto_auth(opens in new tab) block uses the approle auth method enabled at the auth/approle path on the Vault server. The namespace value specifies where your approle auth method is mounted, replace it with the name of your tenant namespace. The role_id_file_path(opens in new tab) is the path to the file with the Role ID and the secret_id_file_path(opens in new tab) is the path to the file with the Secret ID. By default, the Vault agent will delete the Secret ID file after it is read. The secret_id_response_wrapping_path(opens in new tab) tells the Agent that the Secret ID is response-wrapped and should be verified against the response wrapping path set here. Using a response-wrapped Secret ID is recommended as discussed in the AppRole Best Practice section.

The template(opens in new tab) block configures Vault agent templating. The source parameter specifies the source path to the template file. The destination parameter tells the agent where the output will be written to.

Finally, the vault(opens in new tab) block configures how the Agent connects to the remote Vault server.

pid_file = "./pidfile"
 
auto_auth {
 method {
   type = "approle"
   namespace = "demo"
   config = {
     role_id_file_path = "/home/vault/app1_role_id"
     secret_id_file_path = "/home/vault/app1_secret_id"
     secret_id_response_wrapping_path = "auth/approle/role/billing-service-dev/secret-id"
   }
 }
}
 
template {
 source = "/home/vault/kv.tpl"
 destination = "/home/vault/kv.html"
}
 
vault {
 address = "https://dev.vault.example.com:8200"
 ca_cert = "/opt/vault/tls/ca.pem"
}

Step 2: Create the template file

Create the template file referenced in the agent configuration. The example below creates an HTML file by fetching the secret foo from the path secret/billing-service. You can find more information on the consul-templating language here(opens in new tab).

tee /home/vault/kv.tpl <<EOF
{{ with secret "secret/billing-service" }}
<html>
<head>
</head>
  <body>
    <li><strong>foo</strong> &#58; {{ .Data.data.foo }}</li>
</body>
</html>
{{ end }}
EOF

Step 3: Start the Vault Agent

Start the agent by executing the following command.

$ vault agent -config=./vault-agent-template.hcl

You should see the following.

2023-10-17T20:51:51.119Z [INFO]  agent.exec.server: starting exec server
2023-10-17T20:51:51.119Z [INFO]  agent.exec.server: no env templates or exec config, exiting
2023-10-17T20:51:51.119Z [INFO]  agent.auth.handler: starting auth handler
2023-10-17T20:51:51.119Z [INFO]  agent.auth.handler: authenticating
2023-10-17T20:51:51.120Z [INFO]  agent.sink.server: starting sink server
2023-10-17T20:51:51.120Z [INFO]  agent.template.server: starting template server
2023-10-17T20:51:51.120Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-10-17T20:51:51.122Z [INFO] (runner) creating watcher
2023-10-17T20:51:51.189Z [INFO]  agent.auth.handler: authentication successful, sending token to sinks
2023-10-17T20:51:51.189Z [INFO]  agent.auth.handler: starting renewal process
2023-10-17T20:51:51.194Z [INFO]  agent.template.server: template server received new token
2023-10-17T20:51:51.194Z [INFO] (runner) stopping
2023-10-17T20:51:51.198Z [INFO] (runner) creating new runner (dry: false, once: false)
2023-10-17T20:51:51.199Z [INFO] (runner) creating watcher
2023-10-17T20:51:51.203Z [INFO] (runner) starting
2023-10-17T20:51:51.211Z [INFO]  agent.auth.handler: renewed auth token
2023-10-17T20:51:51.237Z [INFO] (runner) rendered "/vault-agent/kv.tpl" => "/vault-agent/kv.html"

Now navigate to the directory where you have configured the agent to output the templated secrets. You should see that the content of the file contains the value of the secret.

$ cat kv.html
 
<html>
<head>
</head>
 <body>
   <li><strong>foo</strong> &#58; bar</li>
</body>
</html>