Vault Agent Quick Start

13min
Vault

What is Vault Agent?

Vault Agent behaves as a client-side daemon to make requests to Vault on behalf of the client application. This includes the authentication to Vault.

Vault clients (human users, applications, etc.) must authenticate with Vault and get a client token to make API requests. Because tokens have time-to-live (TTL), the clients must renew the token's TTL or re-authenticate to Vault based on its TTL. Vault Agent authenticates with Vault and manage the token's lifecycle so that the client application doesn't have to.

In addition, you can inject the Consul Template markup into Vault Agent so that secrets can be rendered to files where the client application load data from.

This eliminates the need to change your application code to invoke the Vault API. Your existing applications can remain Vault-unaware.

Prerequisites

To complete this tutorial, you will need:

Vault binary version 1.13.0 or later

Note

This tutorial highlights some of the Vault Agent enhancements introduced in Vault 1.13.0.

Lab setup

Open a terminal and start a Vault dev server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.
Insecure operation: Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```

Log into Vault.

$ vault login root

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                root
token_accessor       9v9YS37o8lvXELvEQY6NqTtV
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Tip

Vault server is ready and the token value (root) is store in the $HOME/.vault-token file.

Note: If you do not have access to an HCP Vault cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.

$ vault login $VAULT_TOKEN

WARNING! The VAULT_TOKEN environment variable is set! The value of this
variable will take precedence; if this is unwanted please unset VAULT_TOKEN or
update its value accordingly.

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.CAESIGFwyvuxqc319bArV9jVMV_3gZH3OV434JYu8niYpDRRGigKImh2cy5oRjlaWVhtYWtBN1U1QUV4cnpZV1Bjb2kuWDhXaDIQyIAM
token_accessor       Ri7jMehrYjAWvnN7EufsRX9O.X8Wh2
token_duration       5h56m42s
token_renewable      false
token_policies       ["default" "hcp-root"]
identity_policies    []
policies             ["default" "hcp-root"]

Tip

Vault server is ready and the token value is store in the $HOME/.vault-token file.

Enable Key/Value v2 secrets engine at secret/.

$ vault secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/

Create a working directory

Create a directory where you generate test files.
```
$ mkdir $HOME/vault-test && cd $HOME/vault-test
```
You will read the created secrets using Vault Agent.

Create test data

Create a mock data file.

$ tee data.json -<<EOF
{
   "organization": "ACME Inc.",
   "customer_id": "ABXX2398YZPIE7391",
   "region": "US-West",
   "zip_code": "94105",
   "type": "premium",
   "contact_email": "james@acme.com",
   "status": "active"
}
EOF

Create the mock data in the KV v2 secrets engine.

$ vault kv put secret/customers/acme @data.json

======= Secret Path =======
secret/data/customers/acme

======= Metadata =======
Key                Value
---                -----
created_time       2023-02-23T16:25:46.030960819Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Start a Vault Agent

Start a Vault Agent instance that connects to the Vault server running at VAULT_ADDR.

Create the Vault Agent configuration file.

$ tee agent-config.hcl -<<EOF
pid_file = "./pidfile"

vault {
   address = "$VAULT_ADDR"
   tls_skip_verify = true
}

auto_auth {
   method {
      type = "token_file"
      config = {
         token_file_path = "$HOME/.vault-token"
      }
   }
   sink "file" {
      config = {
            path = "$HOME/vault-token-via-agent"
      }
   }
}
EOF

Note

As of Vault 1.13.0, you can use the client token stored in a file (token_file_path). This is a convenient way to authenticate the Vault Agent with Vault server during development. For production deployment, use one of the supported auth methods.

Start a Vault Agent.

$ vault agent -config=agent-config.hcl

Example output:

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

                     Cgo: disabled
               Log Level:
               Version: Vault v1.13.0, built 2023-02-14T14:25:23Z
            Version Sha: c2d6b060c0eec60393e866b761a5990716cb45de

[INFO]  vault-agent.sink.file: creating file sink
[INFO]  vault-agent.sink.file: file sink configured: path=$HOME/vault-token-via-agent mode=-rw-r-----
[INFO]  vault-agent.template.server: starting template server
[INFO]  vault-agent.template.server: no templates found
[INFO]  vault-agent.auth.handler: starting auth handler
[INFO]  vault-agent.auth.handler: authenticating
[INFO]  vault-agent.sink.server: starting sink server
[INFO]  vault-agent.auth.handler: authentication successful, sending token to sinks
...snip...

Create the Vault Agent configuration file.

$ tee agent-config.hcl -<<EOF
pid_file = "./pidfile"

vault {
   address = "$VAULT_ADDR"
   tls_skip_verify = true
}

auto_auth {
   method {
      type = "token_file"
      namespace = "$VAULT_NAMESPACE"
      config = {
         token_file_path = "$HOME/.vault-token"
      }
   }
   sink "file" {
      config = {
            path = "$HOME/vault-token-via-agent"
      }
   }
}
EOF

Note

Start a Vault Agent.

$ vault agent -config=agent-config.hcl

Example output:

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

                     Cgo: disabled
               Log Level:
               Version: Vault v1.13.0, built 2023-02-14T14:25:23Z
            Version Sha: c2d6b060c0eec60393e866b761a5990716cb45de

[INFO]  vault-agent.sink.file: creating file sink
[INFO]  vault-agent.sink.file: file sink configured: path=$HOME/vault-token-via-agent mode=-rw-r-----
[INFO]  vault-agent.template.server: starting template server
[INFO]  vault-agent.template.server: no templates found
[INFO]  vault-agent.auth.handler: starting auth handler
[INFO]  vault-agent.auth.handler: authenticating
[INFO]  vault-agent.sink.server: starting sink server
...snip...

At this point, you still need to change your client application to make Vault requests targeting to VAULT_ADDR using the Vault token stored in the sink location ($HOME/vault-token-via-agent).

Next, explore other Vault Agent features.

Modify the Vault Agent configuration

Starting Vault 1.13.0, you can specify multiple Vault Agent configuration files.

$ vault agent -config=file-1.hcl -config=file-2.hcl

Or, point to a directory where configuration files are located.

$ vault agent -config=dicrectory/

In this section, you are going to define additional Vault Agent configurations.

Define Vault Agent template

In the following scenario, the client application loads data read from customer.json.

Press Ctrl + C to stop the running Vault Agent.

Create a customer.json.tmpl file.

$ tee customer.json.tmpl -<<EOF
{
   {{ with secret "secret/data/customers/acme" }}
   "Organization": "{{ .Data.data.organization }}",
   "ID": "{{ .Data.data.customer_id }}",
   "Contact": "{{ .Data.data.contact_email }}"
   {{ end }}
}
EOF

Create a Vault Agent configuration file with tempalte stanza.

$ tee agent-template.hcl -<<EOF
template {
   source      = "$HOME/vault-test/customer.json.tmpl"
   destination = "$HOME/vault-test/customer.json"
}
EOF

Start Vault Agent again with additional configuration.

$ vault agent -config=agent-config.hcl \
   -config=agent-template.hcl

Example output:

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

         Api Address 1: http://bufconn
                     Cgo: disabled
               Log Level:
               Version: Vault v1.13.0-rc1+ent, built 2023-02-14T14:25:23Z
            Version Sha: c2d6b060c0eec60393e866b761a5990716cb45de

[INFO]  vault-agent.sink.file: creating file sink
[INFO]  vault-agent.sink.file: file sink configured: path=$HOME/vault-token-via-agent mode=-rw-r-----
[INFO]  vault-agent.auth.handler: starting auth handler
[INFO]  vault-agent.auth.handler: authenticating
[INFO]  vault-agent.template.server: starting template server
...snip...
[INFO] (runner) starting
[INFO] (runner) rendered "$HOME/vault-test/customer.json.tmpl" => "$HOME/vault-test/customer.json"

Open a new browser or text editor and verify that the customer.json file has the secrets retrieved from Vault.

$ cat customer.json
{

   "Organization": "ACME Inc.",
   "ID": "ABXX2398YZPIE7391",
   "Contact": "james@acme.com"

}

This should match to the data stored in secret/.

$ vault kv get secret/customers/acme

======= Secret Path =======
secret/data/customers/acme

======= Metadata =======
Key                Value
---                -----
created_time       2023-02-23T16:25:46.030960819Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

======== Data ========
Key              Value
---              -----
contact_email    james@acme.com
customer_id      ABXX2398YZPIE7391
organization     ACME Inc.
region           US-West
status           active
type             premium
zip_code         94105

Troubleshooting for HCP Vault

If the template fails, make sure that the VAULT_NAMESPACE variable is set.

$ echo $VAULT_NAMESPACE
admin

The client application was designed to load data from the customer.json file. By leveraging the Vault Agent Template feature, you did not have to make any code change to the client application while your secrets are managed by Vault.

Add a listener

You can define an API endpoint that the Vault Agent listens to. The client application can send requests to the endpoint rather than VAULT_ADDR.

Press Ctrl + C to stop the running Vault Agent.
Create a file that defines the listener stanza and api_proxy stanza.
```
$ tee agent-listener-config.hcl -<<EOF
listener "tcp" {
   address     = "127.0.0.1:8100"
   tls_disable = true
}

listener "tcp" {
   address     = "127.0.0.1:3000"
   tls_disable = true
   role        = "metrics_only"
}

api_proxy {
   use_auto_auth_token = true
}
EOF
```
This configuration adds two Vault Agent API endpoints, 127.0.0.1:8100 and 127.0.0.1:3000. The second listener stanza defines a role set to metrics_only to use the API endpoint only to collect Vault Agent metrics. (See the Vault Agent documentation.)
The api_proxy stanza enables the use_auto_auth_token parameter. When enabled, Vault Agent will use the auto-auth token for proxied requests that do not carry their own token. Therefore, requests will use the auto-auth token without needing to provide the X-Vault-Token header.
Note
To leverage the metrics_only setting and the api_proxy stanza, you need Vault binary version 1.13.0 or later.

Start the Vault Agent again with additional configuration.

$ vault agent -config=agent-config.hcl \
   -config=agent-template.hcl \
   -config=agent-listener-config.hcl

Example output:

==> Vault Agent started! Log data will stream in below:

==> Vault Agent configuration:

         Api Address 1: http://127.0.0.1:8100
         Api Address 2: http://127.0.0.1:3000
         Api Address 3: http://bufconn
                     Cgo: disabled
               Log Level:
               Version: Vault v1.13.0-rc1+ent, built 2023-02-14T14:25:23Z
            Version Sha: c2d6b060c0eec60393e866b761a5990716cb45de

2023-02-22T23:07:12.663-0800 [INFO]  vault-agent.sink.file: creating file sink
...snip...

Now, Vault Agent listens to the API endpoints http://127.0.0.1:8100 and http://127.0.0.1:3000 (metrics only) and behaves as a Vault proxy.

Vault Agent logs

You can specify the behavior of Vault Agent logs using command flag, environment variable, or configuration parameters.

Start Vault Agent with the -log-file command flags which defines the prefix for the log file.
```
$ vault agent -config=agent-config.hcl \
   -log-file=$HOME/vault-test/vault-agent.log
```
The command generates a log file named $HOME/vault-test/vault-agent-{timestamp}.log.

Verify the generated log file.

$ ls | grep .log
vault-agent-1677285430042580000.log

You can open the file to verify its content.

$ cat vault-agent-1677285430042580000.log

[INFO]  vault-agent.sink.file: creating file sink
[INFO]  vault-agent.sink.file: file sink configured: path=/Users/yoko/vault-token-via-agent mode=-rw-r-----
[INFO]  vault-agent.template.server: starting template server
[INFO]  vault-agent.template.server: no templates found
[INFO]  vault-agent.auth.handler: starting auth handler
[INFO]  vault-agent.sink.server: starting sink server
[INFO]  vault-agent.auth.handler: authenticating
[INFO]  vault-agent.auth.handler: authentication successful, sending token to sinks
...snip...

Edit the Vault Agent configuration with the log_file parameter which defines the prefix for the log file.

$ tee agent-config.hcl -<<EOF
pid_file = "./pidfile"

log_file = "$HOME/vault-test/vault-agent.log"

vault {
   address = "$VAULT_ADDR"
   tls_skip_verify = true
}

auto_auth {
   method {
      type = "token_file"
      config = {
         token_file_path = "$HOME/.vault-token"
      }
   }
   sink "file" {
      config = {
            path = "$HOME/vault-token-via-agent"
      }
   }
}
EOF

Start a Vault Agent.
```
$ vault agent -config=agent-config.hcl
```
A log file named $HOME/vault-test/vault-agent-{timestamp}.log should be generated.

Verify the generated log file.

$ ls | grep .log
vault-agent-1677285430042580000.log

You can open the file to verify its content.

$ cat vault-agent-1677285430042580000.log

[INFO]  vault-agent.sink.file: creating file sink
[INFO]  vault-agent.sink.file: file sink configured: path=/Users/yoko/vault-token-via-agent mode=-rw-r-----
[INFO]  vault-agent.template.server: starting template server
[INFO]  vault-agent.template.server: no templates found
[INFO]  vault-agent.auth.handler: starting auth handler
[INFO]  vault-agent.sink.server: starting sink server
[INFO]  vault-agent.auth.handler: authenticating
[INFO]  vault-agent.auth.handler: authentication successful, sending token to sinks
...snip...

Tip

To see the full list of available parameters, refer to the Vault Agent documentation.

Clean up

Press Ctrl + C to stop the running Vault Agent.
You can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
```
$ pgrep -f vault | xargs kill
```
Unset the VAULT_ADDR environment variable.
```
$ unset VAULT_ADDR
```
Delete the $HOME/vault-test directory.
```
$ cd .. && rm -r $HOME/vault-test
```

Press Ctrl + C to stop the running Vault Agent.
Unset the VAULT_TOKEN, VAULT_NAMESPACE and VAULT_ADDR environment variables.
```
$ unset VAULT_TOKEN VAULT_ADDR VAULT_NAMESPACE
```
Delete the $HOME/vault-test directory.
```
$ cd .. && rm -r $HOME/vault-test
```

Summary

This tutorial demonstrated the basics of Vault Agent. You learned:

Minimum Vault Agent configuration
Used templates to read secrets from Vault
Define an API address that Vault Agent listens to

The use of Vault token file is an convenient way to quickly start the Vault Agent. For a production deployment, use one of the supported auth methods so that the Vault Agent can manage the lifecycle of the token properly.

See the following tutorials for auto auth examples:

The Vault Agent Template demonstrated that the client application can remain Vault-unaware while Vault manages the secrets.

See the following tutorials to see more template examples:

Help and reference

Back to Collection

5 tutorials

Vault 1.13 Release Highlights
The listed tutorials were updated to showcase the new enhancements introduced in Vault 1.13.
- Vault

What is Vault Agent?

Prerequisites

Lab setup

Create a working directory

Create test data

Start a Vault Agent

Modify the Vault Agent configuration

Define Vault Agent template

Add a listener

Vault Agent logs

Clean up

Summary

Help and reference

This tutorial also appears in: