Vault Agent Templates

10min
Terraform
Vault

The adoption of Vault is an incremental journey. First, you move your secrets into Vault so that they are securely encrypted and stored. The next step is to update your applications' behavior so that the secrets are read from Vault.

To establish a connection with Vault, clients must first authenticate with Vault and acquire a token. Vault Agent was introduced to reduce the burden from distributed applications to manage Vault client tokens.

The following guides demonstrate the Auto-Auth method of Vault Agent:

In addition, the Vault Agent Caching tutorial walked through the Caching feature introduced in Vault 1.1.

Challenge & Solution

After the successful authentication, Vault clients can start interacting with the Vault. Many Vault users adopted the Consul Template tool to minimize the level of changes introduced to their existing applications.

The Direct Application Integration tutorial demonstrates the use of Consul Template for Vault users.

In this case, Consul Template is the client directly interacting with Vault; therefore, you had to configure Consul Template to read the token from the agent's sink location in order to interact with Vault. This requires you to operate two distinct tools to provide secrets to applications.

In Vault 1.3, Vault Agent introduced Vault Agent Templates allowing Vault secrets to be rendered to files using the Consul Template markup language. This significantly simplifies the workflow when you are integrating your applications with Vault.

Note

This tutorial focuses on the templates feature of Vault Agent assuming that you are already familiar with Vault Agent Auto-Auth. If you are new to Vault Agent, try the Vault Agent with AWS tutorial or the Vault Agent with Kubernetes tutorial before continuing with this tutorial.

Prerequisites

To complete this section of the tutorial, you will need:

AWS account and associated credentials that allow for the creation of resources
Vault version 1.3 or later
Terraform installed

Provision the cloud resources

Clone the demo assets from the hashicorp/vault-guides GitHub repository to perform the steps described in this tutorial.
```
$ git clone https://github.com/hashicorp/vault-guides.git
```
This repository contains supporting content for all of the Vault learn tutorials. The content specific to this tutorial can be found within a sub-directory.

Be sure to set your working directory to where the /identity/vault-agent-templates/terraform-aws directory is located.

$ cd vault-guides/identity/vault-agent-templates/terraform-aws

The following assets are located under this directory:

$ tree
.
├── aws.tf
├── iam.tf
├── kms.tf
├── network.tf
├── outputs.tf
├── security-groups.tf
├── templates
│   ├── userdata-vault-client.tpl
│   └── userdata-vault-server.tpl
├── terraform.tfvars.example
├── variables.tf
├── vault-client.tf
├── vault-server.tf
└── versions.tf

NOTE: The example Terraform in this repository is created for the demo purpose.

Set an AWS_ACCESS_KEY_ID environment variable to hold your AWS access key ID.
```
$ export AWS_ACCESS_KEY_ID = "<YOUR_AWS_ACCESS_KEY_ID>"
```
Set an AWS_SECRET_ACCESS_KEY environment variable to hold your AWS secret access key.
```
$ export AWS_SECRET_ACCESS_KEY = "<YOUR_AWS_SECRET_ACCESS_KEY>"
```
Tip
The above example uses IAM user authentication. You can use any authentication method described in the AWS provider documentation.
Create a file named terraform.tfvars and specify the key_name. (You can copy the terraform.tfvars.example file and rename it as terraform.tfvars.)
Example terraform.tfvars:
```
# SSH key name to access EC2 instances (should already exist) on the AWS region
key_name = "my-key-pair"
```
If you don't have an EC2 Key Pairs, refer to the AWS documentation and create one.
Perform a terraform init to pull down the necessary provider resources.
```
$ terraform init
```
Perform terraform plan to verify your changes and the resources that will be created.
```
$ terraform plan
```

If all looks good, perform a terraform apply to provision the resources.

$ terraform apply -auto-approve
...
Apply complete! Resources: 20 added, 0 changed, 0 destroyed.

Outputs:

endpoints =
Vault Server IP (public):  192.0.2.224
Vault Server IP (private): 10.0.101.62

For example:
  ssh -i vault-test.pem ubuntu@192.0.2.224

Vault Client IP (public):  198.51.100.24
Vault Client IP (private): 10.0.101.10

For example:
  ssh -i vault-test.pem ubuntu@198.51.100.24

The Terraform output will display the public IP address to SSH into your Vault server and client instances.

Configure AWS IAM auth method

This step should be performed on the Vault Server instance.

In this step, you will configure Vault to allow AWS IAM authentication from specific IAM roles.

SSH into the Vault Server instance.
When you are prompted, enter "yes" to continue.
```
$ ssh -i <path_to_key> ubuntu@<public_ip_of_server>
...
Are you sure you want to continue connecting (yes/no)? yes
```
NOTE: If you received Permissions 0664 for '<key_name>.pem' are too open error, be sure to set the file permission appropriately.
```
$ chmod 600 <key_name>.pem
```
Run the vault operator init command to initialize the Vault server. For the convenience of this tutorial, save the generated recovery keys and an initial root token in a file named, key.txt.
```
$ vault operator init > key.txt
```
The Vault server is configured to auto-unseal with AWS Key Management Service (KMS); therefore, once the server is initialized, it gets automatically unsealed. To learn how to auto-unseal Vault using AWS KMS, refer to the Auto-unseal using AWS KMS tutorial.

Log into Vault using the generated initial root token save in the key.txt

$ vault login $(grep 'Initial Root Token:' key.txt | awk '{print $NF}')

Examine the /home/ubuntu/aws_auth.sh script.
```
$ cat aws_auth.sh
```

Execute the /home/ubuntu/aws_auth.sh script.

$ ./aws_auth.sh

Success! Enabled the kv-v2 secrets engine at: secret/
Key              Value
---              -----
created_time     2019-12-04T00:16:07.107076447Z
deletion_time    n/a
destroyed        false
version          1
Success! Uploaded policy: app-pol
Success! Enabled aws auth method at: aws/
Success! Data written to: auth/aws/config/client
Success! Data written to: auth/aws/role/app-role

The script enabled key/value v2 secrets engine at secret and wrote some data at secret/customers/acme. It also created an app-pol policy, enabled and configured aws auth method, and created a role named app-role.

View the app-pol policy.

$ vault policy read app-pol

path "secret/data/*" {
  capabilities = ["read"]
}
path "auth/token/*" {
  capabilities = ["create", "update"]
}

View the secrets written at secret/customers/acme

$ vault kv get secret/customers/acme
====== Metadata ======
Key              Value
---              -----
created_time     2019-12-04T00:16:07.107076447Z
deletion_time    n/a
destroyed        false
version          1

======== Data ========
Key              Value
---              -----
contact_email    james@acme.com
customer_id      ABXX2398YZPIE7391
organization     ACME Inc.
region           US-West
status           active
type             premium
zip_code         94105

Start Vault Agent

This step should be performed on the Vault Client instance.

In the client node, Vault Agent authenticates with Vault via aws auth method you configured in the configure AWS IAM auth method step. Using the acquired token, the agent pulls secrets from Vault defined by the template file.

Now, SSH into the Vault Client instance.

Output

When you are prompted, enter "yes" to continue.

$ ssh -i <path_to_key> ubuntu@<public_ip_of_client>
...
Are you sure you want to continue connecting (yes/no)? yes

Explore the Vault Agent configuration file, /home/ubuntu/vault-agent.hcl.

$ cat vault-agent.hcl

The contents of the file looks as below.

pid_file = "./pidfile"

auto_auth {
  method "aws" {
    mount_path = "auth/aws"
    config = {
      type = "iam"
      role = "app-role"
    }
  }

  sink "file" {
    config = {
      path = "/home/ubuntu/vault-token-via-agent"
    }
  }
}

vault {
  address = "http://10.0.101.62:8200"
}

template {
  source      = "/home/ubuntu/customer.tmpl"
  destination = "/home/ubuntu/customer.txt"
}

The Vault Agent uses the aws auth method to authenticate with the Vault server running on the server instance as app-role. Also notice that there is template block which sets /home/ubuntu/customer.tmpl as the source template file and the output will be written to the destination, /home/ubuntu/customer.txt.

Execute the following command to start the Vault Agent with log level set to debug.

$ vault agent -config=/home/ubuntu/vault-agent.hcl -log-level=debug

You should find the following:

[INFO]  sink.file: creating file sink
[INFO]  sink.file: file sink configured: path=/home/ubuntu/vault-token-via-agent mode=-rw-r-----
[INFO]  auth.handler: starting auth handler
[INFO]  auth.handler: authenticating
[INFO]  template.server: starting template server
...
[INFO]  auth.handler: authentication successful, sending token to sinks
...
[DEBUG] (runner) running initial templates
[DEBUG] (runner) initiating run
[DEBUG] (runner) checking template 56ba7f3e29857b85850ab5e0eb1151a3
...
[DEBUG] (runner) receiving dependency vault.read(secret/data/customers/acme)
[DEBUG] (runner) initiating run
[DEBUG] (runner) checking template 56ba7f3e29857b85850ab5e0eb1151a3
[DEBUG] (runner) rendering "/home/ubuntu/customer.tmpl" => "/home/ubuntu/customer.txt"

Open a second SSH terminal into the client machine.
Verify the client token stored in /home/ubuntu/vault-token-via-agent.
```
$ more vault-token-via-agent
s.4G9iuH8MtZqrvYtOmWeaGqnJ
```

Check the details about the token (e.g. attached policies, TTL, etc.).

$ VAULT_TOKEN="$(cat vault-token-via-agent)" vault token lookup
Key                  Value
---                  -----
...
orphan               true
path                 auth/aws/login
policies             [app-pol default]
renewable            true
ttl                  22h22m17s
type                 service

The app-pol policy is attached to the token. Remember that the app-pol policy grants read operation against the secret/data/* path.

Review the customer.tmpl file which is written using the Consul Templates markup language.

$ cat customer.tmpl

{{ with secret "secret/data/customers/acme" }}
Organization: {{ .Data.data.organization }}
ID: {{ .Data.data.customer_id }}
Contact: {{ .Data.data.contact_email }}
{{ end }}

Notice that the secret path is set to secret/data/customers/acme.

Examine the resulting customer.txt file.
```
$ cat customer.txt

Organization: ACME Inc.
ID: ABXX2398YZPIE7391
Contact: james@acme.com
```
Based on the customer.tmpl file, Vault Agent Templates read the secrets at secret/data/customers/acme and then generated the customer.txt file.

Challenge

What happens if the data at secret/data/customers/acme was updated?

On the server instance terminal, update the contact_email to jenn@acme.com.
```
$ vault kv patch secret/customers/acme contact_email="jenn@acme.com"
```
Verify that the data was successfully updated.
```
$ vault kv get secret/customers/acme
```
Return to the client instance SSH session where the Vault Agent is running. Wait for a few minutes (about 4 minutes). The agent pulls the secrets again.
```
...
[DEBUG] (runner) checking template 494ea5cbe4765bfe2e5eca2363eff06b
[DEBUG] (runner) rendering "./customer.tmpl" => "./customer.txt"
[INFO] (runner) rendered "./customer.tmpl" => "./customer.txt"
[DEBUG] (runner) diffing and updating dependencies
[DEBUG] (runner) vault.read(secret/data/customers/acme) is still needed
[DEBUG] (runner) watching 1 dependencies
[DEBUG] (runner) all templates rendered
...
```
When reading secrets from Key/Value v2 secrets engine, omitting the ?version parameter reads the latest version of the secrets at the path. If you want to always pull a specific version of the secret, specify the desired version number. For example, {{ with secret "secret/data/customers/acme?version=3" }} will always read the version 3 of the secret/data/customers/acme values.

Clean up

Execute the following commands to destroy cloud resources.

$ terraform apply -destroy -auto-approve

At this point, you can delete the Terraform state files from the directory.

$ rm -rf .terraform terraform.tfstate*

Summary and reference

Vault Agent is a client daemon that solves the secret-zero problem by authenticating with Vault and manage the client tokens on behalf of the client applications. The Consul Template tool is widely adopted by the Vault users since it allowed applications to be "Vault-unaware".

Vault Agent Templates combines the best of the two tools to make the end-to-end workflow even simpler.