Use Consul Template and Envconsul with Vault

16min
Vault
Video

A modern system requires access to a multitude of secrets: database credentials, API keys for external services, credentials for service-oriented architecture communication, etc. Vault steps in to provide a centralized secret management system. The next step is to decide how your applications acquire the secrets from Vault.

This tutorial introduces Consul Template and Envconsul to help you determine if these tools speed up the integration of your applications once secrets are securely managed by Vault.

Note

Both Consul Template and Envconsul are open source tools.

Consul Template

Despite its name, Consul Template does not require a Consul cluster to operate. It retrieves secrets from Vault and manages the acquisition and renewal lifecycle.

Envconsul

Envconsul launches a subprocess which dynamically populates environment variables from secrets read from Vault. Your applications then read those environment variables. Despite its name, Envconsul does not require a Consul cluster to operate. It enables flexibility and portability for applications across systems.

Challenge

If your application code or script contains some secrets (e.g. database credentials), it makes a good sense to manage the secrets using Vault. However, it means that your application will need to retrieve the secrets at runtime. Does that mean the application developers must make some code change?

Is there an easy way to retrieve the secrets from Vault and populate the application code or script with secrets as needed?

Solution

Both Consul Template and Envconsul provide first-class support for Vault. Leveraging these tools can minimize the level of changes introduced to your applications. Depending on the current application design, you may not need to make minimal to no code change.

Prerequisites

This lab was tested on macOS using an x86_64 based processor. If you are running macOS on an Apple silicon-based processor, use a x86_64 based Linux virtual machine in your preferred cloud provider.

To perform the tasks described in this tutorial, you need:

HCP or OSS Vault environment
Consul Template - review the installation documentation for details on how to install this prerequisite.
Envconsul - review the installation documentation for details on how to install this prerequisite.
PostgreSQL or Docker to run a PostgreSQL container
jq
ngrok installed and configured with an auth token (HCP Vault only)

PostgreSQL

This tutorial uses the database secrets engine to demonstrate the use of Consul Template and Envconsul. Therefore you need a PostgreSQL server to connect to.

Complete the Secret as a Service: Dynamic Secrets tutorial first if you are not familiar with database secrets engine.

Policy requirements

Note

For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Enable database secrets engines at "database/" path
path "sys/mounts/database" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Configure the database secrets engine and create roles
path "database/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Write ACL policies
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage tokens for Consul Template & Envconsul to use
path "auth/token/create" {
  capabilities = [ "create", "read", "update", "delete", "list", "sudo" ]
}

If you are not familiar with policies, complete the policies tutorial.

Lab setup

Start PostgreSQL

The tutorial requires a Postgres database. Docker provides a Postgres server image that satisfies this requirement.

Open a new terminal and pull a Postgres server image with docker.
```
$ docker pull postgres:latest
```

Create a Postgres database with a root user named root with the password rootpassword.

$ docker run \
      --name postgres \
      --env POSTGRES_USER=root \
      --env POSTGRES_PASSWORD=rootpassword \
      --detach  \
      --publish 5432:5432 \
      postgres

The database is available.

Connect to the Postgres database via the CLI within the postgres container.
```
$ docker exec -it postgres psql
```
Your system prompt is replaced with a new prompt root=#. Commands issued at this prompt are executed against the Postgres database running within the container.

Create a role named readonly.

$ CREATE ROLE readonly NOINHERIT;
CREATE ROLE

Grant the ability to read all tables to the role named readonly.
```
$ GRANT SELECT ON ALL TABLES IN SCHEMA public TO "readonly";
GRANT
```
The role is created and assigned the appropriate permissions.

Create a database named myapp.

$ CREATE DATABASE myapp;
CREATE DATABASE

Disconnect from the Postgres database.
```
$ \q
```

Start Vault

In a new terminal start a Vault dev server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to running at 127.0.0.1:8200. The server is initialized and unsealed.
Insecure operation
Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.
In a new terminal export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```
The Vault server is ready.
Note
For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.
Set an environment variable for the PostreSQL address.
```
$ export POSTGRES_URL=127.0.0.1:5432
```

You are ready to proceed with the lab.

Note

If you do not have access to an HCP Vault cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```

Type vault status to verify your connectivity to the Vault cluster.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.9.2+ent
Storage Type             raft
...snipped...

The HCP Vault server is ready.

For HCP Vault to interact with resources running on your local machine, a tunnel needs to be established.

In another terminal, start ngrok and connect to PostgreSQL.

$ ngrok tcp 127.0.0.1:5432

Example output:

ngrok                                                                                                                                                              (Ctrl+C to quit)

Session Status                online
Account                       username (Plan: Free)
Update                        update available (version 3.0.5, Ctrl-U to update)
Version                       3.0.3
Region                        United States (us)
Latency                       32.791235ms
Web Interface                 http://127.0.0.1:4040
Forwarding                    tcp://d12b-34-567-89-10.ngrok.io:12345 -> 127.0.0.1:5432

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

Copy the ngrok forwarding address.
Return to the terminal where you set the VAULT_ADDR environment variable and set an environment variable for the ngrok address. Do not include tcp://.
```
$ export POSTGRES_URL=<actual-address-from-ngrok>
```

You are ready to proceed with the lab.

Step 1: Setup Database Secrets Engine

In this step, you are going to enable and configure the database secrets engine using postgresql-database-plugin where the database connection URL is in the format of protocol://username:password@hostname:port/database.

Note

If you are using your own database server, be sure to update the username, password, and URL to use the correct connection URL to match your environment.

Refer to the PostgreSQL Database Secrets Engine documentation or Secret as a Service: Dynamic Secrets tutorial if you are not familiar with database secrets engine. The detailed description of working with database secrets engine is out of scope for this tutorial.

First, enable the database secrets engine.
```
$ vault secrets enable database
```

Configure the secrets engine with appropriate parameter values.

$ vault write database/config/postgresql \
      plugin_name=postgresql-database-plugin \
      allowed_roles=readonly \
      connection_url="postgresql://{{username}}:{{password}}@$POSTGRES_URL/myapp?sslmode=disable" \
      username=root \
      password=rootpassword

Create readonly.sql to define a role permission in SQL.

$ tee readonly.sql <<EOF
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";
EOF

Create a role, "readonly".

$ vault write database/roles/readonly db_name=postgresql \
        creation_statements=@readonly.sql \
        default_ttl=1h max_ttl=24h

Enable database secrets engine using /sys/mounts endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"type":"database"}' \
      $VAULT_ADDR/v1/sys/mounts/database

Create an API request payload specifying the database connection URL according to your environment.

$ tee payload.json <<EOF
{
  "plugin_name": "postgresql-database-plugin",
  "allowed_roles": "readonly",
  "connection_url": "postgresql://{{username}}:{{password}}@$POSTGRES_URL/myapp?sslmode=disable",
  "username": "root",
  "password": "rootpassword"
}
EOF

Configure the database secrets engine by passing the request payload.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload.json \
      $VAULT_ADDR/v1/database/config/postgresql

Create an API request payload containing a role definition.

$ tee payload.json <<EOF
{
  "db_name": "postgresql",
  "creation_statements": ["CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
  GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"],
  "default_ttl": "1h",
  "max_ttl": "24h"
}
EOF

Create a role named readonly.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload.json \
      $VAULT_ADDR/v1/database/roles/readonly

Enable database secrets engine using /sys/mounts endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{"type":"database"}' \
   $VAULT_ADDR/v1/sys/mounts/database

Create an API request payload specifying the database connection URL according to your environment.

$ tee payload.json <<EOF
{
  "plugin_name": "postgresql-database-plugin",
  "allowed_roles": "readonly",
  "connection_url": "postgresql://{{username}}:{{password}}@$POSTGRES_URL/myapp?sslmode=disable",
  "username": "root",
  "password": "rootpassword"
}
EOF

Configure the database secrets engine by passing the request payload.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/database/config/postgresql

Create an API request payload containing a role definition.

$ tee payload.json <<EOF
{
  "db_name": "postgresql",
  "creation_statements": ["CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
  GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"],
  "default_ttl": "1h",
  "max_ttl": "24h"
}
EOF

Create a role named readonly.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/database/roles/readonly

Open a web browser and launch the Vault UI and then login.
Navigate to Secrets and click Enable new engine.
From the Enable a Secrets Engine page Infra section, select Databases and click Next.
Leave all fields as-is, and click Enable Engine.
Note
This tutorial assumes that you enabled the database secrets engine at database. If yours is enabled at a different path, be sure to use the correct path as you follow this tutorial.
The database secrets engine is enabled, but there is not yet an available connection from Vault to the database. Create the connection from Vault to the PostgreSQL server by clicking Create connection.
For Database plugin, select PostgreSQL.
For Connection name, enter postgresql.
For Connection url, enter postgresql://{{username}}:{{password}}@localhost:5432/postgres?sslmode=disable.
Uncheck Connection will be verified.
For Username, enter root.
For Password, enter rootpassword.
Click Create database.
When prompted to rotate the root credentials, click Rotate and enable.

Now configure a dynamic role, "readonly".

On the database page for your PostgreSQL connection, click Add role.
For Role name, enter readonly.
For Database name, choose postgresql.
For Type of role, select dynamic.
For Username, enter root.

For Creation statements, enter the following SQL statements:

CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';
GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}";

Role settings part one

Click Create role.

Step 2: Generate Client Token

Consul Template tool itself is a Vault client. Therefore it must have a valid token with policies permitting it to retrieve secrets from database secret engine you just configured in Step 1.

Create a policy definition file, db_creds.hcl.
```
$ tee db_creds.hcl <<EOF
path "database/creds/readonly" {
  capabilities = [ "read" ]
}

path "sys/leases/renew" {
  capabilities = [ "update" ]
}
EOF
```
This policy allows read operation on the database/creds/readonly path to obtain the dynamically generated username and password to access the PostgreSQL database. In addition, the policy allows renewal of the lease if necessary.

Create a new policy named db_creds.

$ vault policy write db_creds db_creds.hcl

Create a token with db_creds policy attached and save it as an environment variable.

$ DB_TOKEN=$(vault token create -policy="db_creds" -format json | jq -r '.auth | .client_token')

Create an API request payload containing the stringified db_creds policy.

$ tee payload.json <<EOF
{
  "policy": "path \"database/creds/readonly\" {\n capabilities = [ \"read\" ]\n } \n path \"sys/leases/renew\" {\n capabilities = [ \"update\" ] \n}"
}
EOF

Create a new policy named db_creds.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request PUT \
      --data @payload.json \
      $VAULT_ADDR/v1/sys/policies/acl/db_creds

Generate a new token with db_creds policy attached and save it as an environment variable.

$ DB_TOKEN=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data '{"policies": ["db_creds"]}' \
      $VAULT_ADDR/v1/auth/token/create | jq -r '.auth | .client_token')

Create an API request payload containing the stringified db_creds policy.

$ tee payload.json <<EOF
{
  "policy": "path \"database/creds/readonly\" {\n capabilities = [ \"read\" ]\n } \n path \"sys/leases/renew\" {\n capabilities = [ \"update\" ] \n}"
}
EOF

Create a new policy named db_creds.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
      --request PUT \
      --data @payload.json \
      $VAULT_ADDR/v1/sys/policies/acl/db_creds

Generate a new token with db_creds policy attached and save it as an environment variable.

$ DB_TOKEN=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{"policies": ["db_creds"]}' \
   $VAULT_ADDR/v1/auth/token/create | jq -r '.auth | .client_token')

Click the Policies tab, and then select Create ACL policy.
Toggle Upload file sliding switch, and click Choose a file to select your db_creds.hcl file you authored. This loads the policy and sets the Name to be db_creds.
Click Create Policy to complete.
Click the Vault CLI shell icon (>_) to open a command shell. Execute vault write auth/token/create policies=db_creds in the CLI shell to create a new token for Consul Template.

Note

This is the token that Consul Template uses to talk to Vault. Copy the client_token value and proceed to Step 3.

Step 3: Use Consul Template to Populate DB Credentials

Assume that your application requires PostgreSQL database credentials to read data. Its configuration file, config.yml looks like:

username: '<DB_USERNAME>'
password: '<DB_PASSWORD>'
database: 'myapp'

To have Consul Template to populate the <DB_USERNAME> and <DB_PASSWORD>, you need to create a template file with Consul Template templating language.

Create a template file by replacing the username and password with Consul Template syntax and save it as config.yml.tpl. The file should contain the following:
```
$ tee config.yml.tpl <<EOF
---
{{- with secret "database/creds/readonly" }}
username: "{{ .Data.username }}"
password: "{{ .Data.password }}"
database: "myapp"
{{- end }}
EOF
```
NOTE: This template reads secrets from database/creds/readonly path in Vault. Set the username parameter value to ".Data.username" of the secret output. Similarly, set the password to ".Data.password" value.
Execute the consul-template command to populate config.yml file. The Consul Template command is: consul-template -template="<input_file>:<output_file>"
The input file is the config.yml.tpl and specify the desired output file name to be config.yml:
```
$ VAULT_TOKEN=$DB_TOKEN consul-template \
        -template="config.yml.tpl:config.yml" -once
```
Open the generated config.yml file to verify its content. It should look similar to:
```
$ cat config.yml
---
username: "v-token-readonly-tu17xrtz345uz643980r-1527630039"
password: "A1a-7s0z9y223x2rp6v9"
database: "myapp"
```
The username and password were retrieved from Vault and populated in the config.yml file.
Summary: You need to create a templated version of your application scripts to leverage Consul Template. However, it requires minimum effort to do so in comparison to writing an application which invokes Vault API to accomplish the same.

Step 4: Use Envconsul to Retrieve DB Credentials

Create a file named app.sh containing the following:
```
$ tee app.sh <<EOF
#!/usr/bin/env bash

cat <<EOT
My connection info is:

username: "\${DATABASE_CREDS_READONLY_USERNAME}"
password: "\${DATABASE_CREDS_READONLY_PASSWORD}"
database: "my-app"
EOT
EOF
chmod +x app.sh
```
The main difference here is that the app.sh is reading environment variables to set username and password values; therefore no templating is involved.
Notice that the environment variable name is derived from the secret path with key name.
Run the Envconsul tool using the Vault token you generated at Step 2.
```
$ VAULT_TOKEN=$DB_TOKEN envconsul -upcase -secret database/creds/readonly ./app.sh

My connection info is:

username: "v-token-readonly-ww1tq33s7z5uprpxxy68-1527631219"
password: "A1a-u54wut0v605qwz95"
database: "my-app"
```
The output should display the username and password populated.
The -upcase flag tells Envconsul to convert all environment variable keys to uppercase. Otherwise, the default uses lowercase (e.g. database_creds_readonly_username).
Summary
If your application is designed to read secrets from environment variables, Envconsul requires minimal to no code change to integrate with Vault.

Consul Template

Envconsul

Challenge

Solution

Prerequisites

PostgreSQL

Policy requirements

Lab setup

Start PostgreSQL

Start Vault

Step 1: Setup Database Secrets Engine

Step 2: Generate Client Token

Step 3: Use Consul Template to Populate DB Credentials

Step 4: Use Envconsul to Retrieve DB Credentials

Help and Reference