Use Consul Template and Envconsul with Vault
A modern system requires access to a multitude of secrets: database credentials, API keys for external services, credentials for service-oriented architecture communication, etc. Vault steps in to provide a centralized secret management system. The next step is to decide how your applications acquire the secrets from Vault.
This tutorial introduces Consul Template and Envconsul to help you determine if these tools speed up the integration of your applications once secrets are securely managed by Vault.
Note
Both Consul Template and Envconsul are open source tools.
Consul Template
Despite its name, Consul Template does not require a Consul cluster to operate. It retrieves secrets from Vault and manages the acquisition and renewal lifecycle.
Envconsul
Envconsul launches a subprocess which dynamically populates environment variables from secrets read from Vault. Your applications then read those environment variables. Despite its name, Envconsul does not require a Consul cluster to operate. It enables flexibility and portability for applications across systems.
Challenge
If your application code or script contains some secrets (e.g. database credentials), it makes a good sense to manage the secrets using Vault. However, it means that your application will need to retrieve the secrets at runtime. Does that mean the application developers must make some code change?
Is there an easy way to retrieve the secrets from Vault and populate the application code or script with secrets as needed?
Solution
Both Consul Template and Envconsul provide first-class support for Vault. Leveraging these tools can minimize the level of changes introduced to your applications. Depending on the current application design, you may not need to make minimal to no code change.
Prerequisites
This lab was tested on macOS using an x86_64 based processor. If you are running macOS on an Apple silicon-based processor, use a x86_64 based Linux virtual machine in your preferred cloud provider.
To perform the tasks described in this tutorial, you need:
- HCP or OSS Vault environment
- Consul Template - review the installation documentation for details on how to install this prerequisite.
- Envconsul - review the installation documentation for details on how to install this prerequisite.
- PostgreSQL or Docker to run a PostgreSQL container
- jq
- ngrok installed and configured with an auth token (HCP Vault only)
PostgreSQL
This tutorial uses the database secrets engine to demonstrate the use of Consul Template and Envconsul. Therefore you need a PostgreSQL server to connect to.
Complete the Secret as a Service: Dynamic
Secrets tutorial first if you
are not familiar with database
secrets engine.
Policy requirements
Note
For the purpose of this tutorial, you can use root
token to work
with Vault. However, it is recommended that root tokens are only used for just
enough initial setup or in emergencies. As a best practice, use tokens with
appropriate set of policies based on your role in the organization.
To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:
If you are not familiar with policies, complete the policies tutorial.
Lab setup
Start PostgreSQL
The tutorial requires a Postgres database. Docker provides a Postgres server image that satisfies this requirement.
Open a new terminal and pull a Postgres server image with
docker
.Create a Postgres database with a root user named
root
with the passwordrootpassword
.The database is available.
Connect to the Postgres database via the CLI within the
postgres
container.Your system prompt is replaced with a new prompt
root=#
. Commands issued at this prompt are executed against the Postgres database running within the container.Create a role named
readonly
.Grant the ability to read all tables to the role named
readonly
.The role is created and assigned the appropriate permissions.
Create a database named
myapp
.Disconnect from the Postgres database.
Start Vault
In a new terminal start a Vault dev server with
root
as the root token.The Vault dev server defaults to running at
127.0.0.1:8200
. The server is initialized and unsealed.Insecure operation
Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.
In a new terminal export an environment variable for the
vault
CLI to address the Vault server.Export an environment variable for the
vault
CLI to authenticate with the Vault server.The Vault server is ready.
Note
For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.
Set an environment variable for the PostreSQL address.
You are ready to proceed with the lab.
Step 1: Setup Database Secrets Engine
In this step, you are going to enable and configure the database
secrets
engine using postgresql-database-plugin
where the database connection URL is
in the format of protocol://username:password@hostname:port/database
.
Note
If you are using your own database server, be sure to update the username, password, and URL to use the correct connection URL to match your environment.
Refer to the PostgreSQL Database Secrets
Engine
documentation or Secret as a Service: Dynamic
Secrets tutorial if you are not
familiar with database
secrets engine. The detailed description of working
with database
secrets engine is out of scope for this tutorial.
First, enable the database secrets engine.
Configure the secrets engine with appropriate parameter values.
Create
readonly.sql
to define a role permission in SQL.Create a role, "readonly".
Step 2: Generate Client Token
Consul Template tool itself is a Vault client. Therefore it must have a valid
token with policies permitting it to retrieve secrets from database
secret
engine you just configured in Step 1.
Create a policy definition file,
db_creds.hcl
.This policy allows read operation on the
database/creds/readonly
path to obtain the dynamically generated username and password to access the PostgreSQL database. In addition, the policy allows renewal of the lease if necessary.Create a new policy named
db_creds
.Create a token with
db_creds
policy attached and save it as an environment variable.
Step 3: Use Consul Template to Populate DB Credentials
Assume that your application requires PostgreSQL database credentials to read
data. Its configuration file, config.yml
looks like:
To have Consul Template to populate the <DB_USERNAME>
and <DB_PASSWORD>
, you
need to create a template file with Consul Template templating
language.
Create a template file by replacing the username and password with Consul Template syntax and save it as
config.yml.tpl
. The file should contain the following:NOTE: This template reads secrets from
database/creds/readonly
path in Vault. Set theusername
parameter value to ".Data.username
" of the secret output. Similarly, set thepassword
to ".Data.password
" value.Execute the
consul-template
command to populateconfig.yml
file. The Consul Template command is:consul-template -template="<input_file>:<output_file>"
The input file is the
config.yml.tpl
and specify the desired output file name to beconfig.yml
:Open the generated
config.yml
file to verify its content. It should look similar to:The
username
andpassword
were retrieved from Vault and populated in theconfig.yml
file.Summary: You need to create a templated version of your application scripts to leverage Consul Template. However, it requires minimum effort to do so in comparison to writing an application which invokes Vault API to accomplish the same.
Step 4: Use Envconsul to Retrieve DB Credentials
Create a file named
app.sh
containing the following:The main difference here is that the
app.sh
is reading environment variables to setusername
andpassword
values; therefore no templating is involved.Notice that the environment variable name is derived from the secret path with key name.
Run the Envconsul tool using the Vault token you generated at Step 2.
The output should display the
username
andpassword
populated.The
-upcase
flag tells Envconsul to convert all environment variable keys to uppercase. Otherwise, the default uses lowercase (e.g.database_creds_readonly_username
).Summary
If your application is designed to read secrets from environment variables, Envconsul requires minimal to no code change to integrate with Vault.