Static secrets: Key/value secrets engine

18min
|
Vault
Interactive

Deployment

Vault can be used to store any secret in a secure manner. The secrets may be SSL certificates and keys for your organization's domain, credentials to connect to a corporate database server, etc. Storing such sensitive information in plaintext is not desirable.

Note

This tutorial demonstrates the use of Key/Value Secrets Engine Version 1. To learn about Key/Value Secrets Engine Version 2, refer to the Versioned Key/Value Secrets Engine tutorial.

Personas

The end-to-end scenario described in this tutorial involves two personas:

devops with privileged permissions to write secrets
apps reads the secrets from Vault

Challenge

Consider the following situations:

Developers use a single admin account to access a third-party app (e.g. Splunk) and anyone who knows the user ID and password can log in as an admin
SSH keys to connect to remote machines are shared and stored as a plaintext
API keys to invoke external system APIs are stored as a plaintext
An app integrates with LDAP, and its configuration information is in a plaintext

Organizations often seek an uniform workflow to securely store this sensitive information.

Solution

Use Vault as centralized secret storage to secure any sensitive information. Vault encrypts these secrets using 256-bit AES in GCM mode with a randomly generated nonce prior to writing them to its persistent storage. The storage backend never sees the unencrypted value, so even if an attacker gained access to the raw storage, they wouldn't be able to read your secrets.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Prerequisites

To perform the tasks described in this tutorial, you need to have:

Vault binary installed. Refer to the Getting Started tutorial.
jq to format the JSON output to follow the Vault API steps.

Policy requirements

Note

For the purpose of this tutorial, you can use the root token to work with Vault. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Enable key/value secrets engine at the kv-v1 path
path "sys/mounts/kv-v1" {
  capabilities = [ "update" ]
}

# To list the available secrets engines
path "sys/mounts" {
  capabilities = [ "read" ]
}

# Write and manage secrets in key/value secrets engine
path "kv-v1/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Create policies to permit apps to read secrets
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Create tokens for verification & test
path "auth/token/create" {
  capabilities = [ "create", "update", "sudo" ]
}

If you are not familiar with policies, complete the policies tutorial.

Scenario Introduction

This tutorial demonstrates the basic steps to store secrets using Vault. The scenario here is to store the following secrets:

API key (Google API)
Root certificate of a production database (MySQL)

To store your API key within the configured physical storage for Vault, use the key/value secrets engine.

The Key/Value secrets engine passes any operation through to the configured storage backend for Vault. For example, if your Vault server is configured with Consul as its storage backend, a "read" operation turns into a read from Consul at the same path.

You will perform the following:

Lab Setup

In another terminal, start a Vault dev server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.
Insecure operation
Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```

The Vault server is ready.

Enable KV Secrets Engine

Currently, when you start the Vault server in dev mode, it automatically enables v2 of the KV secrets engine at secret/.

This tutorial focuses on key/value v1 secrets engine. The Versioned Key/Value Secret Engine tutorial highlights features that are specific to the key/value v2 secrets engine.

Enable the key/value secrets engine v1 at kv-v1/.

$ vault secrets enable -path="kv-v1" -description="Test K/V v1" kv
Success! Enabled the kv secrets engine at: kv-v1/

List enabled secrets engines.

$ vault secrets list

You should see kv-v1/ listed.

Path          Type            Accessor                 Description
----          ----            --------                 -----------
cubbyhole/    ns_cubbyhole    ns_cubbyhole_445769e6    per-token private secret storage
identity/     ns_identity     ns_identity_f425b741     identity store
kv-v1/        kv              kv_401a50ee              Test K/V v1
sys/          ns_system       ns_system_875996d3       system endpoints used for control, policy and debugging

Enable the key/value secrets engine v1 at kv-v1/.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{ "type": "kv", "description": "Test K/V v1" }' \
    $VAULT_ADDR/v1/sys/mounts/kv-v1

To check the KV secrets engine version, use the sys/mounts endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    $VAULT_ADDR/v1/sys/mounts | jq -r ".data"

Note

The results of the command are sent to jq to format the JSON output for readability.

Example output:

...
  "kv-v1/": {
    "accessor": "kv_fef7dcb8",
    "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0
    },
    "description": "Test K/V v1",
    "external_entropy_access": false,
    "local": false,
    "options": {},
    "seal_wrap": false,
    "type": "kv",
    "uuid": "bac44cf4-12d9-19c2-1f48-768bbfa4dae7"
  },
...

Store the Google API key

Everything after the kv-v1 path is a key-value pair to write to the secrets engine. You can specify multiple values. If the value has a space, you need to surround it with quotes. Having keys with spaces is permitted, but strongly discouraged because it can lead to unexpected client-side behavior.

Let's assume that the path convention in your organization is kv-v1/<OWNER>/apikey/<APP> for API keys. To store the Google API key used by the engineering team, the path would be kv-v1/eng/apikey/Google. If you have an API key for New Relic owned by the DevOps team, the path would look like kv-v1/devops/apikey/New_Relic.

To set key/value secrets:

$ vault kv put kv-v1/<PATH> <KEY>=VALUE>

The <PATH> can be anything you want it to be, and your organization should decide on the naming convention that makes most sense.

Create a secret at path kv-v1/eng/apikey/Google with a key set to AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI.

$ vault kv put kv-v1/eng/apikey/Google key=AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI
Success! Data written to: kv-v1/eng/apikey/Google

Read back the secret at path kv-v1/eng/apikey/Google

$ vault kv get kv-v1/eng/apikey/Google
=== Data ===
Key    Value
---    -----
key    AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI

Use kv-v1/<PATH> endpoint to set secrets.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data @payload.json \
       $VAULT_ADDR/v1/kv-v1/<PATH>

Where kv-v1/<PATH> is the path to your secrets. The payload.json contains the parameters to invoke the endpoint.

Create an API request payload containing the data you wish to store.

$ tee payload.json <<EOF
{
   "key": "AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI"
}
EOF

Create a secret at path kv-v1/eng/apikey/Google with a key set to AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/kv-v1/eng/apikey/Google

Read back the secret at path kv-v1/eng/apikey/Google

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/kv-v1/eng/apikey/Google | jq -r ".data"

Output:

{
   "key": "AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI"
}

Store the root certificate for MySQL

For the purpose of this tutorial, generate a mock certificate using OpenSSL.

$ openssl req -x509 -sha256 -nodes -newkey rsa:2048 -keyout selfsigned.key -out cert.pem

The generated cert.pem displays something similar to the following.

Note

If you don't have OpenSSL, copy the above certificate and save it as cert.pem.

The command is basically the same as the Google API key example. The path convention for certificates is kv-v1/<ENVIRONMENT>/cert/<SYSTEM>. To store the root certificate for production MySQL, the path becomes kv-v1/prod/cert/mysql.

Create a secret at path kv-v1/prod/cert/mysql with a cert set to file contents for cert.pem.

$ vault kv put kv-v1/prod/cert/mysql cert=@cert.pem

Note

Any value that begins with @ indicates a file name. Data will be loaded from this file.

To perform the same task using the Vault API, pass the certificate in the request payload.

$ tee payload-cert.json <<EOF
{
  "cert": "-----BEGIN CERTIFICATE-----\nMIICyjCCAbICCQDrpZYh8et7yTANBgkqhkiG9w0BAQsFADAnMQswCQYDVQQGEwJV\nUzELMAkGA1UECAwCQ0ExCzAJBgNVBAcMAlNGMB4XDTE4MTExMjIwNDEwNVoXDTE4\nMTIxMjIwNDEwNVowJzELMAkGA1UEBhMCVVMxCzAJBgNVBAgMAkNBMQswCQYDVQQH\nDAJTRjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJnIdgpml8+xk+Oj\n1RGMCyJ1P15RiM6rdtszT+DFBg893Lqsjoyd5YgwELLz0Ux8nviG4L5OXOujEpAP\n2cQBxTSLQjBELBZY9q0Qky3+2ewqV6lSfcXrcf/JuDJGR5K8HSqwNG35R3WGnZ+O\nJhY0Dmx06IAs/FF8gP88zTQ8M7zuaThkF8MaF4sWPf6+texQwjzk4rewknGBFzar\n9wFxVwNCyDD6ewIYPtgDxdJ1bwBVoX3KKKXm8GStl/Zva0aEtbSq/161J4VbTro2\ndxArMPKzxjD6NLyF59UNs7vbzyfiw/Wq7BJzU7Kued5KdGt0bEiyWZYO+EvvxGmE\n1pHfqysCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAavj4CA+7XFVHbwYMbK3c9tN/\n73hkMvkAZWix5bfmOo0cNRuCeJnRIX+o6DmusIc8eXJJJV/20+zoSvUwlsLDPXoN\n+c41GfIiEUSaSdSBtETMy8oPga718nIwAvNgYiUHXnV3B0nLYBUpYSnsD00/6VXG\nxZUIEVBd7Ib5aRwmK8U5drxoWaBoG5qdvH9iapwTrCcPsRjsLBq7Iza2oBORGlfF\nCjqiW2+KJzwRiTQj70yceniGVHM+VSpFYCLJ0mXeyLfITy7joqxr4AGYz+EhpLuf\niDpYDNYlr0JDVQqogskWjrnWOh0YcIJKgVtiTh2HDM5TdQgeXg4wv5IqLok0Tw==\n-----END CERTIFICATE-----\n"
}
EOF

Create a secret at path kv-v1/prod/cert/mysql with the cert defined in the file named payload-cert.json.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload-cert.json \
    $VAULT_ADDR/v1/kv-v1/prod/cert/mysql

Generate a token for apps

To read the secrets, apps persona needs "read" ability on those secrets engine paths.

Create a policy file named apps-policy.hcl that permits the read to the paths kv-v1/eng/apikey/Google and kv-v1/prod/cert/mysql.

$ tee apps-policy.hcl <<EOF
# Read-only permit
path "kv-v1/eng/apikey/Google" {
  capabilities = [ "read" ]
}

# Read-only permit
path "kv-v1/prod/cert/mysql" {
  capabilities = [ "read" ]
}
EOF

First create apps policy

$ vault policy write apps apps-policy.hcl
Policy 'apps' written.

Create a variable that stores a new token with apps policy attached.
```
$ APPS_TOKEN=$(vault token create -policy="apps" -field=token)
```
Display the token.
```
$ echo $APPS_TOKEN
```
Now apps can use this token to read the secrets.

Create an API request payload containing the stringified apps that permits the read to the paths kv-v1/eng/apikey/Google and kv-v1/prod/cert/mysql.

$ tee payload-policy.json <<EOF
{
  "policy":  "# Read-only permit\npath \"kv-v1/eng/apikey/Google\" {\n  capabilities = [ \"read\" ]\n}\n\n# Read-only permit\npath \"kv-v1/prod/cert/mysql\" {\n  capabilities = [ \"read\" ]\n}"
}
EOF

Create apps policy.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request PUT \
    --data @payload-policy.json \
    $VAULT_ADDR/v1/sys/policies/acl/apps

Create a variable that stores a new token with apps policy attached.

$ APPS_TOKEN=$(curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"policies": ["apps"]}' \
    $VAULT_ADDR/v1/auth/token/create | jq -r ".auth.client_token")

Display the token.

$ echo $APPS_TOKEN

Now apps can use this token to read the secrets.

Open a web browser and click the Policies tab, and then select Create ACL policy.
Toggle the Upload file sliding switch, and click Choose a file to select your apps-policy.hcl file you authored. Set the Name to apps.
Click Create Policy to complete.
Click the Vault CLI shell icon (>_) to open a command shell. Execute the following command to create a new token: vault write auth/token/create policies=apps

Now apps can use this token to read the secrets.

Create token

Note

For the purposes of this tutorial, you created a policy for the apps persona, and generated a token for it. However, in the real world, you may have a dedicated policy author, or admin to write policies. Also, the consumer of the API key may be different from the consumer of the root certificate. Then each persona would have a policy based on what it needs to access.

Retrieve the secrets

Using the token from the Generate a token for apps step, read the Google API key and the root certificate for MySQL.

The command to read a secret is:

$ vault kv get kv-v1/<PATH>

Read the secret at the path kv-v1/eng/apikey/Google using the apps token.

$ VAULT_TOKEN=$APPS_TOKEN vault kv get kv-v1/eng/apikey/Google
Key                 Value
---                 -----
refresh_interval    768h0m0s
key                 AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI

Read only the key field at the path kv-v1/eng/apikey/Google.

$ VAULT_TOKEN=$APPS_TOKEN vault kv get -field=key kv-v1/eng/apikey/Google
AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI

Read only the cert field at the path kv-v1/prod/cert/mysql.

$ VAULT_TOKEN=$APPS_TOKEN vault kv get -field=cert kv-v1/prod/cert/mysql

-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA6E2Uq0XqreZISgVMUu9pnoMsq+OoK1PI54rsA9vtDE6wiRk0GWhf5vD4DGf1
...

The result displays the certificate.

Read the secret at the path kv-v1/eng/apikey/Google while authenticated with the apps token.

$ curl --header "X-Vault-Token: $APPS_TOKEN" \
    --request GET \
    $VAULT_ADDR/v1/kv-v1/eng/apikey/Google | jq

Output:

{
  "key": "AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI"
}

The same command is issued to Vault. The results of the command are sent to jq to filter the results. The -r option outputs raw strings. The ".data.key" defines a path that traversed to the desired value. The . represents the root element. The key named data maps to a root-level key. The key named key maps to a key within the data object.

Root certificate example:

Read only the cert field at the path kv-v1/prod/cert/mysql while authenticated with the apps token.

$ curl --header "X-Vault-Token: $APPS_TOKEN" \
    --request GET \
    $VAULT_ADDR/v1/kv-v1/prod/cert/mysql | jq -r ".data.cert"

The result displays the certificate.

Additional discussion

Q: How do I enter my secrets without exposing the secret in my shell's history?

As a precaution, you may wish to avoid passing your secret as a part of the CLI command so that the secret won't appear in the history file. Here are a few techniques you can use.

Option 1: Use a dash "-"

An easy technique is to use a dash "-" and then press Enter. This allows you to enter the secret on a new line. After entering the secret, press Ctrl+d to end the pipe which will write the secret to the Vault.

$ vault kv put kv-v1/eng/apikey/Google key=-

AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI
<Ctrl+d>

Option 2: Read the secret from a file

The key-value pairs for a secret may be defined in a file.

Create a file named apikey.json that defines the key field.

$ tee apikey.json <<EOF
{
  "key": "AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI"
}
EOF

Create a secret at path kv-v1/eng/apikey/Google with keys and values defined in apikey.json.

$ vault kv put kv-v1/eng/apikey/Google @apikey.json

Option 3: Disable all vault command history

The first two options ensure that the contents of the secret do not appear in the shell history. The secret path would still be accessible through the shell history.

You can configure your shell to avoid logging any vault commands to your history.

In bash, set the history to ignore all commands that start with vault.

$ export HISTIGNORE="&:vault*"

Note

This prevents vault commands from appearing when using the Up arrow or when searching command history with <Ctrl-r>.

Q: How do I save multiple values at once?

The tutorial examples demonstrating a secret with a single key-value pair. Multiple key-value pairs may be provided in a single command.

Create a secret at path kv-v1/dev/config/mongodb that sets the url, db_name, username, and password.

$ vault kv put kv-v1/dev/config/mongodb \
        url=foo.example.com:35533 \
        db_name=users \
        username=admin password=passw0rd

Multiple key-value pairs may be defined in a file provided to the command.

Create a file named mongodb.json that defines the url, db_name, username, and password fields.

$ tee mongodb.json <<EOF
{
    "url": "foo.example.com:35533",
    "db_name": "users",
    "username": "admin",
    "password": "pa$$w0rd"
}
EOF

Create a secret at path kv-v1/dev/config/mongodb with keys and values defined in mongodb.json.

Q: How do I store secrets generated by Vault in KV secrets engine?

Assuming that you have AppRole auth method enabled (refer to the AppRole Pull Authentication tutorial).

The following command retrieves the Secret ID of the jenkins role, encode it with base64, and stores the value in the kv-v1/secret-id path.

$ vault kv put kv-v1/secret-id \
   jenkins-secret-id="$(vault write -f -field=secret_id auth/approle/role/jenkins/secret-id | base64)"

Output:

Success! Data written to: kv-v1/secret-id

Clean up

If you wish to clean up your environment after completing the tutorial, follow the steps in this section.

Unset the VAULT_TOKEN environment variable.
```
$ unset VAULT_TOKEN
```
Unset the VAULT_ADDR environment variable.
```
$ unset VAULT_ADDR
```
HCP Vault users: Unset the VAULT_NAMESPACE environment variable.
```
$ unset VAULT_NAMESPACE
```
If you are running Vault locally in dev mode, stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
```
$ pgrep -f vault | xargs kill
```

Delete created files.

$ rm payload.json payload-cert.json payload-policy.json cert.pem apps-policy.hcl

Help and Reference

Key/Value Secrets Engine
Key/Value Secrets Engine API
Client libraries for Vault APIfor commonly used languages

Collection Overview

Secrets management

Versioned Key/value secrets engine