Secrets Management

Cubbyhole Response Wrapping

  • 14min

  • Vault
  • Interactive

The term cubbyhole comes from an Americanism where you get a "locker" or "safe place" to store your belongings or valuables. In Vault, the cubbyhole is your "locker". All secrets are namespaced under your token. If that token expires or is revoked, all the secrets in its cubbyhole are revoked as well.

It is not possible to reach into another token's cubbyhole even as the root user. This is an important difference between the cubbyhole and the key/value secrets engine. The secrets in the key/value secrets engine are accessible to any token for as long as its policy allows it.

Personas

The end-to-end scenario described in this tutorial involves two personas:

  • admin with privileged permissions to create tokens
  • apps trusted entity retrieving secrets from Vault

Challenge

In order to tightly manage the secrets, you set the scope of who can do what using the Vault policy and attach that to tokens, roles, entities, etc.

Think of a case where you have a trusted entity (Chef, Jenkins, etc.) which reads secrets from Vault. This trusted entity must obtain a token. If the trusted entity or its host machine was rebooted, it must re-authenticate with Vault using a valid token.

How can you securely distribute the initial token to the trusted entity?

Solution

Use Vault's cubbyhole response wrapping where the initial token is stored in the cubbyhole secrets engine. The wrapped secret can be unwrapped using the single-use wrapping token. Even the user or the system created the initial token won't see the original value. The wrapping token is short-lived and can be revoked just like any other tokens so that the risk of unauthorized access can be minimized.

What is cubbyhole response wrapping?

  • When response wrapping is requested, Vault creates a temporary single-use token (wrapping token) and insert the response into the token's cubbyhole with a short TTL
  • Only the expecting client who has the wrapping token can unwrap this secret
  • Any Vault response can be distributed using the response wrapping

Benefits of using the response wrapping:

  • It provides cover by ensuring that the value being transmitted across the wire is not the actual secret. It's a reference to the secret.
  • It provides malfeasance detection by ensuring that only a single party can ever unwrap the token and see what's inside
  • It limits the lifetime of the secret exposure
    • The TTL of the response-wrapping token is separate from the wrapped secret's lease TTL

Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault environment. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Policy requirements

NOTE: For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for just enough initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# Manage tokens
path "auth/token/*" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}

# Write ACL policies
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage secret/dev secrets engine - for Verification test
path "secret/dev" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage tokens
path "auth/token/*" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}

# Write ACL policies
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Manage secret/dev secrets engine - for Verification test
path "secret/dev" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

If you are not familiar with policies, complete the policies tutorial.

Lab Setup

Open a terminal and start a Vault dev server with root as the root token.

$ vault server -dev -dev-root-token-id root

$ vault server -dev -dev-root-token-id root

The Vault dev server defaults to running at 127.0.0.1:8200. The server is initialized and unsealed.

Insecure operation: Do not run a Vault dev server in production. This approach starts a Vault server with an in-memory database and runs in an insecure way.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://127.0.0.1:8200

$ export VAULT_ADDR=http://127.0.0.1:8200

Export an environment variable for the vault CLI to authenticate with the Vault server.

$ export VAULT_TOKEN=root

$ export VAULT_TOKEN=root

NOTE: For these tasks, you can use Vault's root token. However, it is recommended that root tokens are only used for enough initial setup or in emergencies. As a best practice, use an authentication method or token that meets the policy requirements.

Scenario Introduction

An app needs to retrieve secrets from Vault at the secret/kv path. However, the app does not have a valid client token to read secrets at secret/kv. Vault admin wraps the secret values using the cubbyhole response wrapping, and sends the wrapping token to the app. The app unwraps the secrets before the wrapping token expires.

Response Wrapping Scenario

NOTE: This tutorial demonstrates how the response wrapping works. To learn more about K/V secrets engine, refer to the Versioned Key/Value Secrets Engine tutorial.

Step 1: Create and wrap a token

(Persona: admin)

When a newly created token is wrapped, Vault inserts the generated token into the cubbyhole of a single-use token, returning that single-use wrapping token. Retrieving the secret requires an unwrap operation against this wrapping token.

  1. Write some test data at secret/dev.

    $ vault kv put secret/dev username="webapp" password="my-long-password"

    $ vault kv put secret/dev username="webapp" password="my-long-password"

    Example output:

    = Secret Path =
secret/data/dev

======= Metadata =======
Key                Value
---                -----
created_time       2022-03-30T18:50:59.153742Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

    = Secret Path =
secret/data/dev

======= Metadata =======
Key                Value
---                -----
created_time       2022-03-30T18:50:59.153742Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

  2. Read the secrets at secret/dev and wrap the output using the -wrap-ttl flag to specify that the response should be wrapped and the life of the wrapping token to be 120 seconds (2 minutes).

    $ vault kv get -wrap-ttl=120 secret/dev

    $ vault kv get -wrap-ttl=120 secret/dev

    Example output:

    Key                              Value
---                              -----
wrapping_token:                  hvs.CAESIGu9Ulc0Uhmqa8hmXx7DyvnrFvOGmlecFcl8kkGz-tLoGh4KHGh2cy5uYzZsZEMxa3VKdE5UakpiR25LZXpjczA
wrapping_accessor:               3dyFk8GHlmLNvKEjxcL9TDz2
wrapping_token_ttl:              2m
wrapping_token_creation_time:    2022-04-05 21:09:08.2289 -0700 PDT
wrapping_token_creation_path:    secret/data/dev

    Key                              Value
---                              -----
wrapping_token:                  hvs.CAESIGu9Ulc0Uhmqa8hmXx7DyvnrFvOGmlecFcl8kkGz-tLoGh4KHGh2cy5uYzZsZEMxa3VKdE5UakpiR25LZXpjczA
wrapping_accessor:               3dyFk8GHlmLNvKEjxcL9TDz2
wrapping_token_ttl:              2m
wrapping_token_creation_time:    2022-04-05 21:09:08.2289 -0700 PDT
wrapping_token_creation_path:    secret/data/dev

    Instead of displaying the secrets at secret/dev, the response includes the wrapping token.

  3. Store the returned wrapping token value in a WRAPPING_TOKEN environment variable.

    Example:

    $ export WRAPPING_TOKEN="hvs.CAESIGu9Ulc0Uhmqa8hmXx7DyvnrFvOGmlecFcl8kkGz-tLoGh4KHGh2cy5uYzZsZEMxa3VKdE5UakpiR25LZXpjczA"

    $ export WRAPPING_TOKEN="hvs.CAESIGu9Ulc0Uhmqa8hmXx7DyvnrFvOGmlecFcl8kkGz-tLoGh4KHGh2cy5uYzZsZEMxa3VKdE5UakpiR25LZXpjczA"

Step 2: Unwrap the secret

(Persona: apps)

The apps persona receives a wrapping token from the admin. In order for the apps to acquire a valid token to read secrets from secret/dev path, it must run the unwrap operation using this token.

NOTE: If a client has been expecting delivery of a response-wrapping token and none arrives, this may be due to an attacker intercepting the token and then preventing it from traveling further. This should cause an alert to trigger an immediate investigation.

Unwrap the secret by passing the wrapping token.

$ VAULT_TOKEN=$WRAPPING_TOKEN vault unwrap

$ VAULT_TOKEN=$WRAPPING_TOKEN vault unwrap

To unwrap the secrets using valid wrapping token, the app does not have to have a client token.

Example output:

Key         Value
---         -----
data        map[password:my-long-password username:webapp]
metadata    map[created_time:2022-04-06T04:24:24.391513Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

Key         Value
---         -----
data        map[password:my-long-password username:webapp]
metadata    map[created_time:2022-04-06T04:24:24.391513Z custom_metadata:<nil> deletion_time: destroyed:false version:1]

Notice that the data displays the password and username stored at secret/dev.

Try unwrap the secrets again.

$ VAULT_TOKEN=$WRAPPING_TOKEN vault unwrap

$ VAULT_TOKEN=$WRAPPING_TOKEN vault unwrap

This returns an error, "wrapping token is not valid or does not exist" because wrapping tokens are limited to single-use.

Default policy

The cubbyhole secrets engine is mounted at the cubbyhole/ prefix by default. The secrets you store in the cubbyhole/ path are tied to your token and all tokens are permitted to read and write to the cubbyhole secrets engine by the default policy. Every token has the default policy attached unless you set the -no-default-policy flag.

$ vault policy read default

$ vault policy read default

The default policy grants permission to perform all operations against the cubbyhole/ path. Also notice that the default policy permits update operation on the sys/unwrapping/ paths.

...snip...

# Allow a token to manage its own cubbyhole
path "cubbyhole/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}

# Allow a token to wrap arbitrary values in a response-wrapping token
path "sys/wrapping/wrap" {
    capabilities = ["update"]
}

# Allow a token to look up the creation time and TTL of a given
# response-wrapping token
path "sys/wrapping/lookup" {
    capabilities = ["update"]
}

# Allow a token to unwrap a response-wrapping token. This is a convenience to
# avoid client token swapping since this is also part of the response wrapping
# policy.
path "sys/wrapping/unwrap" {
    capabilities = ["update"]
}

...snip...
default policy
...snip...

# Allow a token to manage its own cubbyhole
path "cubbyhole/*" {
    capabilities = ["create", "read", "update", "delete", "list"]
}

# Allow a token to wrap arbitrary values in a response-wrapping token
path "sys/wrapping/wrap" {
    capabilities = ["update"]
}

# Allow a token to look up the creation time and TTL of a given
# response-wrapping token
path "sys/wrapping/lookup" {
    capabilities = ["update"]
}

# Allow a token to unwrap a response-wrapping token. This is a convenience to
# avoid client token swapping since this is also part of the response wrapping
# policy.
path "sys/wrapping/unwrap" {
    capabilities = ["update"]
}

...snip...

Test the default policy

  1. To better demonstrate the cubbyhole secrets engine, create a token with only default policy attached.

    $ vault token create -policy=default \
    -format=json | jq -r ".auth.client_token" > test_token.txt

    $ vault token create -policy=default \
    -format=json | jq -r ".auth.client_token" > test_token.txt

  2. Write some test data in the token's cubbyhole using the test token.

    $ VAULT_TOKEN=$(cat test_token.txt) vault write cubbyhole/private mobile="123-456-7890"
Success! Data written to: cubbyhole/private

    $ VAULT_TOKEN=$(cat test_token.txt) vault write cubbyhole/private mobile="123-456-7890"
Success! Data written to: cubbyhole/private

  3. Read back the secret you just wrote. It should return the secret.

    $ VAULT_TOKEN=$(cat test_token.txt) vault read cubbyhole/private

Key       Value
---       -----
mobile    123-456-7890

    $ VAULT_TOKEN=$(cat test_token.txt) vault read cubbyhole/private

Key       Value
---       -----
mobile    123-456-7890

  4. Now, try to read the cubbyhole/private path.

    $ vault read cubbyhole/private

    $ vault read cubbyhole/private

    NOTE: The command uses the token stored in the VAULT_TOKEN environment variable instead of test_token.txt. When in doubt, you can use the vault token lookup command to see which token you are using.

    Output:

    No value found at cubbyhole/private

    No value found at cubbyhole/private

    Cubbyhole secret backend provide an isolated secrete storage area for an individual token where no other token can violate.

Help and Reference

Previous
Next