Query Vault secrets with Chef

Author: Daniel Greeninger

This validated pattern guides you through connecting a Chef cookbook to Vault, retrieving a secret, and using the secret value within the recipe. Chef lets you automate your infrastructure and security through infrastructure as code.

Chef is an configuration management tool that lets you configure and manage your infrastructure and services at scale. Vault provides secure storage and access management for sensitive information such as passwords, certificates, and API keys. Integrating Vault with Chef allows you to access sensitive data without exposing secrets in your cookbook code.

You can find the Chef cookbooks and configuration for this guide in the HVD Vault Chef GitHub repository.

Target audience

This guide references the following roles:

• Platform operator: Someone who manages your organization's platform. This could be the Chef server administrator on the Automation team, who deploys certificates or secret-id components for the application servers to consume.
• Vault producer: An administrator to create the authentication method and secrets.
• Vault consumer: Someone who has access to read Vault secrets, such as Application and DevOps engineers.

Prerequisites

To follow this validated pattern, you need the following:

• A Vault cluster accessible with an administrator role. The administrator must have permissions to configure an auth method and secrets engine.
• The ability to authenticate to the Vault cluster.
• Permissions to clone the demo GitHub repository containing the Chef cookbook.
• A Chef installation to run the cookbook on an application server or local machine.
• The chef-solo command line tool.

Limitations

If you are using a version of Chef less than 17.5, you have to use the vault-ruby gem to allow Chef client to connect to your Vault cluster and return a secret.

Validated architecture

The architecture diagram shows how Chef retrieves secrets from a Vault server to configure an existing file on an application server.

This diagram shows the workflow Chef goes through to provision an application server with Vault secrets:

1. The Chef client retrieves the recipes from the Chef server and stores them on the application server.
2. Chef installs and configures the Vault agent on the application server.
3. The Vault agent authenticates to the Vault cluster with a certificate or AppRole and writes a token sink file on the application server.
4. The Chef recipe uses the Vault token from the sink file to send the requested parameters to Vault via the Chef secret method or a custom resource.
5. Vault returns the secret values to the Chef recipe.
6. The Chef recipe uses the returned secret values to populate the existing configuration files.
7. The Application can then read the secret values from the configuration files.

As shown in the diagram, you should use the Chef client to install and configure Vault agent on the application server. The Chef client uses the sink token provided by Vault agent to retrieve secrets from Vault. When the Chef server authenticates to Vault, the Chef server provides the Vault Agent with the `key` or `secret-id` as a sensitive attribute. The Vault agent retrieves the certificate or `role-id` from the base image of the machine — the Chef server should not transfer these values. This ensures that the two components required to authenticate to Vault are separated, as recommended in the Trusted Orchestrator guide.

The Chef client can connect to Vault using the Chef secret or Ruby Gem. As of Chef 17.5, the Chef secret method supports the hashi_vault service, with no custom resource.

Checklist

To retrieve secrets from Vault in Chef, make sure you have the following in place:

• Networking infrastructure is in place to allow servers to connect to Vault.
• An authentication method on the Vault cluster that the application servers will use.
• A Vault policy that will allow the application servers to access the secrets they need.
• Incorporate our example cookbook into your application run-list.
• Optional, but we recommend you retrieve and template secrets with the Vault Agent.

Install Chef on your workstation

The DevOps engineer installs Chef Workstation on the machine they will work from. This can be their laptop or a remote server.

Visit Chef Workstation to download and install Chef Workstation for your operating system. Chef Workstation lets you execute the demo cookbook.

Clone the Chef cookbook

The DevOps engineer clones the Chef cookbook from GitHub.

$ git clone https://github.com/hashicorp-validated-designs/vault-chef

Navigate to the demo repository.

$ cd vault-chef

Configure your Vault cluster

The commands in this validated pattern will enable the Chef cookbook to connect to Vault. However, your specific environment may require additional commands based on how it is configured.

A Vault Administrator will connect to the Vault cluster and run the following commands from the root of the git repository.

First, set the VAULT_ADDR environment variable to your Vault cluster.

$ export VAULT_ADDR="https://vault.example.com:8200"

Next, authenticate to Vault as an administrator. Enter the value of your admin token when prompted.

$ vault login -no-print

Next, configure the certificate authentication for the demo. This example uses a self-signed certificate for the TLS auth method, which is insecure and does not scale. For production use cases, ensure you use a key and certificate provided by your company's certificate authority.

$ openssl req -new -newkey rsa:4096 -noenc -keyout /tmp/vault.key -subj '/CN=vaultagent.example.com' -out /tmp/vault.csr && \
  openssl x509 -req -days 365 -in /tmp/vault.csr -signkey /tmp/vault.key -out /tmp/vault.crt  && \
  vault policy write minimum minimum-policy.hcl  && \
  vault auth enable cert && \
  vault write auth/cert/certs/chef display_name="chef" policies="admins" certificate=@/tmp/vault.crt && \
  export VAULT_PRIVATE_KEY=$(cat /tmp/vault.key)

Create a secret engine to hold the demo secret values.

$ vault secrets enable -version=2 -path=secret kv

Set the demo secret.

$ vault kv put -mount=secret foo username=admin password=12345

Review Chef cookbook

In the demo repository, open the default cookbook. This Chef cookbook installs Vault Agent, authenticates with Vault, and retrieves a secret for use in the recipe.

#install Vault Agent and create template file
include_recipe "vault-chef::vault-agent"

# Execute the Vault agent command to authenticate and exit after auth
execute 'vault agent' do
  command 'vault agent -config=/tmp/vault-agent.hcl -exit-after-auth'
  action :run
  notifies :read, 'vault_secret[secret]', :immediately  # Notify the vault_secret block
end

# Use Chef's built in secret method, connect to Vault, and write to a file
file "/tmp/vault-secret" do
  content lazy {
    secret(name: "secret/data/#{node['vault']['secret-key']}",
                   service: :hashi_vault,
                   config: {
                     vault_addr: node['vault']['address'],
                     namespace: node['vault']['namespace'],
                     auth_method: :token,
                     token: File.read("/tmp/vault-token-via-agent")
                   })[:data].to_s
  }
end

# Before Chef 17.5, the secret method did not exist. Here is a custom resource that can be used
# vault_secret calls the custom resource in resources/vault_secret.rb
# Read the KV secret from Vault using the token generated by the agent
vault_secret 'secret' do
  vault_address node['vault']['address']
  vault_namespace node['vault']['namespace']
  token lazy {  # Lazy evaluation ensures the token file is read at execution time
    File.exist?("/tmp/vault-token-via-agent") ? File.read("/tmp/vault-token-via-agent") : "Didn't get the token from the agent!"
  }
  secret_key node['vault']['secret-key']
  action :nothing  # Triggered only by notification from the execute block
end

# Use the secret in your recipe
ruby_block 'print_secret' do
  block do
    secret_value = node.run_state['vault_secret']
    puts "\n\nThe secret value is: #{secret_value}"
  end
end

It begins by including the vault-chef::vault-agent recipe, which sets up the Vault Agent and its config file. The execute block runs the agent in -exit-after-auth mode to authenticate and write a token to a file (/tmp/vault-token-via-agent). Once complete, it notifies a custom vault_secret resource to run.

The file resource uses Chef's built-in secret helper (available in Chef 17.5+) to connect to Vault using the token and fetch the secret. It writes this secret to /tmp/vault-secret. The lazy block delays evaluation until execution time, ensuring the token is available.

For older Chef versions, a fallback custom resource vault_secret retrieves the secret manually using the token. It runs only when triggered by the agent's execution and stores the secret in node.run_state.

Finally, the ruby_block accesses and prints the secret from node.run_state, showing how you can use the secret elsewhere in the recipe.

Open the vault_secret resource.

provides :vault_secret

property :path, String, name_property: true
property :vault_address, String, required: true
property :vault_namespace , String, required: true
property :token, String, required: true
property :secret_key, String, required: false

action :read do
  require 'vault'
  # Configure Vault client
  Vault.address = new_resource.vault_address
  Vault.namespace = new_resource.vault_namespace
  Vault.token = new_resource.token
  # Read the secret from Vault at the data path created with a kvv2 secret mount
  secret = Vault.logical.read(new_resource.path+'/data/'+new_resource.secret_key)
  # Store the secret in a node attribute
  node.run_state['vault_secret'] = secret.data
end

action :dynamic do
  require 'vault'

  # Configure Vault client
  Vault.address = new_resource.vault_address
  Vault.token = new_resource.token
  Vault.namespace = new_resource.vault_namespace
  # Read the database secret from Vault
  secret = Vault.logical.read(new_resource.path)
  # Store the secret in a node attribute
  node.run_state[new_resource.path] = secret
end

This custom Chef resource, called vault_secret, helps you safely get secrets from Vault when your version of Chef doesn’t support the newer built-in secret() helper. It uses the vault-ruby gem to connect to Vault and read secrets during a Chef run.

There are two actions in this resource. The first one is :read, which is used for Vault’s KV version 2 engine. The following line connects to Vault and reads the secret from a specific path. After it gets the secret, it saves the data to node.run_state['vault_secret'], so you can use it later in the recipe.

Vault.logical.read(new_resource.path + '/data/' + new_resource.secret_key)
node.run_state['vault_secret'] = secret.data

The second action is :dynamic, which is more general. It reads a secret from any path. This is helpful when your secrets aren’t in KVv2, like database passwords or cloud credentials. It saves the full secret in node.run_state, using the path as the key.

Vault.logical.read(new_resource.path)

Verify cookbook functionality

The DevOps engineer should validate the cookbook before publishing it. Testing the cookbook on the workstation against the Vault cluster will help ensure it correctly functions in production.

To test the cookbook, set the environment variables on your workstation to connect to your Vault cluster. You can find the variables in your workstation, located at cookbooks/vault-chef/attributes/default.rb. Once in production, your Chef Server administrator will set these attributes on the application server’s Chef role.

$ export VAULT_ADDR="https://vault.example.com:8200"

Set the namespace to admin if you are using HCP Vault.

$ export VAULT_NAMESPACE=""

Export the Vault private key.

$ export VAULT_PRIVATE_KEY=$(cat /tmp/vault.key)

For local development, you can skip the verification of the self signed certificate.

$ export VAULT_SKIP_VERIFY=1

Once you set the environment variables, navigate to the root directory where /cookbooks is located. Create a file named solo.json and populate the file with a run-list of recipes. For the demo, use the solo.json file in the root of the example repo.

Run chef-solo from the root directory, passing in the run-list of recipes via the solo.json file. Refer to the chef-solo documentation for more information.

Run chef-solo.

$ chef-solo -c solo.rb -j solo.json

At the end of the Chef run, the output of the recipe contains the secret value you set in the previous step. You can use those values in your own Chef recipes by referencing the vault_secret attribute in node.run_state.

Make sure to use the sensitive attribute to prevent sensitive resource data from being logged by Chef Infra Client. Example: password = node.run_state['vault_secret'][:data][:password]

In the demo below, we do not set the sensitive property and log the secret to the terminal output, showing that Chef has access to the secret value.

  * ruby_block[print_secret] action run

The secret value is: {:data=>{:password=>"12345", :username=>"admin"}, :metadata=>{:created_time=>"2024-10-16T23:02:59.892089526Z", :custom_metadata=>nil, :deletion_time=>"", :destroyed=>false, :version=>2}}

    - execute the ruby block print_secret

Upload cookbook

Once you write the cookbook and it and passes your tests, a Chef server administrator can upload the cookbook to the Chef server.

• Customize the cookbook to meet your specific requirements.
• Run the appropriate knife command to upload the cookbook to your Chef Server.
• Add the cookbook to your role's run list..
• Run knife on your application servers to process the new cookbook and recipes.

Cleanup demo resources

The Vault administrator will remove the demo resources from Vault cluster.

$ vault secrets disable -version=2 -path=secret kv && \
  vault auth disable cert  && \
  vault policy delete minimum

The Chef administrator will delete the vault-chef demo cookbook from the Chef Server.

$ knife cookbook delete vault-chef

The DevOps Engineer will remove the demo files and unset environment variables from the workstation to remove Vault Server.

$ rm /tmp/vault.key /tmp/vault.crt /tmp/vault.csr /tmp/role-id /tmp/secret-id /tmp/vault-token-via-agent /tmp/vault-agent.hcl  && \
  unset VAULT_PRIVATE_KEY; unset VAULT_ADDR;

Remove the Vault package from your system using a package manager.

$ brew remove vault && apt remove vault && yum remove vault

Conclusion

By following this guide, you have learned how to integrate Chef with Vault. You learned how Chef recipes can securely access secrets from Vault for deployments.

To learn more, check out the following resources:

• GitHub with example Chef Cookbook
• Dynamic secrets: database credential management
• Chef Secret
• Best practices to protect sensitive data