Synchronize cloud native secrets

11min
|
Enterprise
Vault
Video
Interactive

Deployment

Vault 1.16 includes a new feature for HashiCorp Vault Enterprise - Secrets sync. This allows you to automatically sync secrets from Vault Enterprise to a variety of third party platforms including AWS, Azure, GCP, GitHub, and Vercel.

This tutorial instructs the learner to create a static K/V secret in Vault Enterprise. Then you set up a destination and association to sync that secret between HashiCorp Vault Enterprise and AWS Secrets Manager.

Challenge

Vault Enterprise Secrets Sync addresses two common challenge to Vault adoption.

First, some services cannot communicate with Vault. Cloud providers naturally offer strong support with their own secrets management solutions. For example, AWS RDS Proxy only communicates with AWS Secrets Manager. This contributes to secrets sprawl, and being able to manage secrets in Vault Enterprise but allow access through platform specific tools is invaluable.

Second, organizations interested in leveraging the power of secrets management with Vault but access them in a platform native way. Though there are resources available, adoption of Vault requires developers to learn new technologies. Secrets Sync allows developers to focus on their preexisting platform knowledge and reduce effort an organization to adopt Vault. By focusing on cloud provider tools, they build on preexisting skills and reduce the adoption friction of new technologies.

Solution

secrets sync example

Vault Enterprise Secrets Sync automatically sync secrets between Vault and AWS Secrets Manager. Developers do not need to learn how to consume secrets from Vault. Operators continue utilizing the power of Vault Enterprise for secrets lifecycle management, but developers can consume the secrets through platform native tools and technologies.

associations and destinations

Sync destinations represent a location where to sync secrets with. There are five types of destinations: AWS Secrets Manager, Azure Key Vault, GCP Secret Manager, GitHub repository secret and Vercel Project environment variables. Multiple destinations may share a single secret. For example, a Jira token stored as a Vault secret used in CI workflows can sync to many GitHub repositories, each repository represented by a distinct Vault sync destination.

Secret associations represent a link between an existing Vault KV secret and a sync destination. Associations are granular and each secrets has a association. Many associations can share a destination.

Personas

A Vault administrator using Vault Enterprise for secrets lifecycle management but their organization requires that the secrets to be available through AWS Secrets Manager.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Prerequisites

To complete this tutorial you will need the following:

Vault Enterprise version 1.16 or later with valid license. Learn how to request a trial license.
Vault CLI
Terraform CLI
Amazon Web Services test account. This tutorial requires the creation of new cloud resources and but should not incur costs associated with the deployment and management of these resources. Your IAM user will need permission to create IAM User, and to create access keys for the user.
- Basic understanding of the AWS user interface, and the use of AWS Secrets Manager.
- Creating an IAM user with Terraform requires AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

Lab set up

Clone an GitHub repository, create an IAM user with the proper policy and start an local instance of Enterprise Vault or HCP Vault to complete this lab.

Create AWS IAM user

To set up secrets sync Vault needs programmatic access to create, read, delete, update secrets in AWS Secrets Manager. In this section you will create an IAM User with least privileged access. You can create the IAM user manually or you can run the Terraform configuration included in the GitHub repository.

Create a IAM policy named vault_secrets_sync_policy with the following privileges.

{
  "Version": "2012-10-17",
  "Statement": [
     {
           "Sid": "SecretsManagerAccess",
           "Effect": "Allow",
           "Action": [
              "secretsmanager:DescribeSecret",
              "secretsmanager:GetSecretValue",
              "secretsmanager:CreateSecret",
              "secretsmanager:PutSecretValue",
              "secretsmanager:UpdateSecret",
              "secretsmanager:UpdateSecretVersionStage",
              "secretsmanager:DeleteSecret",
              "secretsmanager:RestoreSecret",
              "secretsmanager:TagResource",
              "secretsmanager:UntagResource"
           ],
           "Resource": "*"
     }
  ]
}

Create a IAM user named vault_secrets_sync_user and attach the vault_secrets_sync_policy.
Create AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for the IAM user. See Managing access keys (console) for instructions.
Open a new terminal, set the VAULT_USER_ACCESS_KEY_ID and VAULT_USER_SECRET_ACCESS_KEY environment variables to the values from the IAM User vault_secrets_sync_user. If you are not running on us-east-1 please update the AWS_REGION.
```
$ export VAULT_USER_ACCESS_KEY_ID=<<IAM_USER_ACCESS_KEY>> && export VAULT_USER_SECRET_ACCESS_KEY=<<IAM_USER_SECRET_KEY>> && export AWS_REGION=us-east-1
```

Clone the repository at learn-vault-enterprise-secrets-sync.

$ git clone https://github.com/hashicorp-education/learn-vault-enterprise-secrets-sync.git

Move into the terraform subdirectory of folder just created.
```
$ cd learn-vault-enterprise-secrets-sync/terraform
```
Working directory
This tutorial assumes that the learner executes the remainder of commands in the Terraform section in this directory.
Export your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY as environment variables. This key pair should have permission to create the AWS IAM resources in the provided Terraform configuration.
If you are not running on us-east-1 please update the AWS_REGION.
```
$ export AWS_ACCESS_KEY_ID=<<YOUR_ACCESS_KEY>> && export AWS_SECRET_ACCESS_KEY=<<YOUR_SECRET_KEY>> && export AWS_REGION=us-east-1
```

Initialize the configuration.

$ terraform init
...
Terraform has been successfully initialized!
...

Look at the plan.

$ terraform plan
...
Plan: 3 to add, 0 to change, 0 to destroy.

Changes to Outputs:

access_key = (known after apply)
secret_key = (sensitive value) ...

Apply the configuration.

$ terraform apply -auto-approve
...

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
Outputs:
access_key = "AKIA3SCJYFFCO3FDO4P3"
secret_key = <sensitive>

In the terraform.tfstate file to locate the secret access key.

$ cat terraform.tfstate | head -n 15
{
"version": 4,
"terraform_version": "1.5.4",
"serial": 4,
"lineage": "76f54aef-b09b-647a-ab25-803ecb8f5a1f",
"outputs": {
   "access_key": {
      "value": "AKIA4V3NG3RZO3FDO4P3",
      "type": "string"
   },
   "secret_key": {
      "value": "tXMencyCLQpz1sRowsaMFGEZbsibIDJli9/Qy/E",
      "type": "string",
      "sensitive": true
   }
},

Near the top of the block just examined will find outputs block with a access_key and secret_key fields. Copy the value for each.
In the same terminal, set the VAULT_USER_ACCESS_KEY_ID and VAULT_USER_SECRET_ACCESS_KEY to the values from the IAM User vault_secrets_sync_user. If you are not running on us-east-1 please update the AWS_REGION.
```
$ export VAULT_USER_ACCESS_KEY_ID=<<IAM_USER_ACCESS_KEY>> && export VAULT_USER_SECRET_ACCESS_KEY=<<IAM_USER_SECRET_KEY>> && export AWS_REGION=us-east-1
```

Set up Vault Enterprise

Note

If you do not have access to an HCP Vault cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within moments, Vault generates a new token.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.
The Vault server is ready.

Enterprise secrets sync

Create a KV secret

Create the KV secrets engine.

$ vault secrets enable -path=kv1 kv-v2
Success! Enabled the kv-v2 secrets engine at: kv1/

Create the secret.

$ vault kv put kv1/database/dev api_key="abc" key_id="123"
==== Secret Path ====
kv1/data/database/dev

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T17:56:26.700919Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            1

Configure destination and association

Configure the destination.

$ vault write sys/sync/destinations/aws-sm/aws-sm1 \
 access_key_id="${VAULT_USER_ACCESS_KEY_ID}" \
 secret_access_key="${VAULT_USER_SECRET_ACCESS_KEY}" \
 region="${AWS_REGION}" \
 secret_name_template="{{ .MountAccessor }}_{{ .SecretBaseName }}"

The output should resemble the following:

Key                   Value
---                   -----
connection_details    map[access_key_id:***** region:us-east-1 secret_access_key:*****]
name                  aws-sm1
options               map[custom_tags:map[] granularity_level:secret-path secret_name_template:{{ .MountAccessor }}_{{ .SecretBaseName }}]
type                  aws-sm

By default, the name of synced secrets follows this format: vault/<accessor>/<secret-path>, but each destination allows you to customize the name through the secret_name_template field. Refer to the Name Template documentation for details.

Set up the association.

$ vault write sys/sync/destinations/aws-sm/aws-sm1/associations/set \
   mount="kv1" \
   secret_name="database/dev"

The output should resemble the following:

Key                   Value
---                   -----
associated_secrets    map[kv_c6c5fb8c/database/dev:map[accessor:kv_c6c5fb8c secret_name:database/dev]]
store_name            aws-sm1
store_type            aws-sm

Log into your AWS Account and take a navigate to the AWS Secrets Manager.
Choose the link ending with /database/dev.
In AWS Secrets Manager choose Retrieve secret value in the Secret value section.
These are the same values as the kv secret in kv1/database/dev.

Modifications to the secret

Changes to the secret in Vault Enterprise are also reflected in AWS Secrets Manager.

Update the secret.

$ vault kv put kv1/database/dev api_key="abc" key_id="updated"
==== Secret Path ====
kv1/data/database/dev

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-12T18:36:42.241359Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            2

Open up your AWS account and reload the secret in AWS Secrets Manager.

Use a patch command to add an last_key field.

$ vault kv patch -mount=kv1 database/dev last_key=new_one
==== Secret Path ====
kv1/data/database/dev

======= Metadata =======
Key                Value
---                -----
created_time       2023-09-08T18:00:49.720538Z
custom_metadata    <nil>
deletion_time      n/a
destroyed          false
version            3

Remove associations (optional)

If you no longer want the secret to sync with AWS Secrets Manager, you can remove the association.

Remove the association between the secret and AWS Secrets Manager.

$ vault write sys/sync/destinations/aws-sm/aws-sm1/associations/remove \
   mount="kv1" \
   secret_name="database/dev"

The output should resemble the following:

Key                   Value
---                   -----
associated_secrets    map[]
store_name            aws-sm1
store_type            aws-sm

Return to your browser windows with AWS Secrets Manager and refresh the page. It should resemble the following:

Clean up

If you created the IAM user manually, delete the IAM user.
If you created the IAM User with Terraform, switch back to the original AWS credentials AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY used with Terraform to create the IAM User.

Destroy the resources Terraform created.

$ terraform destroy -auto-approve
...

Destroy complete! Resources: 3 destroyed.

Unset the environment variables used in this lab.

$ unset VAULT_USER_ACCESS_KEY \
VAULT_USER_SECRET_ACCESS_KEY \
VAULT_ADDR \
VAULT_TOKEN \

Stop the Vault server process.
```
$ pgrep -f vault | xargs kill
```
The Terraform version of this tutorial requires you to go back to the parent directory of learn-vault-enterprise-secrets-sync.
```
$ cd ../..
```

Delete the learn-vault-pki-engine directory.

$ rm -rf learn-vault-enterprise-secrets-sync

Summary

You learned how to set up Vault Enterprise secrets sync, which consists of destinations and associations. Destinations are platforms that Vault will sync a secret to. Associations create a link between a secret and sync destination. Changing a secret in Vault it will change the corresponding secret on the target platform service.

This tutorial guided you to set up a link between a secret in Vault Enterprise and AWS Secrets Manager, and demonstrated when changing secrets in Vault, the corresponding secret in AWS Secrets Manager is also changed.

References

Federated workload identities

Next Collection

Monitor & troubleshoot