Access HCP Vault Secrets with Terraform

9min
|
HCP
Terraform
Vault

In addition to using the command line interface (CLI) or application programming interface (API), you can also use the HCP Terraform provider to retrieve secrets from HCP Vault Secrets.

Prerequisites

An existing HCP account
Terraform installed
HCP Terraform account (HCP Terraform workflow only)
Completed the previous HCP Vault Secrets tutorials
HCP service principal with HCP_CLIENT_ID and HCP_CLIENT_SECRET available

Set environment variables

If you did not store your HCP client ID and secret, generate a new key.

Open your web browser and log into the HCP Portal.
Select the project where HCP Vault Secrets is configured.
Click Access control (IAM).
Click Service principals, and select your service principal name.
Click Keys, and click Generate key.
Copy the Client ID.
Return to your terminal and export an environment variable named HCP_CLIENT_ID.
```
$ export HCP_CLIENT_ID=<copied-id>
```
Return to the HCP portal and copy the Client secret.
Return to your terminal and export an environment variable named HCP_CLIENT_SECRET.
```
$ export HCP_CLIENT_SECRET=<copied-secret>
```

Store the project_id in a environment variable.

$ export HCP_PROJECT_ID=$(hcp profile display --format=json | jq -r .ProjectID)

Retrieve secrets with Terraform

The HCP Terraform provider includes a data source for HCP Vault Secrets. This data source allows you to retrieve secrets and use them in a Terraform configuration.

In this example, you will make use of HCP Terraform variable sets, and place all other required parameters in the Terraform configuration. This is done to give you a full picture of the required parameters. You can also choose to set the HCP_CLIENT_ID, HCP_CLIENT_SECRET, and TF_CLOUD_ORGANIZATION variables in your local shell.

Retrieve the HCP_CLIENT_ID and HCP_CLIENT_SECRET environment variables set during the Install HCP CLI for Vault Secrets tutorial.

$ printenv | grep HCP_
HCP_CLIENT_ID=aBcDefGx2ruWptclGVpIE7MX7uxkU4
HCP_CLIENT_SECRET=S3sAm3stxKjziNhjTWWe9QxEffKyTdiGlxE_suNsh1n3

Log into the HCP Terraform portal and select your organization.
Select Settings > Variable sets from the left navigation. Variables in HCP Terraform allow you to store items similar to setting an environment variable for a shell session or in a .tfvars file.
Click Create variable set.
Enter HCP_SERVICE_PRINCIPAL in the Name textbox.
Select the Apply globally radio button.
Click + Add variable.
Enter HCP_CLIENT_ID in the Key field, and the actual client ID in the Value field.
Click the Sensitive checkbox, then click Add variable.
Click + Add variable again.
Enter HCP_CLIENT_SECRET in the Key field, and the actual client secret in the Value field.
Click the Sensitive checkbox, then click Add variable.
Click Create variable set.
Click < Workspaces to return to the Workspace landing page.
Click New and select Workspace.
Select CLI-driven workflow and enter HCPVaultSecretsLab in the Workspace name field.
Scroll to the bottom of the form and click Create workspace.
Copy the name of your HCP Terraform organization and return to your terminal session.
Set an environment variable with the name of your HCP Terraform organization.
```
$ export HCP_ORG=<actual-org-name>
```

Create an example Terraform configuration with the required provider and HCP Terraform configuration block. The HCP Vault Secrets data source requires the hcp provider at version 0.63.0 or greater.

Note

This tutorial was tested using version 0.91.0 of the HCP Terraform provider. You can check for the latest version of the provider in the Terraform Registry.

$ tee terraform.tf <<EOF
terraform {
  cloud {
    organization = "$HCP_ORG"

    workspaces {
      name = "HCPVaultSecretsLab"
    }
  }
  required_providers {
    hcp = {
      source = "hashicorp/hcp"
      version = "0.91.0"
    }
  }
}

provider "hcp" {
  # Configuration options
  client_id = var.HCP_CLIENT_ID
  client_secret = var.HCP_CLIENT_SECRET
}

EOF

Declare the HCP_CLIENT_ID and HCP_CLIENT_SECRET variables in the Terraform configuration.

$ tee variables.tf <<EOF
variable "HCP_CLIENT_ID" {
  type = string
}

variable "HCP_CLIENT_SECRET" {
  type = string
}

EOF

Add the hcp_vault_secrets_app data block to the Terraform configuration to read the WebApplication application you created in an earlier tutorial.
```
$ tee main.tf <<EOF
data "hcp_vault_secrets_app" "web_application" {
  app_name = "WebApplication"
}

EOF
```
Add an output block to allow Terraform to display the retrieved secret. This step is not necessary for production configurations, but used in this tutorial to validate that Terraform successfully retrieved the secret.
Because you will use HCP Vault Secrets to store sensitive information, you must add sensitive = true to the output block.
```
$ tee -a outputs.tf <<EOF
output "secrets" {
  value = data.hcp_vault_secrets_app.web_application.secrets
  sensitive = true
}
EOF
```

$ terraform login
---------------------------------------------------------------------------------

Terraform must now open a web browser to the tokens page for app.terraform.io.

If a browser does not open this automatically, open the following URL to proceed:
    https://app.terraform.io/app/settings/tokens?source=terraform-login

---------------------------------------------------------------------------------

Generate a token using your browser, and copy-paste it into this prompt.

Terraform will store the token in plain text in the following file
for use by subsequent commands:
    /Users/username/.terraform.d/credentials.tfrc.json

Token for app.terraform.io:
  Enter a value:

Retrieved token for user username

...snip...

Run terraform init to initialize the configuration and install the necessary providers.

$ terraform init

Initializing HCP Terraform...

Initializing provider plugins...
- Finding hashicorp/hcp versions matching "0.63.0"...
- Installing hashicorp/hcp v0.63.0...
- Installed hashicorp/hcp v0.63.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

HCP Terraform has been successfully initialized!

You may now begin working with HCP Terraform. Try running "terraform plan" to
see any changes that are required for your infrastructure.

If you ever set or change modules or Terraform Settings, run "terraform init"
again to reinitialize your working directory.

Run terraform apply to read the WebApplication values from HCP Vault Secrets.

$ terraform apply -auto-approve

Running apply in HCP Terraform. Output will stream here. Pressing Ctrl-C
will cancel the remote apply if it's still pending. If the apply started it
will stop streaming the logs, but will not stop the apply running remotely.

Preparing the remote apply...

To view this run in a browser, visit:
https://app.terraform.io/app/your-org/HCPVaultSecretsLab/runs/run-ps3UvBSumbg7c91C

Waiting for the plan to start...

Terraform v1.5.2
on linux_amd64
Initializing plugins and modules...
data.hcp_vault_secrets_app.WebApplication: Refreshing...
data.hcp_vault_secrets_app.WebApplication: Refresh complete after 1s [id=WebApplication]

Changes to Outputs:
  + secrets = (sensitive value)

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:
secrets = (sensitive value)

Return to the HCP Terraform portal.
On the Overview page you can see the Latest Run is in a stage of Applied.
Click See details.
Expand Apply finished.
Click the State versions created: link.
The username secret you previously added is displayed.

Verify the HCP_CLIENT_ID, HCP_CLIENT_SECRET, and HCP_PROJECT_ID environment variables for your HCP service principal are set.

$ printenv | grep HCP_
HCP_CLIENT_ID=aBcDefGx2ruWptclGVpIE7MX7uxkU4
HCP_CLIENT_SECRET=S3sAm3stxKjziNhjTWWe9QxEffKyTdiGlxE_suNsh1n3
HCP_PROJECT_ID=11eb2a33-4567-1234-ba99-0242ac11000e

Create an example Terraform configuration with the required provider. The HCP Vault Secrets data source requires the hcp provider at version 0.63.0 or greater.
Note
This tutorial was tested using version 0.91.0 of the HCP Terraform provider. You can check for the latest version of the provider in the Terraform Registry.
```
$ tee terraform.tf <<EOF
terraform {
  required_providers {
    hcp = {
      source = "hashicorp/hcp"
      version = "0.91.0"
    }
  }
}

provider "hcp" {
  # Configuration options
}

EOF
```
Add the hcp_vault_secrets_app data block to the Terraform configuration to read the WebApplication application you created in an earlier tutorial.
Note
"web_application" is the name given to the Terraform data source. "WebApplication" is the reference to the specific HCP Vault Secrets app name.
```
$ tee main.tf <<EOF
data "hcp_vault_secrets_app" "web_application" {
  app_name = "WebApplication"
}

EOF
```
Add an output block to allow Terraform to display the retrieved secret. This step is not necessary for production configurations, but used in this tutorial to validate that Terraform successfully retrieved the secret.
Because you use HCP Vault Secrets to store sensitive information, you must add sensitive = true to the output block.
```
$ tee outputs.tf <<EOF
output "secrets" {
  value = data.hcp_vault_secrets_app.web_application.secrets
  sensitive = true
}
EOF
```

Run terraform init to initialize the configuration and install the necessary providers.

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/hcp versions matching "0.63.0"...
- Installing hashicorp/hcp v0.63.0...
- Installed hashicorp/hcp v0.63.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Run terraform apply to read the WebApplication values from HCP Vault Secrets.

$ terraform apply -auto-approve
data.hcp_vault_secrets_app.WebApplication: Reading...
data.hcp_vault_secrets_app.WebApplication: Read complete after 1s [id=WebApplication]

Changes to Outputs:
  + secrets = (sensitive value)

You can apply this plan to save these new output values to the Terraform state, without changing any real infrastructure.

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

secrets = <sensitive>

Run terraform output secrets to display the secret values.
```
$ terraform output secrets
tomap({
  "username" = "db-user"
})
```
The username secret you previously added is displayed.

Help and reference

Retrieve a secret

Create Kubernetes Secrets with VSO