Deploying Consul using Terraform

HashiCorp offers a set of official Terraform(opens in new tab) modules to simplify the deployment of a Consul Enterprise environment that align with the requirements and standards outlined in this HashiCorp Validated Design (HVD).

Tip

HashiCorp Professional Services or one of our trusted partners can help accelerate your Consul Enterprise deployment with confidence. For more information, reach out to your HashiCorp account team.

Platform-specific guidance

Select the tab below for your cloud for further guidance.

HashiCorp provides an official HVD module to deploy Consul Enterprise on AWS EC2(opens in new tab).

By using this module, you can deploy a complete, end-to-end Consul Enterprise environment in your cloud account using Terraform.

Note

The steps provided offer a detailed, step-by-step method for installing Consul using this Terraform module. This method is intended for testing the module’s functionality and familiarizing yourself with the design. We strongly recommend adapting these steps into a more controlled deployment process, such as using HCP Terraform, Terraform Enterprise, or integrating them into your existing CI/CD pipeline.

Warning

NEVER store Terraform state in a version control system or any other unprotected location. Terraform state files contain sensitive information and must be secured. For more information, refer to the Terraform documentation.

While we have provided prescriptive best practices throughout this guide, we understand that each organization has unique requirements and constraints for infrastructure deployment. We have included different considerations for deploying Consul in your cloud environment within the context of this module. If the default variables don’t fully meet your needs, the module offers additional capabilities you may want to explore.

Deployment sequence overview

Create the TLS certificate files.
Obtain the Consul Enterprise license file.
Download the Consul CLI.
Download the Terraform CLI.
Set up your prerequisite resources.
Configure your cloud credentials.
Initialize your Terraform workspace.
Enter the required variables into the module for deployment.
Create a Terraform plan.
Apply the Terraform plan.
Verify that the cluster has been successfully created and is accessible.

Preparation

Create the TLS certificate files

Generate a standard X.509 certificate to be installed on the Consul servers. Follow your organization’s process for creating a certificate that corresponds to the DNS record users will use to access Consul.

You will need the following three files.

The certificate: consul-crt.pem
The certificate’s private key: consul-key.pem
The CA bundle file from the certificate authority: ca.pem

Keep these files accessible, as they will be required later in the installation process.

Obtain the Consul Enterprise license file

Obtain the Consul Enterprise license file from your HashiCorp account team. This file includes a unique license key for your environment and will typically be named something like consul.hclic.

Keep this file accessible, as it will be required later in the installation process.

Download and install the Consul CLI

Note

This guide has been tested with Consul 1.19.2+ent.

Install the Consul CLI for your platform. You can do it manually by downloading the appropiate binary for your operating system, or using your preferred package manager. You can find instructions for the different options in here(opens in new tab).

Download and install the Terraform CLI

Note

This guide has been tested with Terraform 1.9.6+ent.

Install the Terraform CLI for your platform. You can do it manually by downloading the appropiate binary for your operating system, or using your preferred package manager. You can find instructions for the different options in here(opens in new tab).

Configure your cloud credentials.

Once you have downloaded the module, navigate to the examples/default/ directory. Use this as the base working directory during the installation process.

Ensure the appropriate AWS credentials are configured and accessible to Terraform. Terraform can access credentials from the following sources.

Credentials file: Typically located at $HOME/.aws/credentials (or %UserProfile%\.aws\credentials on Windows).
Environment variables:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- AWS_SESSION_TOKEN (if using an IAM role or temporary credentials)
- AWS_DEFAULT_REGION

For detailed instructions on setting up AWS credentials for Terraform, refer to the Terraform AWS provider documentation(opens in new tab).

Make sure the credentials you use have the necessary permissions to perform the required actions in Terraform.

Ensure the appropriate credentials are set and accessible to Terraform.

If you have GCloud CLI configured, you can run the following command to make your Google identity available:

gcloud auth application-default login

You can also impersonate a service account if needed. For detailed documentation on how to configure Google Cloud credentials for Terraform, refer to the Terraform Google Provider documentation(opens in new tab).

Ensure the correct Azure credentials are in place and accessible to Terraform by running az login with your assumed credentials.

For complete details on how to configure Azure credentials for Terraform, see the HashiCorp Terraform Azure provider documentation(opens in new tab).

Ensure that the credentials you will be using have sufficient permissions to perform the necessary actions that Terraform will be performing.

Installation

Set up your prerequisite resources

This section outlines the necessary infrastructure resources required to successfully execute the Terraform modules. These prerequisites include both infrastructure components and the appropriate storage locations for certificates, credentials, and licenses.

For more details about the required input variables, refer to the specific module's documentation.

HashiCorp provides an official HVD module to deploy Consul Enterprise on AWS EC2(opens in new tab).

Before deployment, you need to set up the required infrastructure in AWS.

A functional VPC with at least three public and three private subnets.
A Consul license stored in the AWS Secrets Manager.
- Store the content of the license file in a secret, using a name such as consul_license_text. The ARN of the secret will be used as an input to the Terraform module.
A TLS private key and certificate, valid for the fully qualified domain name intended for Consul, stored in the AWS Secrets Manager.
- Store the TLS certificate, private key, and CA bundle as secrets using names like consul_agent_cert, consul_agent_key, and consul_ca_cert. The ARNs of these secrets will be used as input variables for the Terraform module.
An S3 Bucket for snapshots.
- Ensure the S3 bucket is not publicly accessible and is encrypted.
SSH key name, already registered in AWS, to use for instance access.
An gossip encryption key for Consul.
- You can use the Consul CLI’s consul keygen command to generate a gossip encryption key.
An initial management token in the UUID format.
- You can generate an initial management token using the Linux/UNIX command uuidgen | tr "[:upper:]" "[:lower:]".

HashiCorp provides an official HVD module to deploy Consul Enterprise on GCP GCE(opens in new tab).

Before deployment, you need to set up the prerequisite infrastructure in GCP:

A GCP Project with required permissions.
A VPC with a subnet in a region with 3+ zones.
TLS private key and certificate, valid for the fully qualified domain name (FQDN) used with Consul, uploaded in PEM format and base64 encoded as secrets to GCP Secrets Manager.
A Consul Gossip Encryption Key uploaded as a secret to GCP Secrets Manager.
A Consul license uploaded as a secret to GCP Secrets Manager.
A GCS bucket for backup snapshot storage.

Note

If it is not possible to have GCP project owner permissions, at least you will need permissions to:
- Create IAM service accounts for the project.
- Create/Update compute resources.
- Create/Update networking resources.
- Create/Update secrets in Secrets Manager.

Note

If your network and subnetwork are in a different project than Compute instances, the Terraform module will need read permissions to get networking data from the other project.

HashiCorp provides an official HVD module to deploy Consul Enterprise on Azure VMs(opens in new tab).

Before deployment, you will need to deploy the prerequisite infrastructure in Azure.

An Azure resource group.
A virtual network with a private subnet for the Consul instances.
An Azure Key Vault to store the prerequisite secret material.
A Consul license stored as a secret in the Azure Key Vault.
- Store the base64-encoded contents of the Consul license file as a secret, naming it consul-license.
An gossip encryption key for Consul.
- Generate a gossip encryption key using the consul keygen command in the Consul CLI.
- Base64 encode the key and store it as a secret named gossip-key.
A TLS private key and certificate, valid for the fully qualified domain name (FQDN) used with Consul, uploaded as secrets to the Azure Key Vault.
- Upload the TLS certificate, private key, and CA bundle required for Consul’s FQDN to the Azure Key Vault, they also need to be base64 encoded and stored as secrets named consul_agent_cert, consul_agent_key, and consul_ca_cert.
A storage account with an object container for Consul snapshot storage.
- Store the primary access key of the storage account as a secret, using the name storage-account-key.

Initialize your Terraform workspace

Run terraform init to initialize your Terraform workspace. Review the output to confirm that all providers and modules have been successfully downloaded and that no errors are present before proceeding.

Configure variables for the deployment

Note

You can configure variables in the installation module’s terraform.tfvars file only after all prerequisite resources are set up. Values from the prerequisites will need to be supplied to the Consul module.

Create a file named terraform.tfvars in the examples/amazonlinux-internal-nlb-consul-primary/ directory. Refer to the terraform.tfvars.example file for explanations of each variable, then populate your own file with the relevant values for your environment.

Create and apply the Terraform plan

In the example directory, generate a Terraform plan using the following command:

terraform plan

Review the output to understand the changes that will be made, then apply the changes with:

terraform apply

When prompted, type yes to confirm and proceed with the changes.

Verify that the cluster has been successfully created and is accessible

Once the Terraform run completes successfully, you can find the IP addresses of your Consul servers through the AWS console or by using the following command, specifying the name of the created autoscaling group, such as production-consul-0.0.1.

aws ec2 describe-instances --instance-ids \
  $(aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-name production-consul-0.0.1 \
  --query "AutoScalingGroups[].Instances[].InstanceId" \
  --output text) \
  --query "Reservations[].Instances[].[InstanceId, PrivateIpAddress, PublicIpAddress]" \
  --output table

To connect to an EC2 instance using SSH, replace the IP address with the IP of one of the instances returned from the previous command.

ssh ec2-user@<IP-ADDRESS>

To configure the necessary environment variables for Consul, set the following values within your SSH session on the EC2 instance.

export CONSUL_CACERT=/etc/consul.d/tls/consul-ca.pem
export CONSUL_HTTP_ADDR=127.0.0.1:8501
export CONSUL_HTTP_SSL=true
export CONSUL_HTTP_TOKEN=<aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee>

To check the status of the raft peers in your Consul cluster and confirm the setup, run the following command.

consul operator raft list-peers

With the recommended configuration, this command will display the status of the 6 Consul nodes in the cluster.

Once you have successfully deployed your Terraform configuration, you can view the IP addresses of the load balancer's frontend either in the GCP console or by executing the following command:

gcloud compute forwarding-rules list --filter="name~'consul-forwarding-rule'"

By default, in the Terraform module, the load balancer is configured internally (load_balancing_scheme = "INTERNAL"). To access the Consul cluster, you'll need to use the internal IP address of the forwarding rule and connect to the VPC via VPN (using Cloud NAT) or through a bastion host. For external access, configure the Terraform module variable for the load balancer with load_balancing_scheme = "EXTERNAL" and use the public IP address of the forwarding rule.

The Consul bootstrap token will be securely stored as a secret in GCP Secret Manager during deployment. You can retrieve it using the following command:

gcloud secrets versions access latest --secret consul-consul-management-token

You could set the following environment variables to interact with the Consul cluster:

export CONSUL_HTTP_ADDR="https://$(gcloud compute forwarding-rules list --filter="name~'consul-forwarding-rule'" --format="value(IPAddress)"):8501"
export CONSUL_HTTP_TOKEN="$(gcloud secrets versions access latest --secret consul-consul-management-token)"

Also, if the CA certificate is not in your trusted store, you can access to the Consul CA certificate using the secret name used during the deployment:

gcloud secrets versions access latest --secret consul-tls-ca-cert | base64 -d > /path/to/consul-ca.pem

The previous command saved the CA certificate as a .pem file. Now set the CONSUL_CACERT environment variable to point to its location.

export CONSUL_CACERT=/path/to/consul-ca.pem

Now, you can test that the Consul cluster is running and reachable by running.

consul members

You should now have a running cluster.

You can also list and access server instances by retrieving their IP addresses using the following command:

gcloud compute instances list --filter="tags.items=consul-backend"

After the Terraform run completes successfully, you can retrieve the IP addresses of your Consul servers via the Azure portal or by running the following command, specifying the resource group and scaling set, such as consul-ent-rg and production-consul-agents.

az vmss nic list \
  --resource-group consul-ent-rg \
  --vmss-name production-consul-agents \
  | jq -r '.[0].ipConfigurations[0].privateIPAddress'

The initial Consul management token is auto-generated and stored in the Azure Key Vault under the secret name mgmt-token. Retrieve the token with this command.

az keyvault secret show \
  --vault-name "kv-servers-consul-${PREFIX}" \
  --name "mgmt-token" | jq -r .value

To SSH into one of the Azure VMs, use the IP address from the previous step, substituting <IP-ADDRESS> with the actual IP.

ssh azureuser@<IP-ADDRESS>

Within the SSH session on the VM instance, configure the following Consul environment variables.

export CONSUL_CACERT=/etc/consul.d/tls/ca.pem
export CONSUL_HTTP_ADDR=127.0.0.1:8501
export CONSUL_HTTP_SSL=true
export CONSUL_HTTP_TOKEN=<aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee>

To verify the status of raft peers in the Consul cluster, execute the following command.

consul members

With the recommended configuration, this command will display the status of the 6 Consul nodes in the cluster.

Detailed Design

Use Cases