Automated Install (AWS)
Note
HashiCorp provides validated Terraform modules to install Terraform Enterprise on AWS and will add modules for GCP and Azure in due course. This initial version of the HVD will, correspondingly, provide support only for AWS. This version also only provides direct recommendations for Docker; we will add GCP, Azure and Kubernetes (EKS, GKE, AKS) in subsequent releases of this HVD.AWS Architecture
As detailed in the architecture section of this HVD, we recommend deploying Terraform Enterprise Flexible Deployment Options. This architecture is optimized for high availability and features the following:
Deployment of the Terraform Enterprise container on compute instances that are part of an autoscaling group spanning three availability zones (AZ) and significantly reduces the overhead in managing those services including patching/upgrading etc.
Leveraging AWS's managed versions of PostgreSQL database and clustered Redis cache (note, not specifically Redis Cluster), that ensures replicas are distributed in different AZs.
Use of a layer 4 load balancer to ingress traffic to the Terraform Enterprise instances. This is done as:
Certificates need to be on the compute nodes for FDO to work.
It is more secure to terminate the TLS connection on Terraform Enterprise rather than outside it and have to re-encrypt traffic from the inside of the load balancer requiring an additional certificate.
It is more straight forward to manage than a layer 7 load balancer.
By using multiple AZs, the system has an n-2 failure profile, surviving failure of two AZs. However, if the entire region is unavailable, then there will be an outage of Terraform Enterprise. Currently, the application architecture is single-region.
We do not recommend that Terraform Enterprise be exposed to the public Internet. Users should be on the company network to be able to access the Terraform Enterprise API/UI. However, we recommend Terraform Enterprise is allowed to access certain addresses on the Internet in order to access:
The HashiCorp container registry - where the Terraform Enterprise container is vended from
HashiCorp service APIs (all owned and exclusively operated by HashiCorp except Algolia):
registry.terraform.io
houses the public Terraform module registry which enterprise customers will want to avoid letting users have unfettered access to (see below). However, this is where official providers are indexed.releases.hashicorp.com
is where HashiCorp host Terraform binary releases. We recommend users stay within two minor releases of current in order to access the latest security updates, and new features.reporting.hashicorp.services
is where we aggregate license usage and as such we strongly recommend including this in egress allow lists in order to ensure our partnership with your organization can be right-sized for your needs going forward.- Algolia - this enables Terraform Enterprise to access search facilities.
Additional outbound targets for VCS/SAML etc. depending on the use case.
Public cloud cost estimation APIs as necessary.
In some regulated environments, outbound access is limited or totally inaccessible from server environments. If you need to run Terraform Enterprise in a fully Internet-air gapped mode, it would be necessary to regularly, manually download and host provider and Terraform binary versions in the Terraform Enterprise registry when these are released, in order to offer them to your users.
In order to allow Terraform Enterprise access to the public registry, but prevent your user base from accessing community content, we recommend using Sentinel or OPA as part of platform development in order to limit which providers are signed-off for use.
Terraform modules for installation
The installation code we provide has been used by our partners and HashiCorp professional services to set up best practice Terraform Enterprise instances. We highly recommend leveraging partner or HashiCorp professional services help to accelerate a robust installation process if possible.
If you will be installing Terraform Enterprise yourself we recommend that you follow these high-level steps:
- Import the provided Terraform Enterprise modules into your VCS repository.
- Establish where to store Terraform state. HashiCorp recommends that you store state in HCP Terraform (free access is available or contact your HashiCorp account team for an entitlement for state storage just for Terraform Enterprise installs). If access to HCP Terraform is not possible we recommend using one of the recommend state storage and locking mechanisms documented here. Note: We do not recommend that you store state in VCS or any unprotected location. Terraform state contains sensitive information and needs to be protected. The state from the Terraform Enterprise deployment is the only state you will need to separately secure irrespective of the use of HCP Terraform as all other state generated by your organization will be protected by Terraform Enterprise.
- Select a machine where the Terraform code will be executed. This machine will need to have the Terraform CLI available.
- Ensure that cloud credentials are available on the machine for Terraform execution.
Overview of the process
Your HashiCorp account team should have provided you with a zip containing the Terraform modules used to deploy AWS instances of Terraform Enterprise along with this document. If you do not have this, please contact your HashiCorp representative.
To deploy Terraform Enterprise on AWS using the provided modules, you will need to:
- Ensure you have the TLS certificate and private key for your Terraform Enterprise installation. We expect you will have a standard organizational CA and process for generating these which we recommend using; we include a certificate helper for reference.
- Deploy the prerequisites with the
terraform-aws-tfe-prerequisites
module. - Deploy Terraform Enterprise with the
terraform-aws-tfe
module.
If you have the prerequisites already installed, then the terraform-aws-tfe-prerequisites
module is unnecessary. You can directly proceed to use terraform-aws-tfe
to install Terraform Enterprise. We recommend that you set aside time to thoroughly review the code in both modules as necessary before executing them.
The provided Terraform modules have configurable options to create new AWS resources or reference existing AWS resources. For example, the Terraform Enterprise on AWS prerequisites module contains a conditional variable to determine if the deployment creates an AWS VPC via use of the conditional variable create_vpc
which is true
by default. This should be set to false
in the module configuration if you intend to deploy Terraform Enterprise into an already-existing VPC. When the conditional variable is true, the expected behavior is to create a new VPC for Terraform Enterprise.
You must provide the ID of the target VPC for Terraform Enterprise in the terraform-aws-tfe
module configuration. In this particular case, the variable vpc_id
contains the value for the VPC ID. Note the sample excerpt from a terraform.auto.tfvars
configuration file:
# Excerpt from terraform.auto.tfvars
create_vpc = false
vpc_id = "vpc-1a23456b789c01def"
Running the prerequisites module will result in a series of outputs, depending on the use of these conditional variables. Some of the outputs contain values which are required to be included in the terraform.auto.tfvars
file used by the terraform-aws-tfe
module run. See below for details on this.
Review important security groups and ports
Security group | Description |
---|---|
TLS | Allows TLS inbound traffic. |
EC2 | Allows port 8201 inbound (Vault) and port 443 application traffic. |
Redis | Controls Redis traffic between the Redis Replication Group and the Terraform Enterprise instances. By default it uses TCP port 6379. |
PostgreSQL | Controls PostgreSQL traffic between the database cluster and the Terraform Enterprise instances. By default it uses TCP port 5432. |
Terraform Enterprise license
Before starting the Terraform Enterprise installation, make sure you have a valid Terraform Enterprise license. If you do not, first reach out to your HashiCorp account team to request the license file.
Ensure that the license file contains a single line with no new line character. Running wc -l terraform.hclic
should return 0
.
Terraform Enterprise deployment steps using Terraform modules
The following steps assume both the prerequisites and the main Terraform module are required in full. Ensure to read the README.md files in both modules' root directories, and those in the examples/hvd-tfe
subdirectories to understand where modifications need to be made if you already have prerequisites deployed.
Create the project directory
Create the deployment directories to store the deployment module code, un-compress the zip received from HashiCorp and move the two modules into place:
mkdir -p fdo-deployment/{1-certificate,2-prerequisites,3-tfe}
mkdir -p fdo-deployment/{2-prerequisites,3-tfe}/modules
unzip terraform-aws-tfe-bundle-v1.x.x.zip
mv terraform-aws-tfe-prerequisites fdo-deployment/2-prerequisites/modules
mv terraform-aws-tfe fdo-deployment/3-tfe/modules
tree -L 5
.
|-- fdo-deployment
| |-- 1-certificate
| |-- 2-prerequisites
| | `-- modules
| | `-- terraform-aws-tfe-prerequisites
| | |-- LICENSE
| | |-- README.md
| | |-- Taskfile.yaml
| | |-- examples
| | |-- images
| | |-- main.tf
| | |-- modules
| | |-- outputs.tf
| | |-- pkg_modules.json
| | |-- variables.tf
| | `-- versions.tf
| `-- 3-tfe
| `-- modules
| `-- terraform-aws-tfe
| |-- LICENSE
| |-- README.md
| |-- examples
| |-- main.tf
| |-- modules
| |-- outputs.tf
| |-- pkg_modules.json
| |-- templates
| |-- variables.tf
| `-- versions.tf
`-- terraform-aws-tfe-bundle-v1.0.0.zip
Optionally generate TLS certificate and private key
This section provides instructions on generation of a TLS certificate and private key for use with Terraform Enterprise should your organization not have one available. If you already have a TLS generation process, skip this section.
If necessary, copy the certificate helper code:
cp fdo-deployment/2-prerequisites/modules/terraform-aws-tfe-prerequisites/\
examples/hvd-tfe/supplemental-modules/generate-cert/* fdo-deployment/1-certificate
The directory structure of fdo-deployment/1-certificate
will be as follows.
.
|-- README.md
|-- main.tf
|-- outputs.tf
|-- providers.tf
|-- terraform.auto.tfvars.example
|-- variables.tf
`-- versions.tf
Set up HCP Terraform
This step is optional and will set up workspaces in HCP Terraform to store your state files.
- Log in to HCP Terraform.
- Create a new project named
hvd-tfe-deployment
. - Create the workspaces according to the table below. Set the workspace for local execution mode.
Workspace Name | Purpose |
---|---|
tfe-instance-certificates | The workspace to manage the certificates |
tfe-instance-prerequisites | The workspace to manage the Terraform Enterprise infrastructure prerequisites |
tfe-instance | The workspace to manage the Terraform Enterprise instance |
Create certificate
This step is optional and will create the necessary TLS certificate for Terraform Enterprise.
First change directory:
cd fdo-deployment/1-certificate
Configure backend.tf
This step is optional and required if you are using HCP Terraform to store your state file.
Create the backend.tf
file:
touch backend.tf
Add this content to backend.tf
:
terraform {
cloud {
organization = "<YOUR TFC ORGANIZATION NAME>"
workspaces {
name = "tfe-instance-certificate"
}
}
}
Set the parameters
Copy the example terraform.auto.tfvars
file:
cp terraform.auto.tfvars.example terraform.auto.tfvars
Update the terraform.auto.tfvars
file with your values:
route53_zone_name = "<your Route 53 zone name>"
region = "<target AWS region>"
cert_fqdn = "<FQDN for your Terraform Enterprise installation>"
email_address = "<contact email address>"
Initialize Terraform
terraform init
Run a Terraform plan
terraform plan
Review the plan for any unexpected message.
Run a Terraform apply
terraform apply --auto-approve
The apply operation
will create three files:
tfe-no-root.pub
tfe.key
tfe.pub
The directory should look like this:
.
|-- README.md
|-- backend.tf
|-- main.tf
|-- outputs.tf
|-- providers.tf
|-- terraform.auto.tfvars
|-- terraform.auto.tfvars.example
|-- tfe-no-root.pub
|-- tfe.key
|-- tfe.pub
|-- variables.tf
`-- versions.tf
Terraform Enterprise prerequisites
This step will create the infrastructure resources that are required to run Terraform Enterprise (network, database, etc.).
First, change directory and copy the code example:
cd fdo-deployment/2-prerequisites
rsync -avzh --exclude=".*" --exclude="supplemental-modules" \
modules/terraform-aws-tfe-prerequisites/examples/hvd-tfe/* .
The directory structure of fdo-deployment/2-prerequisites
will look like this:
tree -L 2
.
|-- README.md
|-- main.tf
|-- modules
| `-- terraform-aws-tfe-prerequisites
|-- outputs.tf
|-- providers.tf
|-- terraform.auto.tfvars.example
|-- variables.tf
`-- versions.tf
Adjust the following line in the pre_req_primary
module call in the main.tf
file as follows:
From | To |
---|---|
source = "../../" | source = "./modules/terraform-aws-tfe-prerequisites" |
Copy your Terraform Enterprise licence file (terraform.hclic
) to fdo-deployment/2-prerequisites
.
Optionally configure backend.tf
This step is optional and required if you are using HCP Terraform to store your state file.
Create the backend.tf
file:
touch backend.tf
Replace the content of backend.tf
with the example code taken from TFC:
terraform {
cloud {
organization = "<YOUR TFC ORGANIZATION NAME>"
workspaces {
name = "fdo-instance-prerequisites"
}
}
}
Set the parameters
Copy the example tfvars
file:
cp terraform.auto.tfvars.example terraform.auto.tfvars
Update the terraform.auto.tfvars
file with your values for the following parameters:
friendly_name_prefix = "<enter a prefix for cloud objects>"
region = "<target AWS region>"
ssh_public_key = "<replace example with your ssh public key>"
route53_zone_name = "<your Route 53 zone name>"
db_password = "<replace example with a unique and guarded password>"
redis_password = "<replace example with a unique and guarded password>"
secretsmanager_secrets = {
tfe = {
license = {
name = "tfe-license"
path = "/path/to/terraform.hclic"
}
...
cert_pem_secret = {
name = "cert_pem_public"
path = "/path/to/cert.pem"
}
cert_pem_private_key_secret = {
name = "cert_pem_private"
path = "/path/to/key.pem"
}
ca_certificate_bundle = {
name = "cert_pem_bundle"
path = "/path/to/ca_bundle.pem"
}
}
You will see in the example tfvars
file that we have included cidr_blocks
set to 0.0.0.0/0
for initial convenience. We do not recommend leaving this CIDR in place and recommend it is set to limit access according to your organisational policy, such as, but not limited to, 10.0.0.0/16
.
Optionally update main.tf to create the ASG service IAM role
This step is optional and only needed if you have never created an ASG in the target AWS account.
If you run terraform plan
and you get the following error message, then you are missing the IAM role and should follow these instructions:
Error: reading IAM Role (AWSServiceRoleForAutoScaling): couldn't find resource
with module.pre_req_primary.module.iam[0].data.aws_iam_role.asg_role,
on .terraform/modules/pre_req_primary.iam/main.tf line 19, in data "aws_iam_role" "asg_role":
19: data "aws_iam_role" "asg_role" {
Set the create_asg_service_iam_role
to true
in the IAM section of main.tf
:
module "pre_req_primary" {
...
# Required if you have never created an ASG in your aws account before
create_asg_service_iam_role = true
...
}
Initialize Terraform
terraform init
Run a Terraform plan
terraform plan
Review the plan for any unexpected message.
Run a Terraform apply
terraform apply --auto-approve
The apply should complete and return a large list of outputs. Several of these are required for configuring the terraform-aws-tfe
module for deployment in the next section. Most of the outputs are for information only.
Deploy the Terraform Enterprise instance
This step will deploy the Terraform Enterprise instance.
First move to the proper directory and copy the example:
cd ../../fdo-deployment/3-tfe
rsync -avzh --exclude=".*" modules/terraform-aws-tfe/examples/hvd-tfe/* .
The directory structure of fdo-deployment/3-tfe
will look like this:
tree -L 2
.
|-- README.md
|-- main.tf
|-- modules
| `-- terraform-aws-tfe
|-- outputs.tf
|-- providers.tf
|-- terraform.auto.tfvars.example
|-- variables.tf
`-- versions.tf
Optionally configure backend.tf
This step is optional and required if you are using HCP Terraform to store your state file.
Create the backend.tf
file:
touch backend.tf
Replace the content of backend.tf
with the example code taken from TFC:
terraform {
cloud {
organization = "<YOUR TFC ORGANIZATION NAME>"
workspaces {
name = "tfe-instance"
}
}
}
Set the parameters
Copy the example tfvars
file:
cp terraform.auto.tfvars.example terraform.auto.tfvars
Update the terraform.auto.tfvars
file with your values. Work through the tfvars
file, replacing the following parameters from the Terraform outputs returned by the prerequisites run above:
friendly_name_prefix = "<enter a prefix for cloud objects>"
region = "<use the same region as for the prerequisites>"
tfe_hostname = "<fqdn of resulting TFE instance>"
asg_hook_value = "<paste prerequisite output: asg_hook_value>"
ssh_keypair_name = "<paste prerequisite output: ssh_keypair_name>"
ec2_subnet_ids = [
"<paste prerequisite output: private_subnet_ids[0]>"
"<paste prerequisite output: private_subnet_ids[1]>"
"<paste prerequisite output: private_subnet_ids[2]>"
]
vpc_id = "<paste prerequisite output: vpc_id>"
db_cluster_endpoint = "<paste prerequisite output: db_cluster_endpoint>"
redis_host = "<paste prerequisite output: redis_primary_endpoint>"
redis_security_group_id = "<paste prerequisite output: redis_security_group_id>"
tfe_secrets_arn = "<paste prerequisite output: tfe_secrets_arn>"
tfe_cert_secret_arn = "<paste prerequisite output: cert_pem_secret_arn>"
tfe_privkey_secret_arn = "<paste prerequisite output: cert_pem_private_key_secret_arn>"
ca_bundle_secret_arn = "<paste prerequisite output: ca_certificate_bundle_secret_arn>"
iam_profile_name = "<paste prerequisite output: iam_role_name>"
s3_app_bucket_name = "<paste prerequisite output: s3_tfe_app_bucket_name>"
s3_log_bucket_name = "<paste prerequisite output: s3_log_bucket_name>"
kms_key_arn = "<paste prerequisite output: kms_key_arn>"
lb_type = "network"
lb_tg_arns = ["<paste prerequisite output: lb_tg_arns>"]
cloudwatch_log_group_name = "<paste prerequisite output: cloudwatch_log_group_name>"
Update main.tf
Adjust the main.tf
file as follows:
From | To |
---|---|
source = "../../" | source = "./modules/terraform-aws-tfe |
Initialize Terraform
terraform init
Run a Terraform plan
terraform plan
Review the plan for any unexpected message. Expect to see far fewer resources deployed by this module than that of the prerequisites.
Run a Terraform apply
terraform apply --auto-approve
It will take a few minutes for all services to start, and the successful run should output the URL of the service for your browser.