Deployment (manual)

Our Terraform-based Terraform Enterprise deployment modules focus on public cloud deployments because public CSPs offer a predictable set of services. Even though the overwhelming majority of our private cloud customers use VMware, there is still a bespoke approach to deployment, instead using a wide variety of additional configuration management technologies and deployment approaches to manage software installation. As such, an opinionated private cloud deployment Terraform module would be unlikely to be sufficiently useful for all private cloud customers.

This section thus provides recommendations and best practices for scaled, enterprise private cloud customers who need to apply their organisation-standard tooling to the deployment of Terraform Enterprise. If you are deploying Terraform Enterprise to a public cloud, refer to the Terraform-based Terraform Enterprise installation section for instructions.

Deployment topology

As listed in the architecture section, HashiCorp recommends the Active/Active deployment topology for customers seeking to guarantee high availability, resilience and scale. As an automated deployment is a requirement of the active-active operational mode, engineering to provide this must be a headline planning item for the project manager.

Component considerations

This section discusses a series of considerations specific to active-active private cloud installations irrespective of whether you deploy Terraform Enterprise using Docker or Kubernetes.

Redis

As a result of there being multiple concurrent instance nodes of Terraform Enterprise, your organization must consider the deployment of a Redis instance in your private datacenter as this is a hard requirement for active-active. We recommend customers for whom this proves impossible to consider a public cloud deployment, or HCP Terraform if SaaS is a viable option for your organization.

To avoid arranging product ownership and ongoing maintenance of Redis, the only self-managed alternative is to use the external operational mode instead, but this means there being only one Terraform Enterprise container in the instance. In certain circumstances, an orchestrator, such as Kubernetes, can restart the container fast enough for your needs, and your RTO dictates this as a viable possibility, then you can avoid a separate Redis instance. In our experience however, your business requires higher availability and lower RTO over time, so we recommend, forward planning for Redis support on-premise in due course.

PostgreSQL

Many customers use alternatives to PostgreSQL as their strategic database solutions. Terraform Enterprise depends specifically on PostgreSQL, which means a private cloud deployment requires that your organization has a pattern for the deployment and production management of PostgreSQL at the versions listed on our website(opens in new tab).

Also be aware that there are specific schema requirements in the preceding link, so we advise liaising in the first instance with your DBA team so that all parties are aware of the requirements at the outset.

S3-compatibility

In addition to Redis and PostgreSQL, Terraform Enterprise requires S3-compatible storage to operate. In a private datacenter scenario, this means adopting a third-party technology to present an S3-compatible API to Terraform Enterprise.

Traditionally, HashiCorp has pointed customers who do not already have a solution in place to the most-used examples, however it is not for us to comment on the suitability of one solution for your business over another.

From working with our customers, we see significant success with private cloud deployments of Terraform Enterprise using either Dell ECS or MinIO as the object storage tier. See here(opens in new tab) for MinIO setup guide for Terraform Enterprise. If your organization has a pattern for on-premise S3-compatible object storage, our recommendation is to use that.

Component sizing

Terraform Enterprise and its external components follow the baseline component sizing defined in the detailed design. In a manual private-cloud deployment you provision the application hosts, PostgreSQL, and Redis yourself, so size each of them to meet or exceed that baseline. The notes below are specific to a self-managed deployment.

Operating system

We recommend deploying Terraform Enterprise on one of the following operating systems:

• Red Hat Enterprise Linux
• Ubuntu LTS

Compute

Each Terraform Enterprise application instance runs as a container. Configure CPU and memory identically across instances, and remember the VM must be sized higher than the Terraform Enterprise application limits to allow for the operating system and any adjacent containers. For ongoing right-sizing and scale-out, see the Administration Guide.

The disk backing each host must sustain a minimum of 3000 IOPS, and Terraform Enterprise requires at least 40 GB of available space; for Docker deployments this space must be available to /var/lib/docker.

Object store consumption depends on two things:

1. The number of workspace runs — each run generates objects in the object store irrespective of whether the run succeeds or fails.
2. The size of configuration directories submitted to the API during Terraform runs. Terraform aggregates the configuration directory and sends it to the API, which commits it to encrypted object storage through an internal process called archivist. Limit your configuration repositories to only the files a run needs — a 50 MB .tgz file in a configuration directory needlessly adds 50 MB to your object store and wastes hosting cost.

PostgreSQL and Redis

Provision PostgreSQL and Redis to the baseline component sizing, at the versions documented in the detailed design. Because these are self-managed in a private-cloud deployment, confirm your organization has an established pattern for deploying and operating them at the required versions before you begin.

Observability

Enable the general Terraform Enterprise metrics, logs, and agent telemetry described in Observability setup; that setup is common to all platforms. The guidance below is specific to a manual private-cloud deployment, where you own the underlying hosts and the self-managed PostgreSQL and Redis instances.

Monitor CPU, memory, available disk space, and disk I/O on the application hosts and on the self-managed PostgreSQL and Redis servers using your standard organizational machine telemetry solution. Create alerts for threshold breaches at 50% and 70% utilization; if any parameter exceeds 70% consistently, increase the resource to maintain performance. Report telemetry on an ongoing basis so the team that owns the instance is aware of resource consumption.

Ongoing, day-2 observability and alerting is covered in the Administration Guide.

Networking requirements

Terraform Enterprise has a number of ingress and egress requirements. Please refer to the network requirements page(opens in new tab) for the latest information. If a corporate proxy service is filtering outbound Internet traffic, then add the destinations detailed in the preceding link to your allow-list.

Use a layer 4 load balancer with Terraform Enterprise.

Installation process - Docker

Write an automated Terraform Enterprise installation process which conforms to your organizational requirements. We use Bash for illustration. Write required tasks into automation so that each machine starts up and installs the software the same way.

We recommend reading through the below prior to running the steps manually as part of your planning for automation.

Prepare requirements

The following list summarizes the infrastructure resources required prior to the deployment of Terraform Enterprise.

1. VMs for installation of Terraform Enterprise
2. PostgreSQL database
3. Redis cache
4. S3-compatible object storage
5. Terraform Enterprise license file (if you already have a license file but are on a Replicated-backed instance, you need a replacement file, but the contract is the same)
6. TLS certificate bundle
7. Load balancer

Tasks

As root, perform the equivalent of these commands:

sudo mkdir -p /opt/tfe/certs
cd !$

Put the PEM-encoded server certificate, private key and CA bundle files in this directory. For the sake of this installation, we assume the following filenames cert.pem, key.pem and bundle.pem.

tfe
`-- certs
    |-- bundle.pem
    |-- cert.pem
    `-- key.pem

Create a Docker Compose configuration

Write a Docker configuration with the consolidated information about your active-active environment. Base this compose.ymlfile on this one(opens in new tab). Note that you must be using on-premise S3-compatible object storage, so use the TFE_OBJECT_STORAGE_* variables listed in the link as you would were you to deploy Terraform Enterprise on AWS.

In the configuration properties, we want to highlight the following:

• Treat the TFE_ENCRYPTION_PASSWORD value as a sensitive secret. We recommend using a password generation utility/process to follow corporate policies. Terraform uses this password to decrypt the Vault token stored in the database; Terraform Enterprise uses the Vault token to do all subsequent encryption/decryption processing.
• Terraform Enterprise uses an embedded instance of HashiCorp Vault for data security operations. When identifying the TFE_VAULT_CLUSTER_ADDRESS for the Vault instance, point to the private IP of the VM. The internal instrumentation of Vault performs a discovery to find peer instances and add to a private cluster so the IP addresses of all nodes in the cluster need to be accessible to all the other nodes.
• The TFE_DATABASE_HOST points to your PostgreSQL database cluster endpoint name.
• The TFE_REDIS_HOST points to the endpoint of the primary node in the Redis Replication Group.
• The TFE_LICENSE entry requires a raw string from your Terraform Enterprise license file. The license string is the data.attributes.signed_string object.

This is not an exhaustive list of configuration options. Please refer to the configuration reference(opens in new tab) for documentation about all the configuration options.

Write the compose.yml to a location where your nascent application machines can access it - each of them needs to use it for the installation.

Configure systemd

We recommend using the following systemd manifest to run docker compose on system start, so that if the machine reboots, Terraform Enterprise automatically starts. Create /etc/systemd/system/terraform-enterprise.service containing

[Unit]
Description=Terraform Enterprise Service
Requires=docker.service
After=docker.service network.target
 
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/etc/terraform-enterprise
ExecStart=/usr/local/bin/docker compose up -d
ExecStop=/usr/local/bin/docker compose down
TimeoutStartSec=0
 
[Install]
WantedBy=multi-user.target

then run

systemctl enable --now terraform-enterprise

Download the Terraform Enterprise image

First, login to the Terraform Enterprise container image registry using

cat <PATH_TO_HASHICORP_LICENSE_FILE> \
| docker login --username terraform images.releases.hashicorp.com --password-stdin

If you do not receive the output Login Succeeded using the provided license, contact your HashiCorp representative for support.

Next, pull the container image. Use the form docker pull images.releases.hashicorp.com/hashicorp/terraform-enterprise:<vYYYYMM-#> where <vYYYYMM-#> is the release required for example:

docker pull images.releases.hashicorp.com/hashicorp/terraform-enterprise:v202309-1

Check the releases page(opens in new tab) for the suitable version. We recommend using last month's latest release - if it is October, get v202309-x where x is the highest listed integer.

If necessary, you can pull the image and push it to your private Docker registry. When placing the image in a private Docker registry ensure to re-tag the image to support your hierarchical organization standards.

Run Docker Compose

Place the compose.yml in your working directory. When done, the directory structure must look like this:

tfe
|-- certs
|   |-- bundle.pem
|   |-- cert.pem
|   `-- key.pem
`-- compose.yml

Each node then needs to run

sudo docker compose up --detach

Messages indicating the progress of the running process result. Follow logs using this in a separate shell:

docker compose logs --follow

Run a health check

Terraform Enterprise provides a /_health_check endpoint on the instance. Expect a 200 OK response from a healthy system.

The /_health_check endpoint operates in 2 modes:

• Full check with /_health_check?full=1
• Minimal check with /_health_check

With a full check, the service attempts to verify the status of internal components and PostgreSQL, in contrast to a minimal check which returns 200 OK automatically after a successful full check. To run a health check, use:

sudo docker exec terraform-enterprise curl http://127.0.0.1:9292/_health_check?full=1

The command returns a JSON encoded message as follows:

{"postgres":"UP","redis":"UP","vault":"UP"}

Use the following or equivalent in your code to pause the deployment process until all Terraform Enterprise processes are healthy.

while ! docker compose exec terraform-enterprise curl http://127.0.0.1:9292/_health_check?full=1;
do
  echo "Waiting for Terraform Enterprise to be ready"
  sleep 30
done

The default behavior of this endpoint is to perform a full check during startup of the instance, and minimal checks after Terraform Enterprise is active and running. The Terraform Enterprise application supports a practical tfe-health-check-status command to provide full visibility. You can run the command in the VM as follows:

sudo docker exec terraform-enterprise tfe-health-check-status

When started, a healthy Terraform Enterprise instance returns messages as follows:

checking: Archivist Health Check...
|  checks that Archivist is up and healthy
|- ✓ PASS
 
checking: Terraform Enterprise Health Check...
|  checks that Terraform Enterprise is up and can communicate with Redis and Postgres
|- ✓ PASS
 
checking: Terraform Enterprise Vault Health Check...
|  checks that Terraform Enterprise can connect to Vault and is able to encrypt and decrypt tokens
|- ✓ PASS
 
checking: Fluent Bit Health Check...
|  checks that the configure Fluent Bit server is healthy
|- SKIPPED
 
checking: RabbitMQ Health Check...
|  checks that RabbitMQ can be connected to and that we can send and consume messages
|- SKIPPED
 
checking: Vault Server Health Check...
|  checks that the configured Vault Server is healthy
|- ✓ PASS
 
  All checks passed.

When forcing a full check, take extra caution as every call makes requests to all external services, increasing system load and latency. This is not a concern during the initial installation stages as there are no in-flight operations expected.

Retrieve your initial administrator creation token

When you have deployed all Terraform Enterprise application instances, you can retrieve the initial administrator creation token. Choose one of your EC2 Instances and run the following command in a terminal session:

docker compose exec terraform-enterprise retrieve-iact

The command returns your initial administrator creation token. Protect this sensitive token as part of the handover process to the operating teams. The form is as follows.

5db5c3ae5be1599804bc614ec976e1a64ad1cadf76fec518d61ed9d04ef565eb

To view the startup web page, open a web browser and navigate to the Terraform Enterprise URL.

The syntax for the URL endpoint is: $URL/admin/account/new?token=$TOKEN

For example:

https://alb_entry_point.your_domain.com/admin/account/new?
token=5db5c3ae5be1599804bc614ec976e1a64ad1cadf76fec518d61ed9d04ef565eb

When the URL loads, the operator sees a form. The first user created in Terraform Enterprise becomes the site administrator. The only reason to complete this form is when the deployment team is responsible for post-installation tasks of the Terraform Enterprise deployment team.