Manual install (private cloud)

Our Terraform-based Terraform Enterprise deployment modules focus on public cloud deployments because public CSPs offer a predictable set of services. Even though the overwhelming majority of our private cloud customers use VMware, there is still a bespoke approach to deployment, instead using a wide variety of additional configuration management technologies and deployment approaches to manage software installation. As such, an opinionated private cloud deployment Terraform module would be unlikely to be sufficiently useful for all private cloud customers.

This section thus provides recommendations and best practices for scaled, enterprise private cloud customers who need to apply their organisation-standard tooling to the deployment of Terraform Enterprise. If you are deploying Terraform Enterprise to a public cloud, refer to the Terraform-based Terraform Enterprise installation section for instructions.

Deployment topology

As listed in the architecture section, HashiCorp recommends the Active/Active deployment topology for customers seeking to guarantee high availability, resilience and scale. As an automated deployment is a requirement of the active-active operational mode, engineering to provide this must be a headline planning item for the project manager.

Component considerations

This section discusses a series of considerations specific to active-active private cloud installations irrespective of whether you deploy Terraform Enterprise using Docker or Kubernetes.

Redis

As a result of there being multiple concurrent instance nodes of Terraform Enterprise, your organization must consider the deployment of a Redis instance in your private datacenter as this is a hard requirement for active-active. We recommend customers for whom this proves impossible to consider a public cloud deployment, or HCP Terraform if SaaS is a viable option for your organization.

To avoid arranging product ownership and ongoing maintenance of Redis, the only self-managed alternative is to use the external operational mode instead, but this means there being only one Terraform Enterprise container in the instance. In certain circumstances, an orchestrator, such as Kubernetes, can restart the container fast enough for your needs, and your RTO dictates this as a viable possibility, then you can avoid a separate Redis instance. In our experience however, your business requires higher availability and lower RTO over time, so we recommend, forward planning for Redis support on-premise in due course.

PostgreSQL

Many customers use alternatives to PostgreSQL as their strategic database solutions. Terraform Enterprise depends specifically on PostgreSQL, which means a private cloud deployment requires that your organization has a pattern for the deployment and production management of PostgreSQL at the versions listed on our website(opens in new tab).

Also be aware that there are specific schema requirements in the preceding link, so we advise liaising in the first instance with your DBA team so that all parties are aware of the requirements at the outset.

S3-compatibility

In addition to Redis and PostgreSQL, Terraform Enterprise requires S3-compatible storage to operate. In a private datacenter scenario, this means adopting a third-party technology to present an S3-compatible API to Terraform Enterprise.

Traditionally, HashiCorp has pointed customers who do not already have a solution in place to the most-used examples, however it is not for us to comment on the suitability of one solution for your business over another.

From working with our customers, we see significant success with private cloud deployments of Terraform Enterprise using either Dell ECS or MinIO as the object storage tier. See here(opens in new tab) for MinIO setup guide for Terraform Enterprise. If your organization has a pattern for on-premise S3-compatible object storage, our recommendation is to use that.

Compatibility and components sizing

Compute

We recommend deploying Terraform Enterprise on one of the following operating systems:

• Red Hat Enterprise Linux
• Ubuntu LTS

Sizing

Each instance of the Terraform Enterprise application service runs as a container in Docker/Kubernetes. The following compute requirements are appropriate for most initial production deployments, or for development/testing environments.

• CPU - 4 vCPU
• Memory - 32 GB RAM
• Disk - 1 TB

We recommend configuring the CPU and memory availability identically for each Terraform Enterprise application instance. Ultimately, actual computational workload requirements reflect your organization's needs, but these are safe limits HashiCorp uses in scaling tests.

Many of our scaled customers use 8 vCPU and 32 GB RAM as an initial production specification. Either way, expect to scale past this as you begin to scale out.

When allocating CPU and memory resources, it is essential to consider the computational needs of the VM(s) and any adjacent containers running in the same orchestration instance. Thus, the CPU and memory allocation of your VM instances must be higher than the maximum configured for the Terraform Enterprise application. Consult the Terraform Operations Guides for more information on ongoing right-sizing and scale out of your Terraform Enterprise instances.

The disk space used by the object store depends on two things.

1. The number of workspace runs - each run generates objects in the object store irrespective of whether the run succeeds or fails.
2. The size of configuration directories submitted to the API during Terraform runs. Terraform runs aggregate the configuration directory and sends it to the API which, in turn, commit it to encrypted object storage using an internal process called archivist. We recommend limiting your configuration repositories to containing only necessary files needed for the Terraform workspace run to proceed. A 50 MB .tgz file in a configuration directory needlessly adds 50 MB to your object store and wastes hosting costs.

Disk

The performance of disk storage available to the Linux operating environment plays a role in the performance of Terraform Enterprise. We recommend using a disk with a minimum of 3000 IOPS to support data operations.

Terraform Enterprise requires a disk with at least 40 GB of available space. For Docker deployments this space must be available to /var/lib/docker. This limit does not apply to Kubernetes deployments.

PostgreSQL sizing

We recommend the following minimum specification for the DB server. Terraform Enterprise requires PostgreSQL version 13.x and later. We recommend versions later than 15.x for new installations because some versions of 14.x are not supported(opens in new tab). Support(opens in new tab) for version 17.x runs until the end of 2029 so if your organization supports this version, this would make the most logical choice.

• CPU - 4 core
• Memory - 32 GB RAM
• Disk - 2 TB

Redis cache sizing

The supported Redis cache versions are 6.2 and later and version 7. We recommend the latter. Redis Sentinel is not supported.

Sizing

We recommend the following characteristics for the cache.

• CPU - 4 core
• Memory - 16 GB RAM
• Disk - 500 GB

Monitoring PostgreSQL and Redis servers

As a best practice we recommend you monitor CPU, memory, available disk space and disk IO using your standard organizational machine telemetry solution. Create alerts for threshold breaches of 50% and 70% utilization. If utilization of any of the parameters exceeds 70% consistently, consider increasing the resource to ensure optimal performance. We also recommend ongoing telemetry reporting, so the team owning the instance are aware of resource consumption.

Networking requirements

Terraform Enterprise has a number of ingress and egress requirements. Please refer to the network requirements page(opens in new tab) for the latest information. If a corporate proxy service is filtering outbound Internet traffic, then add the destinations detailed in the preceding link to your allow-list.

Use a layer 4 load balancer with Terraform Enterprise.

Installation process - Docker

Write an automated Terraform Enterprise installation process which conforms to your organizational requirements. We use Bash for illustration. Write required tasks into automation so that each machine starts up and installs the software the same way.

We recommend reading through the below prior to running the steps manually as part of your planning for automation.

Prepare requirements

The following list summarizes the infrastructure resources required prior to the deployment of Terraform Enterprise.

1. VMs for installation of Terraform Enterprise
2. PostgreSQL database
3. Redis cache
4. S3-compatible object storage
5. Terraform Enterprise license file (if you already have a license file but are on a Replicated-backed instance, you need a replacement file, but the contract is the same)
6. TLS certificate bundle
7. Load balancer

Tasks

As root, perform the equivalent of these commands:

sudo mkdir -p /opt/tfe/certs
cd !$

Put the PEM-encoded server certificate, private key and CA bundle files in this directory. For the sake of this installation, we assume the following filenames cert.pem, key.pem and bundle.pem.

tfe
`-- certs
    |-- bundle.pem
    |-- cert.pem
    `-- key.pem

Create a Docker Compose configuration

Write a Docker configuration with the consolidated information about your active-active environment. Base this compose.ymlfile on this one(opens in new tab). Note that you must be using on-premise S3-compatible object storage, so use the TFE_OBJECT_STORAGE_* variables listed in the link as you would were you to deploy Terraform Enterprise on AWS.

In the configuration properties, we want to highlight the following:

• Treat the TFE_ENCRYPTION_PASSWORD value as a sensitive secret. We recommend using a password generation utility/process to follow corporate policies. Terraform uses this password to decrypt the Vault token stored in the database; Terraform Enterprise uses the Vault token to do all subsequent encryption/decryption processing.
• Terraform Enterprise uses an embedded instance of HashiCorp Vault for data security operations. When identifying the TFE_VAULT_CLUSTER_ADDRESS for the Vault instance, point to the private IP of the VM. The internal instrumentation of Vault performs a discovery to find peer instances and add to a private cluster so the IP addresses of all nodes in the cluster need to be accessible to all the other nodes.
• The TFE_DATABASE_HOST points to your PostgreSQL database cluster endpoint name.
• The TFE_REDIS_HOST points to the endpoint of the primary node in the Redis Replication Group.
• The TFE_LICENSE entry requires a raw string from your Terraform Enterprise license file. The license string is the data.attributes.signed_string object.

This is not an exhaustive list of configuration options. Please refer to the configuration reference(opens in new tab) for documentation about all the configuration options.

Write the compose.yml to a location where your nascent application machines can access it - each of them needs to use it for the installation.

Configure systemd

We recommend using the following systemd manifest to run docker compose on system start, so that if the machine reboots, Terraform Enterprise automatically starts. Create /etc/systemd/system/terraform-enterprise.service containing

[Unit]
Description=Terraform Enterprise Service
Requires=docker.service
After=docker.service network.target
 
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/etc/terraform-enterprise
ExecStart=/usr/local/bin/docker compose up -d
ExecStop=/usr/local/bin/docker compose down
TimeoutStartSec=0
 
[Install]
WantedBy=multi-user.target

then run

systemctl enable --now terraform-enterprise

Download the Terraform Enterprise image

First, login to the Terraform Enterprise container image registry using

cat <PATH_TO_HASHICORP_LICENSE_FILE> \
| docker login --username terraform images.releases.hashicorp.com --password-stdin

If you do not receive the output Login Succeeded using the provided license, contact your HashiCorp representative for support.

Next, pull the container image. Use the form docker pull images.releases.hashicorp.com/hashicorp/terraform-enterprise:<vYYYYMM-#> where <vYYYYMM-#> is the release required for example:

docker pull images.releases.hashicorp.com/hashicorp/terraform-enterprise:v202309-1

Check the releases page(opens in new tab) for the suitable version. We recommend using last month's latest release - if it is October, get v202309-x where x is the highest listed integer.

If necessary, you can pull the image and push it to your private Docker registry. When placing the image in a private Docker registry ensure to re-tag the image to support your hierarchical organization standards.

Run Docker Compose

Place the compose.yml in your working directory. When done, the directory structure must look like this:

tfe
|-- certs
|   |-- bundle.pem
|   |-- cert.pem
|   `-- key.pem
`-- compose.yml

Each node then needs to run

sudo docker compose up --detach

Messages indicating the progress of the running process result. Follow logs using this in a separate shell:

docker compose logs --follow

Run a health check

Terraform Enterprise provides a /_health_check endpoint on the instance. Expect a 200 OK response from a healthy system.

The /_health_check endpoint operates in 2 modes:

• Full check with /_health_check?full=1
• Minimal check with /_health_check

With a full check, the service attempts to verify the status of internal components and PostgreSQL, in contrast to a minimal check which returns 200 OK automatically after a successful full check. To run a health check, use:

sudo docker exec terraform-enterprise curl http://127.0.0.1:9292/_health_check?full=1

The command returns a JSON encoded message as follows:

{"postgres":"UP","redis":"UP","vault":"UP"}

Use the following or equivalent in your code to pause the deployment process until all Terraform Enterprise processes are healthy.

while ! docker compose exec terraform-enterprise curl http://127.0.0.1:9292/_health_check?full=1;
do
  echo "Waiting for Terraform Enterprise to be ready"
  sleep 30
done

The default behavior of this endpoint is to perform a full check during startup of the instance, and minimal checks after Terraform Enterprise is active and running. The Terraform Enterprise application supports a practical tfe-health-check-status command to provide full visibility. You can run the command in the VM as follows:

sudo docker exec terraform-enterprise tfe-health-check-status

When started, a healthy Terraform Enterprise instance returns messages as follows:

checking: Archivist Health Check...
|  checks that Archivist is up and healthy
|- ✓ PASS
 
checking: Terraform Enterprise Health Check...
|  checks that Terraform Enterprise is up and can communicate with Redis and Postgres
|- ✓ PASS
 
checking: Terraform Enterprise Vault Health Check...
|  checks that Terraform Enterprise can connect to Vault and is able to encrypt and decrypt tokens
|- ✓ PASS
 
checking: Fluent Bit Health Check...
|  checks that the configure Fluent Bit server is healthy
|- SKIPPED
 
checking: RabbitMQ Health Check...
|  checks that RabbitMQ can be connected to and that we can send and consume messages
|- SKIPPED
 
checking: Vault Server Health Check...
|  checks that the configured Vault Server is healthy
|- ✓ PASS
 
  All checks passed.

When forcing a full check, take extra caution as every call makes requests to all external services, increasing system load and latency. This is not a concern during the initial installation stages as there are no in-flight operations expected.

Retrieve your initial administrator creation token

When you have deployed all Terraform Enterprise application instances, you can retrieve the initial administrator creation token. Choose one of your EC2 Instances and run the following command in a terminal session:

docker compose exec terraform-enterprise retrieve-iact

The command returns your initial administrator creation token. Protect this sensitive token as part of the handover process to the operating teams. The form is as follows.

5db5c3ae5be1599804bc614ec976e1a64ad1cadf76fec518d61ed9d04ef565eb

To view the startup web page, open a web browser and navigate to the Terraform Enterprise URL.

The syntax for the URL endpoint is: $URL/admin/account/new?token=$TOKEN

For example:

https://alb_entry_point.your_domain.com/admin/account/new?
token=5db5c3ae5be1599804bc614ec976e1a64ad1cadf76fec518d61ed9d04ef565eb

When the URL loads, the operator sees a form. The first user created in Terraform Enterprise becomes the site administrator. The only reason to complete this form is when the deployment team is responsible for post-installation tasks of the Terraform Enterprise deployment team.