- Terraform Enterprise
- 2.1.x (latest)
- 2.0.x
- 1.2.x
- 1.1.x
- 1.0.x
- v202507-1
- v202506-1
- v202505-1
- v202504-1
- v202503-1
- v202502-2
- v202502-1
- v202501-1
- v202411-2
- v202411-1
- v202410-1
- v202409-3
- v202409-2
- v202409-1
- v202408-1
- No versions of this document exist before v202408-1. Click below to redirect to the version homepage.
- v202407-1
- v202406-1
- v202405-1
- v202404-2
- v202404-1
- v202402-2
- v202402-1
- v202401-2
- v202401-1
- v202312-1
- v202311-1
- v202310-1
- v202309-1
- v202308-1
- v202307-1
- v202306-1
- v202305-2
- v202305-1
- v202304-1
- v202303-1
- v202302-1
- v202301-2
- v202301-1
- v202212-2
- v202212-1
- v202211-1
- v202210-1
- v202209-2
- v202209-1
- v202208-3
- v202208-2
- v202208-1
- v202207-2
- v202207-1
- v202206-1
Blue-Green Upgrades
A release series is certified as blue-green compatible when the Terraform Enterprise team has tested that both versions can run interoperably during the upgrade process and that deployments can be rolled back without major issues.
Prerequisites
Before you plan a blue-green upgrade, ensure you meet all prerequisites:
- Terraform Enterprise is running in an active-active configuration with at least two instances.
- Terraform Enterprise is not running on Nomad.
- You are an advanced Terraform Enterprise operator who is confident with the primitives of your autoscaling group, load balancer, and deployment runtime.
- (Kubernetes/OpenShift) Graceful draining is enabled as a pre-stop lifecycle method. Enabling it requires one-time downtime for a Terraform Enterprise deployment.
(Kubernetes/OpenShift) Enable graceful draining as a pre-stop lifecycle method
This requires version 2.2.1 or later of the Helm chart.
Update your values.yaml file with settings to enable graceful draining:
waitForDrain:
enabled: true
After you set this, you must conduct a Helm upgrade because changes to lifecycle hooks do not apply to existing pods. Existing pods will not wait gracefully for draining, so this requires taking the full cluster offline first. You do not need to conduct the upgrade at the same time as this. So long as this remains in your values.yaml file, this should be a one-time instance of downtime.
Whenever a pod stops, it will automatically trigger a node drain and then wait until tasks are finished. This can take up to 24 hours, but is often faster, especially if stuck tasks are investigated.
Skipping this step will mean that following the steps below with the Helm chart will ungracefully terminate pods, which can lead to unexpected errors.
You can confirm that pre-stop is enabled on a Kubernetes pod:
$ kubectl get pod <POD_NAME> -n <NAMESPACE> -o yaml
Under spec.containers, find the Terraform Enterprise container and confirm that its lifecycle.preStop hook contains the configured pre-stop command. Also confirm that spec.terminationGracePeriodSeconds is set to 86400, which is 24 hours.
Blue-green planning checklist
To conduct a rolling, blue-green, zero-downtime upgrade, follow this checklist and the steps in the following sections.
- Select a target version and confirm its compatibility with your environments and with blue-green upgrades.
- Take a live backup of PostgreSQL and Redis (blob storage is not required)
- Download the container images for the target release
- Run the pre-upgrade checks for your runtime and resolve any failures or warnings
- Define rollback criteria and rehearse the rollback procedure
- Upgrade Terraform Enterprise
Select a target version
Complete the following tasks before you begin your upgrade:
Confirm compatibility: Identify the Terraform Enterprise version you plan to install. Check the release notes for the target version to confirm that it supports a blue-green upgrade from your current version. Compare the supported platforms and dependency versions in the release tables with your current infrastructure. Review the release notes for the target version and every intermediate version for breaking changes, deprecations, and new requirements.
Stage release artifacts: Download the Terraform Enterprise container images for every release in your upgrade path. Import the images into internal registries or hosts that cannot reach the public registry so you can update configuration files without extending downtime. Prune unused local images or volumes to free capacity where needed.
The requirements for deploying Terraform Enterprise depend on the version you want to deploy, as well as the environment you plan to operate the software in, such as operating system, runtime, and storage systems. Check the software product compatibility report page for details.
Take a live backup of PostgreSQL and Redis
Blue-green compatible releases have been tested to not mutate the state of Redis or the database. In the event of unexpected issues, take a live backup of your PostgreSQL database and Redis cache. A live backup is a backup of a datastore taken while the application is still running. The blob store is skipped because data stored in it is immutable. Use native primitives, such as pg_dump or the AWS Backup tool, to conduct these backups.
Conduct the blue-green upgrade
After you capture backups, apply the target release. Use the artifacts you staged during planning so you can complete these steps within the maintenance window.
- Update your configuration to reference the staged Terraform Enterprise image for the target release:
images.releases.hashicorp.com/hashicorp/terraform-enterprise:x.y.z.
Podman Kube and Docker Compose do not explicitly support zero-downtime upgrades. However, if you used a HashiCorp Validated Design to deploy Terraform Enterprise on virtual machines in AWS, GCP, or Azure, you can perform a zero-downtime upgrade with the following steps.
First, update the deployment configuration to use the target image tag without starting an automatic rollout:
- AWS Auto Scaling group (ASG): Update the launch template version used by the group. Do not start an instance refresh before draining the instances.
- GCP Managed Instance Group (MIG): Update the instance template used by the group and select an opportunistic update policy. See Google's docs about opportunistic updates.
- Azure Virtual Machine Scale Set (VMSS): Set the upgrade policy to manual before updating the scale set model. See Azure's docs about manual upgrade policy mode.
Then follow these steps for one instance at a time. Keep the other instances running and confirm they have enough capacity to handle the workload:
Connect to the Terraform Enterprise container on the instance you plan to replace and drain that node:
$ tfectl node drainWait for the node to be ready to be terminated (status can be checked with the
tfectl node taskscommand on the instance). Do not use--all, which drains every node, which could cause downtime. Refer to Gracefully stop work on a node for more information.Remove the drained instance from the load balancer and wait for any in-flight requests to finish.
Replace or update only the drained instance using the target configuration:
- AWS: Terminate the instance without decreasing the group's desired capacity so the ASG creates a replacement.
- GCP: Use the MIG's recreate operation for the instance rather than deleting it from the group.
- Azure: Manually apply the updated scale set model to the selected instance. Reimage it if the configuration change requires it. Do not assume deleting an instance automatically creates a replacement.
Confirm that the updated or replacement instance runs the target Terraform Enterprise version and passes the startup and readiness checks in Validate the blue-green upgrade.
Return the healthy instance to the load balancer. If registration is automatic, ensure the load balancer sends traffic only after the instance passes its health checks.
Confirm that the instance serves traffic successfully before moving to the next one. If validation fails, stop the rollout and follow your rollback criteria.
If you are not running Terraform Enterprise behind an autoscaling group, follow the same drain, remove, replace, and validate sequence, but create the replacement instance manually. The specifics vary based on your deployment configuration.
Validate the blue-green upgrade
Perform these checks during the upgrade, after it, or both.
Verify startup checks passed. Use the Terraform Enterprise CLI to trigger and review startup checks. Refer to Trigger startup checks for instructions on running the checks and Startup checks reference for details about each check and common failure scenarios.
Check application health. Query the readiness endpoint to verify the application responds correctly. The endpoint returns
200 OKwhen the node is ready to accept requests. The readiness endpoint reports the status of a single node, so in multi-node deployments query each node directly rather than through a load balancer, which may route the request to any node. Refer to Perform diagnostics for additional information about checking node readiness.$ curl https://<TFE_HOSTNAME>/api/v1/health/readinessPerform a test run. Execute a test Terraform plan and apply in a non-production workspace to confirm core functionality.
Verify release-specific features. Confirm that features or fixes from the target release appear as expected. Consult the release notes for validation steps.
Document the upgrade. Record the upgrade path, any issues encountered, and resolution steps for future reference.
Troubleshoot blue-green upgrades
If a validation step fails or the application does not stabilize:
- Pause the rollout so no additional instances or pods are replaced.
- Review application logs for migration errors or dependency mismatches. Refer to Access the Terraform Enterprise CLI for instructions on accessing logs in your environment. You can also download logs through the Admin Console or the System API.
- Roll back with zero downtime by repeating the upgrade steps one instance or pod at a time with the previous image tag.
- Contact HashiCorp Support with collected logs, the upgrade path attempted, and error messages for assistance.