Migrate Alternative Worker Images to Agents: v202302-1
Terraform Enterprise v202302-1 introduced the
agent run pipeline mode which
changes the container image used to perform Terraform runs. The previous run
pipeline mode is now referred to as the
legacy run pipeline mode.
If you currently use the default image, you do not need to take action. Terraform Enterprise will automatically migrate you to the new default agent image when you adopt v202302-1 or later.
If you currently use a legacy custom
Enterprise will continue using your legacy custom image until you migrate to an
agent custom image and enable the
agent run pipeline mode. Before adopting
v202306-1, you must build an agent custom
image. If your
legacy custom image also executes custom
you must update migrate your customization agent
Terraform Enterprise will stop supporting legacy custom images in v202306-1.
Continue reading to determine if this change impacts you and what action to take.
An alternative worker image is the custom image Terraform Enterprise uses to perform Terraform runs.
We are replacing three components of the run pipeline,
tfe-rabbitmq with local Terraform Cloud/Enterprise
agents. If you use an alternative worker image to execute custom logic within
the run lifecycle, you will need to migrate that logic to an agent custom image.
Like Terraform runs that use alternative worker images, runs that use custom agents will still execute in isolated, short-lived Docker containers. This change does not affect the Terraform Runs API and UI.
This change is part of an ongoing effort to refresh the architecture of Terraform Enterprise, improve performance and reliability of runs, and support new application-level features. Some immediate benefits of this change include:
- The run pipeline contains fewer components, making it easier to understand and debug.
- Local runs and remote agent runs can now use the same base images, and the same pre- and post- run hooks.
- Runs are more evenly distributed across Active/Active nodes by eliminating a redundant queue.
- You can now access previously inaccessible run logs (Sentinel, cost
estimation, and plan-export-worker logs) via Docker logs and any tool that can
interact with it. This includes the
docker logscommand, the Docker API, and TFE log forwarding.
In v202302-1, Terraform Enterprise installations using the default worker image (not an alternative worker image) will automatically use the new run pipeline.
Installations that use an alternative worker image will continue to use the legacy run pipeline when you upgrade to v202302-1. You must migrate your Terraform Enterprise installation to use an agent custom image before adopting v202306-1, which requires rebuilding your alternative worker image as an agent custom image.
Once you rebuild your image, you must manually switch to the new run pipeline by setting the
run_pipeline_mode = "agent". You can revert back to the legacy run pipeline at any time before you adopt v202306-1 by setting
run_pipeline_mode = "legacy". The
legacy option will become unavailable in v202306-1 and later.
By default, Terraform Enterprise uses the standard image included in a release. You can verify whether your pipeline uses an alternative worker image by reviewing your
settings.json file or your admin console.
In the admin console's Settings tab, check if Provide the location of a custom image is selected. If it is, your pipeline uses the alternative worker image specified in the Custom image tag field.
You can also review your
settings.json file to check if your pipeline uses alternative worker image. If your file includes the
custom_image_tag field, your pipeline uses the worker image specified.
To migrate to a custom agent image, first review the customizations in your worker image. Then build your new agent image, add your customizations, and test the new, customized agent image.
Before making any changes, you must evaluate your current image and familiarize yourself with all of the implemented customizations.
Consider the following:
- Which custom tools does your image include?
- Does the image use the initialize and/or finalize hook(s)?
- If yes, what do your initialize and/or finalize script(s) do?
Starting with v202302-1, the Docker container that Terraform Cloud and Enterprise uses for Terraform
operations is based on the
image. Like the legacy worker image, the agent image supports customizations to perform
custom logic, but there are several implementation differences between
worker images and agent images to consider.
The following tables describe the differences between the legacy alternative worker image and the new custom agent image.
Note: You must build custom images from
hashicorp/tfc-agent:1.6 or later.
|Alternative worker image||Custom agent image|
|Building a custom image||Documentation||Documentation|
|Executing custom scripts||Documentation||Documentation|
|Set the ||Set the |
|Image location||The image must be located on the host.||If Terraform Enterprise cannot find the image on the host, it will try to pull it from the specified location.|
|Alternative Worker Image||Custom Agent Image|
|Base image||The base image must be ||The base image must be the |
|Image location||The image must exist on the Terraform Enterprise host. You can add it by running ||We recommend that the image exist locally, but if it does not, the |
|Required software||The image must contain the software packages defined here.||N/A|
|Certificates||The ||The |
This will be similar to the certificate config in this Dockerfile example.
|Terraform binary||The image cannot include Terraform. Terraform Enterprise installs Terraform at runtime.||The image cannot include Terraform. Terraform Enterprise installs Terraform at runtime.|
|Rhel 7||In addition to the packages that ||N/A|
|Run before plans and applies||The same initialize script runs before a ||You can have separate hooks that run before and after plans and applies: |
|Run after plans and applies||The same finalize script runs after ||You can have separate hooks that run before and after plans and applies: |
|Configuration||Ensure your worker image contains an executable shell script at: ||Store hooks in a hooks directory, and name them according to their purpose. For example: |
|Non-zero exit codes||If the script exits with a non-zero exit code, the Terraform Enterprise run will immediately fail with an error.||If a |
|The build worker logs whether or not a custom script has executed but does not log the script's standard output and standard error. If the custom script exits non-zero, then its standard output and standard error print to the UI logs.||The standard output and standard error from the hook will print alongside the Terraform run output in the Terraform Cloud/Enterprise user interface, but not in the UI logs.|
|File locations||You cannot customize the name or location of the script.||The hook name must match one of the supported hooks. You cannot customize or change these names. Because of this, you can only configure one hook of each type for each agent. For example, you could create a pre-plan and pre-apply hook, but you cannot create two pre-plan hooks.|
|Script permissions||The script must have execute permissions.||Each hook must have the execute permission set.|
|Execution timeout||The execution of the script does not have a timeout. It is up to the Terraform Enterprise administrator to ensure scripts execute in a timely fashion.||Each hook has a 60 second timeout. If a hook times out the Terraform run will fail immediately.|
|Execution context||The execution of the script is not sandboxed. The script executes in the same container where ||Like with alternative worker images, the execution of the script is not sandboxed. The script executes in the same container where |
|Reading and writing variables||Users can "export" environment variables for subsequent Terraform commands (plan, apply, etc) to use, by writing their environment variable value to a file ||Writing to environment variables requires a minimum agent version of 1.9.0. Instead of directly calling “export”, write to environment variables using “echo”. Terraform parses and sets the environment variable later when other Terraform commands execute. For example, write |
|Running in Docker||Example Dockerfile snippet: ||When running |
|Execute/running as a binary||Scripts always run in the worker container. To execute, initialize and finalize scripts and all the commands they invoke, ensure your worker image contains an executable shell script at ||If you want to run agents outside of containers, you must create your own remote agent pool. |
To run hooks as a binary, you need to change the permissions to the hook scripts.
We strongly recommend that you test your custom agent image before applying it to your production installation of Terraform Enterprise.
Terraform Enterprise v202301 and earlier use worker images, but you can still test your agent image by configuring a test workspace to use a remote agent. You will need to install your agent on the host of your choice, and configure your workspace to use that agent. Follow the process described here for more details.
To test in v202302 or later, install a test installation of Terraform Enterprise, define a
custom_agent_image_tag, and set
run_pipeline_mode = "agent" in your test installation’s
After building and testing the new custom agent image, adopt the new image in your production Terraform Enterprise installation (v202302 or later) by defining a
custom_agent_image_tag, and setting
run_pipeline_mode = "agent" in your
Q. I currently use an alternative worker image, but I no longer need the customizations. What should I do?
A. If you want to use the default agent image going forward instead of a customized image, please remove your alternative worker image from use before you upgrade to v202302-1. If you want to remove the alternative worker image after upgrading to v202302-1 or later, set
run_pipeline_mode = "agent" in your
settings.json file without specifying a custom agent image.
Q. I already use a custom image for my remote agents. Can I re-use that image for my local agent image?
A. Yes. If your remote agent image accomplishes everything you want, you can specify that image as your custom agent image.
Q. I use an alternative worker image, when do I need to complete my migration to a custom agent image?
A. Terraform Enterprise will stop supporting workers in v202306-1. To maintain your customizations, you must complete your migration to a custom agent image before upgrading to v202306-1 or later.
Q. Can I switch back to the legacy run pipeline?
A. Yes, until v202306-1 you can switch between workers and agents. To switch back to the worker run pipeline, set
run_pipeline_mode = "legacy" in
settings.json. We will remove
legacy mode in v202306-1.
Q. I use an alternative worker image to dynamically manage run credentials. What is the best way to accomplish this with a custom agent image?
A. You can dynamically manage credentials using workspace variables and/or agent hooks. Upcoming enhancements to Terraform Enterprise will make this easier. Please reach out to your HashiCorp account representative for more information.
Q. I have a custom workflow on my alternative worker image that does not seem possible on a custom agent image. What should I do?
A. Please reach out to your HashiCorp account representative for further guidance.