The Terraform Cloud Business Tier allows you to manage isolated, private, or on-premises infrastructure using self-hosted Terraform Cloud agents. The agent polls Terraform Cloud or Terraform Enterprise for any changes to your configuration and executes the changes locally, so you do not need to allow public ingress traffic to your resources. Agents allow you to control infrastructure in private environments without modifying your network perimeter.
In this tutorial, you will create a Terraform Cloud agent pool, launch local Dockerized agents on your machine, and configure a Terraform Cloud workspace to use the agents to manage an Nginx container on your machine. You will also manage tokens for multiple Terraform Cloud agents. By the end of the tutorial, you will understand how to set up a similar configuration in your own environment.
Note: Agents are only available for the Business tier of Terraform Cloud. If you are interested in upgrading, contact a sales representative.
For this tutorial, you will need:
Agent pools are groups of agents that can share tokens. When you associate a workspace with an agent pool, any agent in the pool can execute a run in that workspace.
To create an agent pool, navigate to the "Agents" panel within your Terraform Cloud's "Settings" page and click "Create agent pool."
education as the name, then click "Continue."
Next, Terraform Cloud will prompt you to generate a token for the agent pool.
agent1 as the description and click "Create token."
The next page displays your agent token and example commands for getting your agent running. This page only displays the token on creation — leave it open for now.
The page suggests a command to run the Docker agent that passes in environment variables
-e flags. In this tutorial, you will use the
parameter instead, which makes it easier to manage multiple variables within a
single file, so you will not have to re-enter them each time you need to relaunch
the container. Using an environment variable file also prevents credentials
from appearing in your process tables, granting an extra layer of security.
You can configure multiple tokens per agent pool, or have one shared token across all agents in the pool. By using unique tokens, you can revoke the token of one agent and stop its work without disrupting other agents in the pool, which you will do later in this tutorial.
Now configure a version-control-driven Terraform Cloud workspace to use the agent pool.
In a new browser window, create a fork of the demo repository in your own GitHub account. This repository contains sample configuration to spin up an Nginx Docker container.
In that same browser window, create a new
learn-terraform-cloud-agents workspace associated with your forked
repository. For a detailed example of how to configure a VCS integration in Terraform Cloud, revisit the
Use VCS-Driven Workflow
Next, navigate to your workspace's "General" settings.
Under "Execution Mode," select "Agent" and select "education" in the drop down under "Agent Pool."
Make sure to click "Save Settings" at the bottom of the page.
Create a file called
agent1.list file and paste in the contents below. Update
the value of
TFC_AGENT_TOKEN with the token you created in the previous step.
Name the agent
agent1 for this
exercise. This name will show up in the agent management UI and on runs,
allowing you to identify specific agents in the future. If you plan on using
unique tokens for each agent, use the same name for the agent and token for
TFC_AGENT_TOKEN=<YOUR TOKEN> TFC_AGENT_NAME=agent1
The only required environment variable is
TFC_AGENT_TOKEN, but the agent
loads all variables in your Docker environment. You can use this as an
alternative to storing credentials and environment variables in your Terraform
Cloud Workspace's configuration.
Start your first containerized agent on your local machine. For this exercise,
you are mounting the Docker socket using
-v /var/run/docker.sock:/var/run/docker.sock. Mounting the socket allows the
containerized agent to use the Docker provider to manage other containers on
$ docker run --name tfc_agent --env-file agent1.list -v /var/run/docker.sock:/var/run/docker.sock hashicorp/tfc-agent:latest [INFO] agent: Starting: name=agent1 version=0.4.1 [INFO] core: Starting: version=0.4.1 [INFO] core: Agent registered successfully with Terraform Cloud: id=agent-9F5TXbYJoA7s7c18 pool-id=apool-wqx8cnrSrFn2AUfi [INFO] agent: Core version is up to date: version=0.4.1 [INFO] core: Waiting for next job
Keep this terminal open.
Because this scenario uses the containerized Terraform to manage other Docker
containers, the agent container needs to access the Docker socket. The
Terraform Cloud Agent runs as the non-root
tfc-agent user within the
container, so you need to explicitly modify the permissions for the Docker socket.
Open a new terminal window.
First, verify the group ID of the
docker group on your system. It will be the third field in the output returned, in this case
$ grep docker /etc/group docker:x:281:root
Exec into the agent container as the root user.
$ docker exec -it -u 0 tfc_agent /bin/bash
docker group within the container, replacing
<GROUP_ID> with the
docker group ID of your host.
$ groupadd -g <GROUP_ID> docker
tfc-agent user, which owns the agent process, to the
$ usermod -aG docker tfc-agent
Exit out of the container.
Note: Mounting the Docker socket and manipulating its permissions is only necessary if you are running the Dockerized agent and need it to manage other Docker containers on the host machine.
Once the agent container launches, verify that it has registered with the pool in the Terraform Cloud interface.
Notice "1 out of 5 purchased agents" next to "Agent Pools" — the number of agents available to you is determined by your Terraform Cloud for Business plan.
Agents are available as Docker containers and as standalone x86 binaries. For guidance on using the binaries, refer to the agent documentation.
In your browser, navigate back to your Terraform Cloud
learn-terraform-cloud-agents workspace and
trigger a run by clicking "Queue plan."
The run "Execution Mode" shows that it is running in your local Docker agent. The plan details show the names of the agent pool and agent responsible for the run.
In your terminal, the Docker agent logs display the agent's Terraform actions.
[INFO] core: Job received: type=plan id=run-NpfMDfGTsnY9ai6A [INFO] terraform: Handling run: id=run-NpfMDfGTsnY9ai6A type=plan org=hashicorp-training workspace=learn-terraform-cloud-agents [INFO] terraform: Extracting Terraform from release archive [INFO] terraform: Terraform CLI details: version=0.14.5 [INFO] terraform: Downloading Terraform configuration [INFO] terraform: Running terraform init [INFO] terraform: Running terraform plan [INFO] terraform: Generating and uploading plan JSON [INFO] terraform: Generating and uploading provider schemas JSON [INFO] terraform: Persisting filesystem to remote storage [INFO] terraform: Finished handling run [INFO] core: Waiting for next job
Approve the proposed changes in Terraform Cloud by clicking on "Confirm & Apply," and then confirming the plan.
The agent will now execute the work. The agent is self-contained and will not spin up additional containers for the Terraform execution itself — it downloads the Terraform version your configuration specifies and any providers necessary to complete the work task. In this tutorial, the agent launches an additional Docker container because that is the resource declared in the sample Terraform configuration.
[INFO] terraform: Handling run: id=run-NpfMDfGTsnY9ai6A type=apply org=hashicorp-training workspace=learn-terraform-cloud-agents [INFO] core: Job received: type=apply id=run-NpfMDfGTsnY9ai6A [INFO] terraform: Extracting Terraform from release archive [INFO] terraform: Terraform CLI details: version=0.14.5 [INFO] terraform: Recovering filesystem from remote storage [INFO] terraform: Running terraform init [INFO] terraform: Running terraform apply [INFO] terraform: Finished handling run [INFO] core: Waiting for next job
Once the apply is complete, open a new terminal window and confirm that your Nginx container is running by
docker ps or visiting
$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c9469d90663e f6d0b4767a6c "/docker-entrypoint.…" About a minute ago Up About a minute 0.0.0.0:8000->80/tcp nginx 515ed219aa66 hashicorp/tfc-agent:latest "/bin/tfc-agent" 28 minutes ago Up 28 minutes admiring_swartz
Though this tutorial targets your local machine, you can use the same configuration on any instance of your infrastructure. While running, the agent polls Terraform Cloud for any new workloads it needs to complete. Since this is a pull-based pattern, you only need to allow TCP/443 egress traffic and DNS resolution. This is a lightweight way to use Terraform Cloud within your isolated network segments without needing to configure your own installation of Terraform Enterprise.
Each agent is single-threaded and can only execute the work of one run at a time. For concurrent workloads, you must provision multiple agents. Agents within a pool can share tokens, or use unique ones so you can easily revoke the token of one agent without disrupting others.
Launch a second agent with a unique token.
- Navigate back to your Terraform Cloud organization settings.
- Select the "Agents" panel.
- Click on the "education" agent pool.
- Click "+ New token."
- Name the new agent
This will take you through the same steps as your first token creation. You will use this second token to launch another agent.
In a new terminal, create a file named
agent2.list and open it. Copy and
paste the contents below, and set the value of
TFC_AGENT_TOKEN to be your new
TFC_AGENT_TOKEN=<YOUR TOKEN> TFC_AGENT_NAME=agent2
Now launch your second agent.
$ docker run --env-file agent2.list -v /var/run/docker.sock:/var/run/docker.sock hashicorp/tfc-agent:latest
The "Agents" page in Terraform Cloud will now display 2 agents ready to retrieve and process workloads.
Any other containers that have exited will appear here as well, but do not count toward your allotted number of agents. Refer to the Terraform Cloud Agent documentation for a detailed list of possible agent statuses and how they count toward your agent totals.
You can configure additional agent pools as well, allowing you to maintain agents in each of your data centers and network segments.
Next, revoke the token of one of your agents. On the
pool page, click "Revoke Token" for the
Once you revoke the token, the
agent2 container will log its graceful
shutdown and exit.
[ERROR] agent: Unrecoverable error, shutting down: error="GET https://app.terraform.io/api/agent/jobs: unexpected status code (401 Unauthorized): Agent token invalid" [INFO] agent: Shutting down [INFO] agent: Core plugin is shutting down [ERROR] core: Failed updating status: error="PUT https://app.terraform.io/api/agent/status: unexpected status code (401 Unauthorized): Agent token invalid" [INFO] core: Shutdown complete Graceful shutdown complete
agent1 container will continue running. This demonstrates the
advantage of using unique tokens for your agents: you can revoke a token
without disrupting any other agents. The state of
agent2 will change in the "Agents"
dashboard in Terraform Cloud.
The exited agent is in an "Unknown" state but will expire out of the list and your agent allocation count within 2 hours. If this agent comes back online before that timeout, it will return to an "Idle" state. Only agents in the "Idle," "Busy," or "Unknown" states count against your purchased agent limit.
Now that you have created and used Terraform Cloud agents to create resources in an isolated environment, clean up all resources created in this tutorial.
Prior to removing the agent pool, remove the resources created by queueing a destroy plan in your workspace. Follow the Destroy Infrastructure step of the clean up tutorial for step-by-step instructions if needed.
Your agent container will log the destruction plan as well.
[INFO] core: Waiting for next job [INFO] core: Job received: type=apply id=run-dVh4azMHBwS49QR5 [INFO] terraform: Handling run: id=run-dVh4azMHBwS49QR5 type=apply org=hashicorp-training workspace=learn-terraform-cloud-agents [INFO] terraform: Extracting Terraform from release archive [INFO] terraform: Terraform CLI details: version=0.14.6 [INFO] terraform: Recovering filesystem from remote storage [INFO] terraform: Running terraform init [INFO] terraform: Running terraform apply [INFO] terraform: Finished handling run [INFO] core: Waiting for next job
Note: Destroy your infrastructure resources prior to destroying the agent pool. Terraform Cloud will not be able to destroy your local containers without the agent.
To delete an agent pool, you must first disassociate it from all workspaces. In your workspace settings, change the "Execution Mode" to "Remote." Be sure to click "Save Settings."
On the "Agent Pool" page in your Terraform Cloud organization settings, scroll to the bottom for the "Delete Agent Pool" section. Click "Delete agent pool" on the "Agents" page and confirm the prompt "Yes, delete agent pool."
Terraform Cloud's self-hosted agents allow you to manage more of your resources using Terraform Cloud or Terraform Enterprise without modifying your network perimeter. Since run execution modes are workspace-specific, you can continue using remote execution for any publicly accessible-resources and use the self-hosted agent for resources that require extra security.