Manage agent pools with the Terraform Cloud Kubernetes Operator v2

The Terraform Cloud Kubernetes Operator lets you create and manage Terraform Cloud agents, agent pools, and tokens through a single Kubernetes custom resource.

Terraform Cloud Agents let Terraform Cloud communicate with isolated, private, or on-premises infrastructure. By deploying agents within a specific network segment, you can connect your environment and Terraform Cloud, which lets Terraform provision and manage private infrastructure.

In this tutorial you will use the Terraform Cloud Kubernetes Operator to create and manage a Terraform Cloud Agent. The operator manages the lifecycle of agents, making it easier to scale them quickly and securely.

Prerequisites

This tutorial assumes that you are familiar with the standard Terraform workflow, Terraform Cloud, and basic Kubernetes usage.

For this tutorial, you will need:

• A Terraform Cloud organization owner account
• kubectl
• A running Kubernetes cluster v1.16+ with the Terraform Cloud Kubernetes Operator installed

If you do not have the Terraform Cloud Kubernetes Operator installed already, follow the Deploy Infrastructure with the Terraform Cloud Operator for Kubernetes tutorial through the Deploy the operator step and do not delete the Kind cluster or any of the resources created.

Clone repository

In your terminal, clone the Learn Terraform Kubernetes Operator repository.

$ git clone https://github.com/hashicorp/learn-terraform-kubernetes-operator

Review configuration

Navigate into the repository directory.

$ cd learn-terraform-kubernetes-operator

Checkout the v2beta branch of the repository.

$ git checkout v2beta

Navigate into the operator directory, which contains configuration files for the operator.

$ cd operator

Open the agentpool.yml file. This file defines a Terraform Cloud agent pool with one agent. Replace ORGANIZATION-NAME with your own Terraform Cloud organization and save the file.

apiVersion: app.terraform.io/v1alpha2
kind: AgentPool
metadata:
  name: agent-pool-demo
spec:
  organization: ORGANIZATION_NAME
  token:
    secretKeyRef:
    name: terraformrc
    key: token
  name: agent-pool-demo
  agentTokens:
    - name: agent-pool-demo-token
  agentDeployment:
    replicas: 1
    spec:
    containers:
        - name: tfc-agent
        image: "hashicorp/tfc-agent:1.7.0"

Configure Terraform Cloud access

The operator requires owner privileges in your Terraform Cloud organization to manage your agents. To do so, create an API token for your owners team, then add it to your cluster as a Kubernetes secret.

First, sign into your Terraform Cloud account, then select Settings -> Teams.

Click on the owners team then click Create a team token to generate a new team API token. If you already have an active token for your owners team, retrieve it or regenerate it if necessary.

Copy this token and store it somewhere safe.

Create the agent pool

Create an environment variable to configure the Kubernetes namespace to deploy to named NAMESPACE and set it to edu.

$ export NAMESPACE=edu

Create a secret containing your Terraform Cloud Team API token. Replace APITOKEN with the token that you created earlier.

$ kubectl create secret generic -n $NAMESPACE tfc-owner --from-literal=token=APITOKEN

Apply the AgentPool specification to the namespace.

$ kubectl apply -n $NAMESPACE -f agentpool.yml

Review the operator logs to confirm you launched the agent and agent pool.

$ kubectl logs deployment/terraform-operator-terraform-cloud-operator -n tfc-operator-system -f
2023-03-28T12:58:05Z    INFO    Reconcile Agent Pool    {"agentpool": "edu/agent-pool-demo", "msg": "status.agentPoolID is empty, creating a new agent pool"}
2023-03-28T12:58:05Z    INFO    Reconcile Agent Pool    {"agentpool": "edu/agent-pool-demo", "msg": "successfully created a new agent pool with ID apool-REDACTED"}

Verify the agent pool

Navigate to your Terraform Cloud organization settings. Go to the Agents page and verify that you created a new agent pool named agent-pool-demo with one agent.

Enable autoscaling

Provide the .spec.autoscaling configuration in your AgentPool specification so that the Terraform Cloud Kubernetes Operator autoscales the number of agents based on the number of pending Terraform workloads.

Open the agentpool.yml file and add the following configuration.

apiVersion: app.terraform.io/v1alpha2
kind: AgentPool
spec:
  ##...
  autoscaling:
    minReplicas: 0
    maxReplicas: 1
    cooldownPeriodSeconds: 300
    targetWorkspaces:
    - name: greetings

The minReplicas and maxReplicas fields define the number of agents that the operator will deploy, and targetWorkspaces configuration enables autoscaling on the defined workspaces. By default, the operator will wait 300 seconds between scaling events, but you can configure the wait time with the cooldownPeriodSeconds field.

Apply the updated AgentPool spec.

$ kubectl apply -n $NAMESPACE -f agentpool.yml

Open the workspace.yml file and configure the workspace to use the agent pool by adding the .spec.agentPool configuration with the agent pool name.

apiVersion: app.terraform.io/v1alpha2
kind: Workspace
spec:
  ##...
  agentPool:
    name: agent-pool-demo

Next, update the workspace executionMode and set the vale to agent.

apiVersion: app.terraform.io/v1alpha2
kind: Workspace
spec:
  ##...
  executionMode: agent

Finally, apply the updated Workspace spec.

$ kubectl apply -n $NAMESPACE -f workspace.yml

With no pending Terraform workloads, the operator automatically scales the number of agents to match the minReplicas defined in your configuration. In this case, the operator stops all running agents. Watch the pods in the operator namespace and verify that the agent pods are stopped.

$ kubectl -n $NAMESPACE get pods -w
##...
edu                   agents-of-agent-pool-demo-78477fbb6f-2bsch                     0/1     Terminating         0          9m25s

Clean up resources

First, destroy the agent pool and agent by removing the resource definition from your cluster.

$ kubectl delete -n $NAMESPACE agentpool agent-pool-demo

Next, delete the Kubernetes resources. Navigate to the root directory.

$ cd ..

Destroy the namespace, secrets, and the operator. Confirm the deletion by typing yes when prompted.

$ terraform destroy
##...
Plan: 0 to add, 0 to change, 5 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes
##...
Destroy complete! Resources: 5 destroyed.

Finally, if you are running on Kind, delete the cluster.

$ kind delete cluster --name terraform-learn
Deleting cluster "terraform-learn" …

Next steps

In this tutorial, you used the Terraform Cloud Kubernetes operator to create a Terraform Cloud agent pool. You can modify and extend the example configuration to deploy multiple agent pools, and scale the number of agents in each agent pool.

Visit the following resources to learn more about the Terraform Cloud Kubernetes Operator.

• Read the Deploy infrastructure with the Terraform Cloud Kubernetes Operator v2 tutorial.
• To learn more about the Operator and its design, check out the hashicorp/terraform-cloud-operator repository.
• To discover more about managing Kubernetes with Terraform, review the HashiCorp Kubernetes tutorials.