Deploy Infrastructure with the Terraform Cloud Operator for Kubernetes

The Terraform Cloud Operator for Kubernetes (Operator) allows you to manage the lifecycle of cloud and on-prem infrastructure through a single Kubernetes custom resource.

You can create application-related infrastructure from a Kubernetes cluster by adding the Operator to your Kubernetes namespace. The Operator uses a Kubernetes Custom Resource Definition (CRD) to manage Terraform Cloud workspaces. These workspaces execute a Terraform Cloud run to provision Terraform modules. By using Terraform Cloud, the Operator leverages its proper state handling and locking, sequential execution of runs, and established patterns for injecting secrets and provisioning resources.

In this tutorial, you will configure and deploy the Operator to a Kubernetes cluster and use it to create a Terraform Cloud workspace. You will also use the Operator to provision a message queue that the example application needs for deployment to Kubernetes.

Prerequisites

The tutorial assumes some basic familiarity with Kubernetes and kubectl.

You should also be familiar with:

• The Terraform workflow — All Get Started tutorials
• Terraform Cloud — All Get Started with Terraform Cloud tutorials

For this tutorial, you will need:

• A Terraform Cloud account

• kubectl

• An AWS account and AWS Access Credentials

  Note

  This tutorial will provision resources that qualify under the AWS free-tier. If your account doesn't qualify under the AWS free-tier, we're not responsible for any charges that you may incur.

Install and configure kubectl

To install the kubectl (Kubernetes CLI), follow these instructions or choose a package manager based on your operating system.

$ brew install kubernetes-cli

$ choco install kubernetes-cli

You will also need a sample kubectl config. We recommend using kind to provision a local Kubernetes cluster and using that config for this tutorial.

$ brew install kind

$ choco install kind

Then, create a kind Kubernetes cluster called terraform-learn.

$ kind create cluster --name terraform-learn
Creating cluster "terraform-learn" ...
 ✓ Ensuring node image (kindest/node:v1.20.2) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-terraform-learn"
You can now use your cluster with:

kubectl cluster-info --context kind-terraform-learn

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

Verify that your cluster exists by listing your kind clusters.

$ kind get clusters
terraform-learn

Then, point kubectl to interact with this cluster.

$ kubectl cluster-info --context kind-terraform-learn
Kubernetes master is running at https://127.0.0.1:32769
KubeDNS is running at https://127.0.0.1:32769/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Clone repository

In your terminal, clone the Learn Terraform Kubernetes Operator repository.

$ git clone https://github.com/hashicorp/learn-terraform-kubernetes-operator

Navigate into the repository.

$ cd learn-terraform-kubernetes-operator

This repository contains the following files.

.
├── README.md
├── aws-sqs-test
│   ├── Dockerfile
│   └── message.sh
├── operator
│   ├── application.yml
│   ├── configmap.yml
│   └── workspace.yml
├── main.tf
└── terraform.tfvars.example
├── credentials.example

• The root directory of this repository contains the Terraform configuration for a Kubernetes namespace and the Operator helm chart.
• The operator directory contains the Kubernetes .yml files that you will use to create a Terraform Cloud workspace using the Operator.
• The aws-sqs-test directory contains the files that build the Docker image that tests the message queue. This is provided as reference only. You will use an image from DockerHub to test the message queue.

Configure the Operator

The Operator must have access to Terraform Cloud and your AWS account. It also needs to run in its own Kubernetes namespace. Below you will configure the Operator and deploy it into your Kubernetes cluster using a Terraform configuration that we have provided for you.

Configure Terraform Cloud access

The Operator must authenticate to Terraform Cloud. To do this, you must create a Terraform Cloud Team API token, then add it as a secret for the Operator to access.

First, sign into your Terraform Cloud account, then select "Settings" -> "Teams".

Click on "Create a team token" to generate a new team token. Copy this token.

Copy the contents of credentials.example into a new file named credentials.

$ cp credentials.example credentials

Then replace TERRAFORM_CLOUD_API_TOKEN with the Terraform Cloud Teams token you previously created.

credentials app.terraform.io {
  token = "TERRAFORM_CLOUD_API_TOKEN"
}

Explore Terraform configuration

The main.tf file has Terraform configuration that will deploy the Operator into your Kubernetes cluster. It includes:

• A Kubernetes Namespace. This is where you will deploy the Operator, Secrets, and Workspace custom resource.

  resource "kubernetes_namespace" "edu" {
    metadata {
      name = "edu"
    }
  }
  

• A terraformrc generic secret for your TFC Teams token. This is the default secret name the Operator uses for your Terraform Cloud credentials. The secret will contain the contents of your credentials file.

  resource "kubernetes_secret" "terraformrc" {
    metadata {
      name = "terraformrc"
      namespace = kubernetes_namespace.edu.metadata[0].name
    }
  
    data = {
      "credentials" = file("${path.cwd}/credentials")
    }
  }
  

• A generic secret named workspacesecrets containing your AWS credentials. In addition to the Terraform Cloud Teams token, Terraform Cloud needs your cloud provider credentials to create infrastructure. This configuration adds your credentials to the namespace, which will pass them to Terraform Cloud. You will add the credential values as variables in a later step.

  resource "kubernetes_secret" "workspacesecrets" {
    metadata {
      name = "workspacesecrets"
      namespace = kubernetes_namespace.edu.metadata[0].name
    }
  
    data = {
      "AWS_ACCESS_KEY_ID"     = var.aws_access_key_id
      "AWS_SECRET_ACCESS_KEY" = var.aws_secret_access_key
    }
  }
  

• The Operator Helm Chart. This is the configuration for the Operator, which is dependent on the terraformrc and workspacesecrets secrets.

  resource "helm_release" "operator" {
    name       = "terraform-operator"
    repository = "https://helm.releases.hashicorp.com"
    chart      = "terraform"
  
    namespace = kubernetes_namespace.edu.metadata[0].name
  
    depends_on = [
      kubernetes_secret.terraformrc,
      kubernetes_secret.workspacesecrets
    ]
  }
  

In order to use this configuration, you need to define the variables that authenticate to the kind cluster and AWS.

Run the following command. It will generate a terraform.tfvars file with your kind cluster configuration.

$ kubectl config view --minify --flatten --context=kind-terraform-learn -o go-template-file=tfvars.gotemplate > terraform.tfvars

Open terraform.tfvars and add your AWS credentials in aws_access_key_id and aws_secret_access_key respectively.

You should end up with something similar to the following.

host                   = "https://127.0.0.1:32768"
client_certificate     = "LS0tLS1CRUdJTiB..."
client_key             = "LS0tLS1CRUdJTiB..."
cluster_ca_certificate = "LS0tLS1CRUdJTiB..."
aws_access_key_id      = "REDACTED"
aws_secret_access_key  = "REDACTED"

Deploy the Operator

Now that you have defined the variables, you are ready to create the Kubernetes resources.

Initialize your configuration.

$ terraform init

Apply your configuration. Remember to confirm your apply with a yes.

$ terraform apply
## ...
kubernetes_namespace.edu: Creating...
kubernetes_namespace.edu: Creation complete after 0s [id=edu]
kubernetes_secret.terraformrc: Creating...
kubernetes_secret.workspacesecrets: Creating...
kubernetes_secret.terraformrc: Creation complete after 0s [id=edu/terraformrc]
kubernetes_secret.workspacesecrets: Creation complete after 0s [id=edu/workspacesecrets]
helm_release.operator: Creating...
helm_release.operator: Still creating... [10s elapsed]
helm_release.operator: Creation complete after 14s [id=terraform-operator]

Apply complete! Resources: 4 added, 0 changed, 0 destroyed.

Create an environment variable named NAMESPACE and set it to edu.

$ export NAMESPACE=edu

The Operator runs as a pod in the namespace. Verify the pod is running.

$ kubectl get -n $NAMESPACE pod
NAME                                                             READY   STATUS     RESTARTS   AGE
terraform-1613122278-terraform-sync-workspace-5c8695bf59-pgbpm   1/1     Running    0          108s

In addition to deploying the Operator, the Helm chart adds a Workspace custom resource definition to the cluster.

$ kubectl get crds
NAME                          CREATED AT
workspaces.app.terraform.io   2021-02-12T09:31:19Z

Explore workspace specification

Now you are ready to create infrastructure using the Operator.

First, navigate to the operator directory.

$ cd operator

Open workspace.yml, the workspace specification, and customize it with your Terraform Cloud organization name. The workspace specification both creates a Terraform Cloud workspace, and uses it to deploy your application's required infrastructure.

This workspace specification is equivalent to the following Terraform configuration.

module "queue" {
  source = "terraform-aws-modules/sqs/aws"
  version = "2.0.0"
  name = var.name
  fifo_queue = var.fifo_queue
}

You can find the following items in workspace.yml, which you use to apply the Workspace custom resource to a Kubernetes cluster.

• The workspace name suffix. The workspace name is a combination of your namespace and metadata.name (in this case: edu-greetings)

  metadata:
    name: greetings
  

• The Terraform Cloud organization. This organization must match the one you generate the Teams token for. Replace ORGANIZATION_NAME with your Terraform Cloud organization name.

  spec:
    organization: ORGANIZATION_NAME
  

• The file path to secrets on the Operator. By default, the Operator helm chart mounts your workspacesecrets secrets to /tmp/secrets.

  spec:
    organization: ORGANIZATION_NAME
    secretsMountPath: '/tmp/secrets'
  

• A Terraform module. This Workspace specification uses the AWS SQS module.

  module:
    source: 'terraform-aws-modules/sqs/aws'
    version: '2.0.0'
  

• Input variables. For variables that must be passed to the module, the variable key in the specification must match the name of the module variable. You can set whether a variable is sensitive or an environment variable, and the variable's value using ConfigMaps. We have set reasonable defaults for these values which you will review in a later step.

  variables:
    - key: name
      value: greetings
      sensitive: false
      environmentVariable: false
    - key: AWS_DEFAULT_REGION
      valueFrom:
        configMapKeyRef:
          name: aws-configuration
          key: region
      sensitive: false
      environmentVariable: true
    ## ...
    - key: AWS_ACCESS_KEY_ID
      sensitive: true
      environmentVariable: true
  

• Outputs you would like to find in the Kubernetes status. The example maps this_sqs_queue_id to an output named url .

  outputs:
    - key: url
      moduleOutputName: this_sqs_queue_id
  

Explore configmap.yml

In workspace.yml, the AWS_DEFAULT_REGION variable is defined by a ConfigMap named aws-configuration.

Open configmap.yml. Here you will find the specifications for the aws-configuration ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-configuration
data:
  region: us-east-1

Create the message queue

Apply the ConfigMap specifications to the namespace.

$ kubectl apply -n $NAMESPACE -f configmap.yml
configmap/aws-configuration created

Then, apply the Workspace specifications to the namespace.

$ kubectl apply -n $NAMESPACE -f workspace.yml
workspace.app.terraform.io/greetings created

Debug the Operator by accessing its logs and checking if the workspace creation ran into any errors.

$ kubectl logs -n $NAMESPACE $(kubectl get pods -n $NAMESPACE --selector "component=sync-workspace" -o jsonpath="{.items[0].metadata.name}")
## ...
{"level":"info","ts":1613124305.9530287,"logger":"terraform-k8s","msg":"Run incomplete","Organization":"hashicorp-training","RunID":"run-xxxxxxxxxxxxxxxx","RunStatus":"applying"}
{"level":"info","ts":1613124306.7574627,"logger":"terraform-k8s","msg":"Checking outputs","Organization":"hashicorp-training","WorkspaceID":"ws-xxxxxxxxxxxxxxxx","RunID":"run-xxxxxxxxxxxxxxxx"}
{"level":"info","ts":1613124307.0337532,"logger":"terraform-k8s","msg":"Updated outputs","Organization":"hashicorp-training","WorkspaceID":"ws-xxxxxxxxxxxxxxxx"}
{"level":"info","ts":1613124307.0339234,"logger":"terraform-k8s","msg":"Updating secrets","name":"greetings-outputs"}

View the Terraform configuration uploaded to Terraform Cloud. The Terraform configuration includes the module's source, version, and inputs.

$ kubectl describe -n $NAMESPACE configmap greetings
Name:         greetings
Namespace:    edu
Labels:       <none>
Annotations:  <none>

Data
====
terraform:
----
terraform {
           backend "remote" {
             organization = "hashicorp-training"

             workspaces {
               name = "edu-greetings"
             }
           }
         }
         variable "name" {}
         variable "fifo_queue" {}
         output "url" {
           value = module.operator.this_sqs_queue_id
         }
         module "operator" {
           source = "terraform-aws-modules/sqs/aws"
           version = "2.0.0"
           name = var.name
           fifo_queue = var.fifo_queue
         }
Events:  <none>

Check the status of the workspace via kubectl or the Terraform Cloud web UI to determine the run status, outputs, and run identifiers.

$ kubectl describe -n $NAMESPACE workspace greetings
Name:         greetings
Namespace:    edu
## ...
Status:
  Config Version ID:
  Outputs:
    Key:         url
    Value:       "https://sqs.us-east-1.amazonaws.com/656261198433/greetings"
  Run ID:        run-xxxxxxxxxxxxxxxx
  Run Status:    applied
  Workspace ID:  ws-xxxxxxxxxxxxxxxx

In addition to the workspace status, the Operator creates a Kubernetes Secret containing the outputs of the Terraform Cloud workspace. The Secret is formatted <workspace_name>-outputs.

$ kubectl describe -n $NAMESPACE secrets greetings-outputs
Name:         greetings-outputs
Namespace:    edu
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
url:  60 bytes

Verify message queue

Now that you have deployed the queue, you will now send and receive messages on the queue.

The application.yml contains a spec that runs a containerized application in your kind cluster. That app calls a script called message.sh, which sends and receives messages from the queue, using the same AWS credentials that the Operator used.

To give the script access to the queue's location, the application.yml spec creates a new environment variable named QUEUE_URL, and sets it to the Kubernetes Secret containing the queue url from the Terraform Cloud workspace output.

- name: QUEUE_URL
  valueFrom:
    secretKeyRef:
      name: greetings-outputs
      key: url

Open aws-sqs-test/message.sh. This bash script tests the message queue. To access the queue, it creates environment variables with your AWS credentials and the queue URL. Since Terraform Cloud outputs from the Kubernetes Secret contain double quotes, the script strips the double quotes from the output (QUEUE_URL) to ensure the script works as expected.

## ...
export SQS_URL=$(eval echo $QUEUE_URL | sed 's/"//g')
## ...

Deploy the job and examine the logs from the pod associated with the job.

$ kubectl apply -n $NAMESPACE -f application.yml
job.batch/greetings created

View the job's logs.

$ kubectl logs -n $NAMESPACE $(kubectl get pods -n $NAMESPACE --selector "app=greetings" -o jsonpath="{.items[0].metadata.name}")
https://sqs.us-east-1.amazonaws.com/REDACTED/greetings.fifo
sending a sdfgsdf message to queue https://sqs.us-east-1.amazonaws.com/REDACTED/greetings.fifo
{
    "MD5OfMessageBody": "fc3ff98e8c6a0d3087d515c0473f8677",
    "SequenceNumber": "xxxxxxxxxxxxxxxx",
    "MessageId": "xxxxxxxxxxxxxxxx"
}
reading a message from queue https://sqs.us-east-1.amazonaws.com/656261198433/greetings.fifo
{
    "Messages": [
        {
            "Body": "hello world!",
            "ReceiptHandle": "xxxxxxxxxxxxxxxx",
            "MD5OfBody": "fc3ff98e8c6a0d3087d515c0473f8677",
            "MessageId": "xxxxxxxxxxxxxxxx"
        }
    ]
}

Change the queue name

Once your infrastructure is running, you can use the Operator to modify it. Update the workspace.yml file to change the queue's name, and the type of the queue from FIFO to standard.

# workspace.yml
 apiVersion: app.terraform.io/v1alpha1
 kind: Workspace
 metadata:
  name: greetings
 spec:
    ## ...
  variables:
    - key: name
-    value: greetings.fifo
+     value: greetings
    - key: fifo_queue
-     value: "true"
+     value: "false"
    ## ...

Changing inline, non-sensitive variables, module source, and module version in the Kubernetes Workspace custom resource will trigger a new run in the Terraform Cloud workspace. Changing sensitive variables or variables with ConfigMap references will not trigger updates or runs in Terraform Cloud.

Apply the updated workspace configuration. The Terraform Operator retrieves the configuration update, pushes it to Terraform Cloud, and executes a run.

$ kubectl apply -n $NAMESPACE -f workspace.yml
workspace.app.terraform.io/greetings configured

Examine the run for the workspace in the Terraform Cloud UI. The plan indicates that Terraform Cloud replaced the queue.

You can audit updates to the workspace from the Operator through Terraform Cloud, which maintains a history of runs and the current state.

Clean up resources

Now that you have created and modified a Terraform Cloud workspace using the Operator, delete the workspace.

Delete workspace

Delete the Workspace custom resource.

$ kubectl delete -n $NAMESPACE workspace greetings
workspace.app.terraform.io "greetings" deleted

You may notice that the command hangs for a few minutes. This is because the Operator executes a finalizer, a pre-delete hook. It executes a terraform destroy on workspace resources and deletes the workspace in Terraform Cloud.

Once the finalizer completes, Kubernetes will delete the Workspace custom resource.

Delete resources and kind cluster

Navigate to the root directory.

$ cd ..

Destroy the namespace, secrets and the Operator. Remember to confirm the destroy with a yes.

$ terraform destroy

Finally, delete the kind cluster.

$ kind delete cluster --name terraform-learn
Deleting cluster "terraform-learn" ...

Next steps

Congrats! You have configured and deployed the Operator to a Kubernetes namespace, explored the Workspace specification, and created a Terraform workspace using the Operator. In doing so, you deployed a message queue from kubectl. This pattern can extend to other application infrastructure, such as DNS servers, databases, and identity and access management rules.

Visit the following resources to learn more about the Terraform Cloud Operator for Kubernetes.

• To learn more about the Operator and its design, check out the hashicorp/terraform-k8s repository.
• To learn more about the Operator helm chart, check out the hashicorp/terraform-helm repository. You can also view example Workspace specifications here too.
• To watch a demo of the Operator, visit Demo: GitOps configuration using the Terraform Cloud Operator for Kubernetes or Demo: HashiCorp Terraform Operator for Kubernetes.
• To discover more about managing Kubernetes with Terraform, review the Hashicorp Kubernetes tutorials.