Nomad installation

You can deploy Terraform Enterprise to HashiCorp Nomad, which lets you control the deployment scaling of the application. Because Nomad and Terraform are HashiCorp products, you do not need not to depend on third-party tools and their support. Nomad has a lower learning curve compared to Kubernetes and is easier to manage.

Complete the following steps to deploy Terraform Enterprise to Nomad-orchestrated containers:

Parameterize the Terraform Enterprise license, host, and TLS encryption settings by adding Nomad variables to your job specifications. This enable you to use the same job specification with different configurations. Refer to Job Specification in the Nomad documentation for additional information.
Add Terraform Enterprise environment variables to your Nomad job specification to configure Terraform behavior. Refer to the Terraform Enterprise configuration reference for additional information.
Create a Nomad job specification for operating the Terraform Enterprise agent. Refer to Custom Worker Image for additional information about the Terraform Enterprise agent.
Run the Nomad command for pulling the Terraform Enterprise image and installing the binary.
Nomad in beta
Running Terraform Enterprise on Nomad is still in beta. Do not deploy beta features in production environments. Provision a dedicated test environment before installing Terraform Enterprise on Nomad. If you have questions or feedback about the Nomad beta, contact your HashiCorp account representative.

Requirements

Before you begin, ensure you meet the Nomad requirements for installing Terraform Enterprise on Nomad.

Parameterize Terraform Enterprise settings

Add the following variables to your Terraform Enterprise Nomad job:

tfe_license: Specifies the Terraform Enterprise license key.
tfe_hostname: Specifies the hostname of the Terraform Enterprise instance.
tfe_tls_cert_file: Specifies the base64 encoded TLS certificate file.
tfe_tls_key_file: Specifies the base64 encoded TLS key file.
tfe_tls_ca_bundle_file: Specifies the base64 encoded TLS CA bundle file.

Refer to Nomad Variables in the Nomad documentation for additional information.

Refer to the example Nomad job specification for additional guidance.

Configure Terraform Enterprise Nomad job specification

This job is responsible for running the Terraform Enterprise image on Nomad. Pass the variables that you defined in the Parameterize Terraform Enterprise settings section. Refer to Assigning Values to job Variables in the Nomad documentation for instructions. The following variables are required:

The following variables are optional:

Complete the following steps if you are deploying to Nomad v1.4.x and older:

Manually create an ACL token. Refer to Command: acl token create in the Nomad documentation for instructions.
Remove the identity stanza.
Pass the ACL token to the Terraform Enterprise job. Export the token to the NOMAD_TOKEN environment variable and add it to the env stanza.

Refer to the example Nomad job specification for TFE for a template that you can copy and modify. Run the nomad job run command and specify job configuration to submit the changes. Refer to Command: job run in the Nomad documentation for additional information about the command.

Configure a Nomad batch job to run the Terraform Enterprise agent

Create a Nomad job specification that defines the behavior of Terraform Enterprise agent running on Nomad. Configure the job as a Nomad batch job so that Nomad starts a new agent job each time you run the terraform plan or terraform apply command. You do not need to run batch jobs manually after creating them. Refer to Batch Job in the Nomad documentation for additional information. Refer to the example Nomad batch job specification for TFE agent for a template that you can copy and modify.

Run the Nomad jobs

Run the nomad run command to pull the Terraform Enterprise image and install the application. Pass the Terraform Enterprise job specification as the command argument. You must also provide the credentials for the registry to download the image:

$ nomad run -var="tfe_image_username=$TFE_REGISTRY_USERNAME" -var="tfe_image_password=$TFE_REGISTRY_PASSWORD" <path-to-tfe-job-spec>

Run the nomad run command and pass the Terraform agent job specification to register the batch job in Nomad :

$ nomad run <path-to-tfe-agent-job-spec>

Alternatively, you can pull and install the Terraform Enterprise image using the Terraform Enterprise On Nomad Pack tool. Refer to the terraform-enterprise-fdo-nomad-pack repository on GitHub for instructions.

Create initial admin user

Provision your first administrative user and start using Terraform Enterprise.

Examples

You can copy the following examples and modify the values to match your deployment.

Nomad job specification for TFE

The following example configuration defines a Terraform Enterprise job specification. You can copy the example and modify the values to match your deployment.

  variable "tfe_image" {
    description = "The TFE image to use"
    type        = string
    default     = "images.releases.hashicorp.com/hashicorp/terraform-enterprise:latest"
  }

  variable "tfe_image_username" {
    description = "Username for the registry to download TFE image"
    type        = string
  }

  variable "tfe_image_password" {
    description = "Password for the registry to download TFE image"
    type        = string
  }

  variable "namespace" {
    description = "The Nomad namespace to run the job"
    type        = string
    default     = ""
  }

  job "tfe-job" {
    datacenters = ["dc1"]
    namespace   = var.namespace
    type = "service"

    group "tfe-group" {
      count = 1
      network {
        port "tfe" {
          static = 443
        }
        port "vault" {
          static = 8201
        }
      }

      service {
        name     = "tfe-svc"
        port     = "tfe"
        provider = "nomad"
        check {
          name     = "tfe_probe"
          type     = "http"
          protocol = "https"
          port     = "tfe"
          path     = "/_health_check"
          interval = "5s"
          timeout  = "2s"
          method   = "GET"
        }
      }

      task "tfe-task" {
        driver = "docker"

        identity {
          # Expose Workload Identity in NOMAD_TOKEN env var
          env = true
        }

        template {
          data = <<EOF
          {{- with nomadVar "nomad/jobs/tfe-job/tfe-group/tfe-task" -}}
          TFE_LICENSE={{ .tfe_license }}
          TFE_HOSTNAME={{ .tfe_hostname }}
          {{- end -}}
          EOF
          destination = "secrets/env.env"
          env         = true
          change_mode = "restart"
        }

        template {
          data = <<EOF
          {{- with nomadVar "nomad/jobs/tfe-job/tfe-group/tfe-task" -}}
          {{ base64Decode .tfe_tls_cert_file.Value }}
          {{- end -}}
          EOF
          destination = "secrets/cert.pem"
          env         = false
          change_mode = "restart"
        }

        template {
          data = <<EOF
          {{- with nomadVar "nomad/jobs/tfe-job/tfe-group/tfe-task" -}}
          {{ base64Decode .tfe_tls_key_file.Value }}
          {{- end -}}
          EOF
          destination = "secrets/key.pem"
          env         = false
          change_mode = "restart"
        }

        template {
          data = <<EOF
          {{- with nomadVar "nomad/jobs/tfe-job/tfe-group/tfe-task" -}}
          {{ base64Decode .tfe_tls_ca_bundle_file.Value }}
          {{- end -}}
          EOF
          destination = "secrets/bundle.pem"
          env         = false
          change_mode = "restart"
        }

        config {
          image = var.tfe_image
          ports = ["tfe", "vault"]

          auth {
            username = var.tfe_image_username
            password = var.tfe_image_password
          }

          volumes = [
            "secrets:/etc/ssl/private/terraform-enterprise",
          ]
        }

        resources {
          cpu    = 2500
          memory = 2048
        }

        env {
          TFE_DATABASE_HOST = "<Database hostname and port e.g. postgres:5432>"
          TFE_DATABASE_USER = "<Database user e.g. postgres>"
          TFE_DATABASE_PASSWORD = "<Database password e.g. postgres>"
          TFE_DATABASE_NAME = "<Database name e.g. hashicorp>"
          TFE_DATABASE_PARAMETERS = "<Database parameters e.g. sslmode=disable>"

          TFE_OBJECT_STORAGE_S3_ENDPOINT = "<S3 hostname and port e.g. localhost:9000>"
          TFE_OBJECT_STORAGE_TYPE = "s3"
          TFE_OBJECT_STORAGE_S3_ACCESS_KEY_ID = "<AWS Access Key ID>"
          TFE_OBJECT_STORAGE_S3_SECRET_ACCESS_KEY = "<AWS Secret Access Key>"
          TFE_OBJECT_STORAGE_S3_REGION = "<AWS Region e.g.us-east-1>"
          TFE_OBJECT_STORAGE_S3_BUCKET = "<Bucket name>"

          TFE_REDIS_HOST = "<Redis hostname and port e.g. redis:6379>"
          TFE_REDIS_USER = "<Redis username>"
          TFE_REDIS_PASSWORD = "<Redis password>"
          TFE_REDIS_USE_TLS = "<To use tls? e.g. false>"
          TFE_REDIS_USE_AUTH = "<To use customized credential to authenticate? e.g. false>"

          TFE_RUN_PIPELINE_NOMAD_ADDRESS = "${NOMAD_ADDR}"
          TFE_RUN_PIPELINE_NOMAD_TLS_CONFIG_CA_CERT = "<path to CA certificate for mTLS/TLS communication with Nomad>"
          TFE_RUN_PIPELINE_NOMAD_TLS_CONFIG_CLIENT_CERT = "<path to Client certificate for mTLS/TLS communication with Nomad>"
          TFE_RUN_PIPELINE_NOMAD_TLS_CONFIG_CLIENT_KEY = "<path to Client certificate's key for mTLS/TLS communication with Nomad>"
          TFE_RUN_PIPELINE_NOMAD_TLS_CONFIG_INSECURE = "<if set SSL is disabled>"
          TFE_RUN_PIPELINE_DRIVER = "nomad"
          TFE_VAULT_DISABLE_MLOCK = "true"
          TFE_ENCRYPTION_PASSWORD = "<Encryption password>"
          TFE_OPERATIONAL_MODE = "active-active"

          # If you are using the default internal vault, this should be the private routable IP address of the node itself.
          TFE_VAULT_CLUSTER_ADDRESS = "http://${NOMAD_HOST_ADDR_vault}"

          TFE_TLS_CERT_FILE = "/etc/ssl/private/terraform-enterprise/cert.pem"
          TFE_TLS_KEY_FILE = "/etc/ssl/private/terraform-enterprise/key.pem"
          TFE_TLS_CA_BUNDLE_FILE = "/etc/ssl/private/terraform-enterprise/bundle.pem"
        }
      }
    }
  }

Nomad batch job specification for TFE agent

The following example configuration defines a Terraform Enterprise agent job. You can copy the example and modify the values to match your deployment. Do not modify the label for the main job field. The job must be named tfe-agent-job so that Nomad can properly process the configuration.

job "tfe-agent-job" {
  type = "batch"
  namespace = "tfe-agents"
  datacenters = ["dc1"]
  node_pool = "node_pool_tfe_agents"

  constraint {
    attribute = "${attr.kernel.name}"
    value     = "linux"
  }

  parameterized {
    payload       = "forbidden"

    meta_required = [
      "TFC_AGENT_TOKEN",
      "TFC_ADDRESS"
    ]

    meta_optional = [
      "TFE_RUN_PIPELINE_IMAGE",
        "TFC_AGENT_AUTO_UPDATE",
      "TFC_AGENT_CACHE_DIR",
      "TFC_AGENT_SINGLE",
      "HTTPS_PROXY",
      "HTTP_PROXY",
      "NO_PROXY"
    ]
  }

  group "tfe-agent-group" {
    task "tfc-agent-task" {

      driver = "docker"

      template {
              destination = "local/image.env"
              env         = true
              change_mode = "noop"
              data        = <<EOF
      {{ $image := env "NOMAD_META_TFE_RUN_PIPELINE_IMAGE" }}
      {{ if ne $image "" }}TFE_RUN_PIPELINE_IMAGE={{$image}} {{ else }}TFE_RUN_PIPELINE_IMAGE="hashicorp/tfc-agent:latest"  {{ end }}
      EOF
      }

      config {
        image = "${TFE_RUN_PIPELINE_IMAGE}"
      }

        env {
        TFC_ADDRESS           = "${NOMAD_META_TFC_ADDRESS}"
        TFC_AGENT_TOKEN       = "${NOMAD_META_TFC_AGENT_TOKEN}"
        TFC_AGENT_AUTO_UPDATE = "${NOMAD_META_TFC_AGENT_AUTO_UPDATE}"
        TFC_AGENT_CACHE_DIR   = "${NOMAD_META_TFC_AGENT_CACHE_DIR}"
        TFC_AGENT_SINGLE      = "${NOMAD_META_TFC_AGENT_SINGLE}"
        HTTPS_PROXY           = "${NOMAD_META_HTTPS_PROXY}"
        HTTP_PROXY            = "${NOMAD_META_HTTP_PROXY}"
        NO_PROXY              = "${NOMAD_META_NO_PROXY}"
      }

      resources {
        cpu = 500
        memory = 2048
      }
    }
  }
}