Build a GitOps pipeline to deploy a three-tier application

The GitOps operational strategy uses automated deployments to implement changes to a designated Git repository. This approach lets you manage infrastructure and applications using the same predictable workflow, where your declarative configuration is the single source of truth for your infrastructure.

Standardized workflows help platform operators enforce best practices across their organizations. By codifying the deployment of the infrastructure and applications into GitOps pipelines, operators can enforce policy, provide golden base images, and more.

GitOps also helps security teams implement code scanning, container scanning, or policy enforcement. It also makes it easier for the security team to audit deployments and production environments. The GitOps framework enables either scanning the code in the repository directly, or adding steps to GitOps pipelines to gather metadata about each build. Using GitOps also helps enforce the principle of least privilege by ensuring that only the pipeline has access to make changes to infrastructure.

For application developers, GitOps pipelines can provide better, faster feedback to code changes. As operators define policy changes or other organization standards, application developers can implement the required changes and validate them with each commit.

In this tutorial, you will build and run a GitOps pipeline to deploy a three-tier web application and the Nomad cluster it runs on.

Workflow overview

In this tutorial you will deploy Terramino, a brick-stacking browser game. Terramino is a three-tier application made up of a web frontend, a REST API backend, and a Redis server to persist data. You will create and use a GitOps pipeline to publish changes to both the application and the underlying infrastructure.

In the example configuration, Terramino runs as a containerized service in a Nomad cluster. Your GitOps pipeline uses Terraform to build a Nomad cluster configured with Consul for service discovery and internal DNS. In a mature organization, these infrastructure resources are likely managed by a team of platform operators.

The GitOps pipeline also builds your containerized three-tier web application and deploys it to Nomad. Application-level changes are typically managed by downstream development teams.

In this tutorial, you will perform operations for both the platform operators and application developers, observing how GitOps enables using a consistent workflow for changes across your stack.

Prerequisites

This tutorial assumes that you are familiar with the Terraform and HCP Terraform workflows. If you are new to Terraform, complete the Get Started collection first. If you are new to HCP Terraform, complete the HCP Terraform Get Started tutorials first.

To complete this tutorial, you need the following:

• An AWS account and AWS access credentials
• A GitHub account
• An HCP Terraform account and organization with your GitHub account added as a VCS provider.
• An HCP account with an HCP Packer Registry enabled

Create an HCP service principal

This tutorial uses HCP Packer to store metadata about the machine images you build. To upload artifact metadata to HCP Packer using the HCP API, you need an HCP service principal key pair.

Log in to your HCP portal, choose your organization and project, then complete the following steps to create a new service principal:

1. Click Access control (IAM).
2. Click Service principals.
3. Click Create service principal.
4. Name the service principal gitops-lab.
5. Under Select service, choose Project.
6. Under Select role(s), choose Contributor.
7. Click Create service principal.

Next, click Keys in the sidebar, then click Generate key to create a new keypair for the service principal. Copy this information somewhere secure. You will use these values in a later step and HCP does not show the client secret again.

After recording and exporting your HCP client ID and client secret, click the Close button.

You will also need your HCP project and organization ID. To retrieve your project ID, navigate to your HCP Project settings page. Record this value for later in the tutorial.

Next, click the Go to organization settings link and record your Organization ID.

Configure HCP Terraform

Create the HCP Terraform project that you will use to deploy your application. HCP Terraform projects let you organize groups of workspaces and manage their sensitive values and permissions together. Projects let platform operators grant teams permissions to manage the workspaces they need while keeping control over the management of sensitive values such as cloud credentials.

In HCP Terraform, click Projects in the left navigation bar, then click New project.

Name the project gitops, then click Create.

Next, create a variable set that HCP Terraform will use to authenticate with AWS and HCP for each Terraform run in each workspace in the project. From your project overview page, click on Settings, then Variable sets, then Create project variable set.

Create a variable set named aws-gitops and in the Variable set scope choose Apply to the entire project. Add the following values as environment variables to your variable set.

Click Create variable set to save the variable set to the project.

Create example repository

Navigate to the template repository for this tutorial. Click the Use this template button and select Create a new repository. Select the GitHub account configured as the VCS provider for your organization and name the new repository learn-lab-gitops. Make the repository Public. Leave the rest of the settings at their default values and click Create repository.

Your repository contains the Packer template, Terraform configuration, and application code to deploy Nomad and Terramino. The repository contains the following files and directories:

• The Packer template, image.pkr.hcl, is in the root directory of the repository.
• The Terraform configuration is split between multiple files, all ending with the .tf suffix.
• The /app directory contains the application code for Terramino, as well as the Dockerfiles to build the frontend and backend application containers.
• The /nomad directory contains two Nomad job specifications that deploy Terramino, as well as an Nginx reverse proxy. The reverse proxy uses Consul's internal DNS to find the Nomad containers running the Terramino frontend.
• The /shared directory contains scripts that Packer and Terraform use to build and deploy Nomad and Consul.

Open the .github/workflows/build_and_tag.yml file and review the GitHub Action. First, this action parses the changed files and determines if it needs to build the AMI, the application, or both. This action builds the Nomad AMI when you modify the Packer template or any of the files in the shared/ directory, and builds the application container when you modify any of the files in the app/ directory.

If the action runs either build step, it creates a new tag in the repository to trigger a new HCP Terraform run.

name: Build and Tag

on:
 push:
   branches: [main]
##...
jobs:
   check-files-changed:
      name : Check changed files
      runs-on: ubuntu-latest
      if: github.event.commits[0].message != 'Initial commit'
      
      outputs:
          build-packer:  ${{ steps.check-packer.outputs.changed }}
          build-container: ${{ steps.check-app.outputs.changed }}
          update-terraform: ${{ steps.check-terraform.outputs.changed }}
          manual-run: ${{ steps.manual-run.outputs.manual }}

      steps:
          - name: Checkout Repository
            uses: actions/checkout@3df4ab11eba7bda6032a0b82a6bb43b11571feac # v4.0.0
            with:
              fetch-depth: 2

          - name: Check for Packer changes
            id: check-packer
            working-directory: .github/scripts
            run: ./check_files_changed.sh "shared/** image.pkr.hcl" "${{ github.event_name }}" "${{ github.event.pull_request.base.sha }}" "${{ github.sha }}"
            continue-on-error: true
      ##...
  
   build-packer:
       ##...

   build-application:
      ##...
  
   tag-release:
    ##...       

Configure your GitHub repository

To use this GitOps pipeline, you must enable GitHub Actions for your repository and grant the permissions needed to run the workflows.

1. In your new repository, navigate to the Settings page.
2. Click the Actions drop-down in the left navigation menu and choose General.
3. If GitHub prompts you to enable Actions, follow the presented instructions.
4. Under the Workflow permissions section, select Read and write permissions.
5. Click Save.

Next, give the action the credentials it needs to run Packer builds. In your repository Settings menu, open the Secrets and variables drop-down in the left navigation menu, then select Actions.

Add the following values as repository secrets. The GitHub action will use these values to build the Nomad image with Packer and push the artifact metadata to HCP Packer.

Create HCP Terraform workspace

Create an HCP Terraform workspace to deploy your infrastructure and application. Navigate to your gitops project overview page, click the New drop-down in the top-right corner, then choose Workspace.

On the Create a new workspace screen, choose the Version Control Workflow. Choose the GitHub account that you created the learn-lab-gitops repository in, then choose <USERNAME>/learn-lab-gitops from the list of repositories.

On the Configure Settings screen, leave the workspace name of learn-lab-gitops. Next, click the Advanced options drop-down to see additional configuration options. In this drop-down, enable the following options:

• Enable Auto-apply API, CLI, & VCS runs. If you do not enable this option, HCP Terraform will automatically run a plan, but waits for manual approval to apply it.
• Select the Tag-based VCS trigger type. The last step of the GitHub action creates a new tag on the repository whenever you commit changes to the main branch. This lets you run any additional checks, validations, or tests that you need before HCP Terraform starts the plan and apply run.

Click Create at the bottom of the page.

To prevent accidental Terraform runs before your repository is fully configured, HCP Terraform requires that you create at least one manual run before it will automatically create runs from VCS triggers. After you create the workspace, click Start new plan. Change the run type to Plan only, then click Start. This plan will fail since the action has not built the AMI or application yet, but this allows VCS triggers to create new Terraform runs.

Trigger a build

Since you configured the HCP Terraform workspace to use the VCS workflow and chose the tag-based trigger type, HCP Terraform will trigger a run for every new tag in the GitHub repository. Since you have not created any tags in your repository yet, you can manually run the GitHub Action to build, deploy, and tag everything to get your Nomad cluster and Terramino deployment running.

Navigate to your learn-lab-gitops repository in GitHub and click the Actions tab. On the left-side menu, click the Build and Tag workflow. Click the Run workflow drop-down, then click Run workflow. The GitHub action runs both the Packer and application builds when you manually trigger the workflow.

Refresh the page until you see a new workflow run titled Build and Tag with a yellow circle to the left of the name, then click this new run to watch the progress. This workflow takes roughly ten minutes to build the AMI and container images.

Once GitHub completes the build, the action creates a new tag on the repository to trigger a new run in HCP Terraform. Open your learn-lab-gitops workspace in HCP Terraform to see a new run under Latest Run named Triggered via Git tag "1.1.0".

Once HCP Terraform completes the Terraform apply, refresh the overview page, then click the Outputs tab. The first output named app_url is the URL to the load balancer for your Terramino deployment. Open this page in your web browser to see a running version of Terramino.

Implement application changes

As an application developer, you can use the GitOps pipeline to deploy application changes. Imagine your organization implements a new security requirement that all Go applications must use a minimum version of 1.24 of the Go runtime. In this case, either the application development team or the security team could open a pull request against this repository to bump the version of the Go runtime. Update your repository to meet this new security requirement.

Open the GitHub repository in your web browser and open the app/Dockerfile.backend file. Notice that the Dockerfile's FROM statement uses the 1.22 tag of the golang container. This does not meet this new security requirement.

Click the pencil icon in the top-right of your code to open the file editor and change the tag to 1.24.

- FROM golang:1.22
+ FROM golang:1.24

Click the Commit changes… button in the top-right corner, then click Commit changes.

Since this change only updates the application code, the GitOps pipeline skips the Packer build step and only builds the application. Once it finishes building the application, it creates a new tag on the repository, which triggers a new Terraform run in your HCP Terraform workspace.

Open the Actions tab in your repository to track the build progress. Once the "Create tag" step finishes, open your workspace in HCP Terraform and note that the new Terraform run only updates the Nomad job specification.

Once HCP Terraform completes the Terraform run, open the Terramino web application and click the Debug button. Note that the "Go Version" value now shows that the application is running on Go 1.24.

Update the Nomad cluster

As a platform operator, you can manage infrastructure using the same workflows that an application developer uses to manage their application. For example, as application receives traffic grows, you can use the GitOps workflow to scale your infrastructure and add another client node to your Nomad cluster.

In the GitHub web UI, navigate to your GitHub repository, then open the variables.tf file. Click the pencil icon in the top-right of your code to open the file editor. Change the default value for the public_client_count from two to three.

variable "public_client_count" {
 description = "The number of clients to provision."
- default     = "2"
+ default     = "3"
}

Click the Commit changes… button in the top-right corner, then click Commit changes.

Click the Actions tab to see the new action that you triggered by updating the file. Since you didn't change any files that require a new Packer or application build, your GitOps pipeline does not perform a Packer or application build. Instead, it only creates a new tag to start a Terraform run in HCP Terraform. Once the "Create tag" step finishes, open your workspace in HCP Terraform and note that the new Terraform run only updates the number of client EC2 instances.

Next, verify that Nomad added the new client node to the cluster. On your learn-lab-gitops workspace Overview page, click the Outputs tab. Click the clipboard icon next to the nomad_token output to copy your token, then open the link in the nomad_ui output.

Since you deployed Nomad using a self-signed certificate, your browser may prompt you to verify that you intended to access this website.

In your Nomad dashboard, click the user icon in the top-right corner, enter the value of the nomad_token output in the Secret ID text box, then click Sign in with secret.

Next, click Clients in the left navigation menu. Nomad shows the number of client nodes responsible for running Nomad jobs currently connected to the cluster. Notice that this list now has three client nodes.

Destroy infrastructure

Clean up your infrastructure to ensure that you do not incur additional cost.

Clean up HCP Terraform

In your learn-lab-gitops workspace, click on Settings, then click Destruction and Deletion.

Click Queue destroy plan and follow the instructions shown to start a Terraform destroy run.

Once HCP Terraform destroys your infrastructure, delete the workspace. Click on Settings, then click Destruction and Deletion.

Click Delete from HCP Terraform to delete the workspace.

Next, delete your "gitops" project from HCP Terraform. Click Projects, then choose your gitops project. Click Settings, then under Delete project, click Delete. Follow the instructions shown to delete your project.

Delete the GitHub repository

To prevent unintended GitHub action runs now that you've completed the tutorial, delete the repository from GitHub.

Open your learn-lab-gitops repository in GitHub and click Settings.

Scroll to the bottom of the page, and under Danger Zone, click Delete this repository. Follow the instructions shown to delete the repository from GitHub.

Clean up HCP service principal

You should always delete your HCP service principals if you no longer plan to use them. In your HCP project overview page, click Access control (IAM), click Service principals, then click your gitops-lab service principal.

On the gitops-lab service principal overview page, click the Manage drop-down in the top-right, then click Delete service principal. Follow the instructions shown to delete the service principal and access key from HCP.

Clean up HCP Packer

Navigate back to your HCP project overview page and click Packer in the left navigation menu, then click the nomad-consul bucket.

On your bucket overview page, click the Manage drop-down in the top-right corner, then click Delete bucket. Follow the instructions shown to delete the bucket from HCP Terraform.

Next, deactivate your HCP Packer registry. On the HCP Packer Buckets page, click the Manage drop-down and choose Deactivate registry. Type DEACTIVATE in the text box, then click Deactivate.

Next steps

In this tutorial, you learned how you can apply GitOps principles to manage and automate applications and infrastructure using the same workflows and concepts. You then built a GitOps pipeline to Nomad cluster with a three-tier application running on top of it. Using the same GitOps workflow, you introduced changes to both your application and your Nomad cluster. To learn more about using HashiCorp tools in automation pipelines, refer to the following: Review the following resources to learn more about how HashiCorp products can help you automate your workflows and implement cloud best practices.

• Learn how to use HCP Terraform to enforce policies and detect infrastructure drift.
• Learn how to build a golden image pipeline with HCP Packer.
• Learn how to use no-code modules to provision infrastructure.
• Review the Well-Architected Framework to learn more about best practices.