Build a golden image pipeline with HCP Packer
A golden image is an image on top of which developers can build applications, letting them focus on the application itself instead of system dependencies and patches. A typical golden image includes common system, logging, and monitoring tools, recent security patches, and application dependencies.
Traditionally, operations and security teams had to cross-reference spreadsheets, personally inform downstream developers, and manually update build files when they released new golden images. Sophisticated organizations automated this process by building and maintaining effort-intensive continuous integration (CI) pipelines. The HCP Packer registry improves this process by tracking images' metadata and storage location, and providing the correct image to developers automatically through Packer and Terraform integrations. It also allows you to revoke images to remove them from circulation if they become stale or have security vulnerabilities.
After you build your image with Packer and push its metadata to HCP Packer, you can reference the image in your Terraform configuration to deploy it. HCP Packer has a Terraform Cloud run task integration, which validates that the machine images in your Terraform configuration are not revoked.
Note
Terraform Cloud Free Edition includes one run task integration that you can apply to up to ten workspaces. Refer to Terraform Cloud pricing for details.
In this tutorial, you will use HCP Packer to define a golden image pipeline and build parent golden and child application images. You will then deploy the application image to AWS using Terraform.
To accomplish this, you will first deploy an EC2 instance running Loki for log aggregation and Grafana for visualization. Then, you will build a golden image with configuration that references the Loki and Grafana instance's details, and build an application image that uses the golden image as a base. Then, you will schedule a revocation and learn how image revocation prevents downstream image consumers from referencing outdated images. Finally, you will use Terraform to deploy an EC2 instance running the application image, and view the application logs in Grafana.
Prerequisites
This tutorial assumes that you are familiar with the standard Packer and HCP Packer workflows. If you are new to Packer, complete the Get Started tutorials first. If you are new to HCP Packer, complete the Get Started HCP Packer tutorials first.
This tutorial assumes that you are familiar with the Terraform and Terraform Cloud workflows. If you are new to Terraform, complete the Get Started tutorials first. If you are new to Terraform Cloud, complete the Terraform Cloud Get Started tutorials first.
To follow along with this tutorial, you will need:
- Packer 1.7.9 installed locally.
- Terraform 1.2 or later installed locally.
- A Terraform Cloud account with workspace admin permissions.
- An HCP account.
- An HCP Packer registry with Plus tier.
- Create a registry: click Packer > Create a free registry. You only need to do this once.
- Enable Plus tier: click Manage > Edit registry and select Plus. If you have free-trial credits HCP will apply them to enable the Plus tier.
- An AWS account with credentials set as local environment variables.
Clone the example repository
In your terminal, clone the tutorial repository. It contains configuration for building and publishing images with Packer and deploying them to AWS with Terraform.
Navigate to the cloned repository.
Architecture overview
The diagram below shows the infrastructure and services you will deploy in this tutorial. You will provision one instance that runs Loki and Grafana and two instances for HashiCups — an example application for ordering HashiCorp-branded coffee. You will deploy the HashiCups instances across two AWS regions, us-east-2
and us-west-2
. The HashiCups instances contain baseline tools, including Docker and promtail
, which they will inherit from the golden image that HashiCups is based on.
HashiCups is an application consisting of an API and a database. The components run as separate Docker containers and are provisioned with Docker Compose. Docker stores the logs generated by both the API and database.
Promtail is an agent that sends logs from a local log store to an instance of Loki. In this scenario, Promtail forwards the HashiCups Docker container logs to the Loki instance using a Loki Docker plugin.
Loki is a log aggregation tool that provides log data for querying and runs on port
3100
. Grafana visualizes the Loki logs and provides its own web user interface on port3000
.
Review configuration
The example repository contains several directories:
- The
loki
directory contains a Packer template file, a Loki configuration file, and scripts that configure and enable Loki and Grafana. - The
golden
directory contains a Packer template file, Docker and Promtail configuration files, and scripts that configure and enable Docker and Promtail. - The
hashicups
directory contains a Packer template file, a Docker Compose file, and the HashiCups start script. - The
terraform
directory contains Terraform configuration files to deploy AWS EC2 instances that run the images for this scenario, and a script to query the HashiCups API.
Warning
This configuration provisions a publicly accessible Loki and Grafana instance, which is not recommended for production services.
First, you will build the Loki image and deploy it to an EC2 instance. Then, you will build the golden image, which requires the public address of the Loki instance for its configuration. Finally, you will build and provision the HashiCups image, which uses the golden image as a parent image. You will configure and use a Terraform Cloud run task to verify that the images referenced in the Terraform configurations have not been revoked.
The Loki instance simulates an existing implementation of Loki running in your organization's network. In a production scenario, you would configure a DNS entry for your Loki instance(s) rather than the EC2 instance address.
Review Loki image configuration
Open loki/start-loki-grafana.sh
and note that both Loki and Grafana run on the same instance — Loki as a system process and Grafana as a Docker container.
Next, open loki/loki.pkr.hcl
. Packer uses this file to build an Amazon Machine Image (AMI) that runs Loki and Grafana. This tutorial will refer to this image as "Loki image" even though it contains both Loki and Grafana.
The amazon-ami.ubuntu-focal
data block retrieves an Ubuntu 20.04 image from the us-east-2
region to use as a base. The amazon-ebs.base
source block then references the ID of that image from the amazon-ami.ubuntu-focal
data block for the source_ami
.
The build
block uses the image retrieved by the amazon-ebs.base
source block, and adds an SSH public key, the Loki configuration file, and the startup script to the image.
Then, Packer executes the loki-setup.sh
script to set up sudo
, install dependencies, the SSH key, and Loki.
Finally, Packer sends the image metadata to the HCP Packer registry so downstream Terraform deployments can use it.
Review golden image configuration
A golden image typically includes baseline tools, services, and configurations. The golden image for this tutorial contains Docker and Docker Compose for running applications, promtail
for log export, grafana/loki-docker-driver:latest
for collecting Docker logs, and auditd
for securing Docker.
Open golden/golden.pkr.hcl
. This configuration defines two amazon-ebs
source blocks which each reference a corresponding amazon-ami
data block. There is one block for each AWS region where you will publish your AMI. AMIs are region specific, so you must build a separate AMI for each region.
The build
block uses the images retrieved by the amazon-ebs.base_east
and amazon-ebs.base_west
sources. It adds an SSH public key, runs the setup.sh
script to install dependencies, adds the audit rules for Docker, the Docker daemon config file, the promtail
config file, and the run-promtail.sh
script file.
The two values in the sources
attribute let Packer build these two images in parallel, reducing the build time. Refer to the AWS Get Started Tutorial for more details about parallel builds.
After it builds the images, Packer moves the configuration files to the correct directories and runs the setup-promtail.sh
script to configure promtail
and its Docker plugin.
Finally, Packer sends the image metadata to the HCP Packer registry so that downstream Packer builds and Terraform deployments can reference it.
Review HashiCups image configuration
Open hashicups/hashicups.pkr.hcl
.
The hcp-packer-iteration
data source retrieves information about the iteration from the HCP Packer bucket_name
and channel
. The value of bucket_name
matches the one defined in the hcp_packer_registry
block of the golden image Packer template (golden/golden.pkr.hcl
).
The hcp-packer-image
data source uses the iteration details to retrieve the image for the specified cloud_provider
and region
. This data source is necessary because an iteration can include images from different cloud providers and regions.
The two hcp-packer-image
data sources use the same iteration_id
but reference different images based on the region
value.
The source_ami
references the hcp-packer-image
data source, using the AMI ID stored in the HCP Packer registry.
The build
block uses the golden images defined in the amazon-ebs.hashicups_east
and amazon-ebs.hashicups_west
sources and adds an SSH public key, the conf.json
file for application configuration, the Docker Compose file to create the HashiCups containers, and the HashiCups start script. Packer then moves the start script to the correct directory.
Like the golden images, Packer builds these images in parallel.
Finally, Packer sends the image metadata to the HCP Packer registry so downstream Terraform deployments can reference it.
Review Terraform configuration
Open terraform/main.tf
. This Terraform configuration defines EC2 instances that run the Loki and HashiCups images.
Terraform retrieves image source information from the HCP Packer registry in a similar way as Packer. The hcp_packer_image_iteration
data source gets the latest iteration from the bucket and channel provided.
The hcp_packer_image
data sources then use the iteration ID to retrieve and store the AMI IDs of the images in the regions specified. Notice that the ami
value of the aws_instance
resource references the hcp_packer_image
data source's AMI ID.
The remaining Terraform configuration files define input variables, output values, and network infrastructure that the Loki and HashiCups instances depend on, including a VPC, internet gateway, subnet, route table, and security groups. This Terraform configuration deploys these resources to both the us-east-2
and us-west-2
regions.
Prepare your environment
The configuration scripts included in the AMIs rely on a user named terraform
. Create a local SSH key to pair with the user so that you can securely connect to your instances.
Generate a new SSH key named learn-packer
. The argument provided with the -f
flag creates the key in the current directory and creates two files called learn-packer
and learn-packer.pub
. Change the placeholder email address to your email address.
When prompted, press enter to leave the passphrase blank on this key.
Set your Terraform Cloud organization
Set the TF_CLOUD_ORGANIZATION
environment variable to your Terraform Cloud
organization name.
Log in to Terraform Cloud
In this tutorial, you will use the Terraform CLI to create the Terraform Cloud workspace and trigger remote plan and apply runs.
Log in to your Terraform Cloud account in your terminal.
Confirm with a yes
and follow the workflow in the browser window that automatically opens. Paste the generated API key into your Terminal when prompted. Review the Authenticate the CLI with Terraform Cloud tutorial for more details about logging in.
Create HCP service principal
A service principal allows Packer and Terraform to interact with HCP Packer to push and reference image metadata.
Log in to HashiCorp Cloud Platform, choose your organization, and click the Access control (IAM) link from the left navigation.
Click on the Service principals link from the left navigation, then Create service principal on the top right of the page.
Name the service principal learn-hcp-packer
, assign the "Contributor" role, then click Save.
Click Create service principal key at the bottom of the page.
Record the Client ID and Client secret — HCP only displays these values upon creation.
In your terminal, set an environment variable for your client ID.
Then, set an environment variable for your client secret.
Later in this tutorial, you will also create Terraform Cloud environment variables for these values.
Retrieve HCP Packer run task information
On the HCP Packer page, click Integrate with Terraform Cloud.
This displays information for you to use to configure your Terraform Cloud run task.
The Endpoint URL is a unique HCP Packer URL, specific to your HCP organization and HCP Packer registry. The Terraform Cloud run task will send a payload to this URL for image validation.
The HMAC Key is a secret key that lets HCP Packer verify the run task request.
Warning
Do not share these values. If your HMAC key is compromised, re-generate it and update your Terraform Cloud run task to use the new value.
Leave this tab open to reference the displayed values for the next step.
Set up run task in Terraform Cloud
In Terraform Cloud, go to Settings then click Run tasks on the left sidebar.
Click Create run task.
On the Create a Run Task page:
Verify Enabled is checked.
Set Name to
HCP-Packer
.Set Endpoint URL to the endpoint URL you retrieved in the previous step.
Set HMAC key to the HMAC key you retrieved in the previous step.
Note
Although labeled as optional in the UI, you must enter the HMAC key provided by HCP Packer. The HCP Packer integration requires an HMAC key to authenticate requests.
Click Create run task.
The Run Tasks page now shows the HCP-Packer
run task.
Build and deploy the Loki image
Use Packer to build the Loki image. Once you create the image and deploy it to an EC2 instance, you will add the instance IP address to the golden image Packer template.
Navigate to the loki
directory.
Initialize the template file for the Loki image.
Build the Loki image.
Notice that Packer assigned a randomly generated Unique Lexicographical Identifier (ULID) to this build, which lets HCP packer identify it. Packer then built the Loki image, stored it in AWS, and published the build's metadata to the HCP Packer registry in the final build step.
Inspect Packer Build on HCP
Visit HCP and click on Packer in the left navigation menu.
This page displays a list of buckets and their latest associated iterations. Click on the Loki bucket, which is named learn-packer-hcp-loki-image
.
Here, you can find information published to the registry from the Loki Packer build including the description and labels defined in the hcp_packer_registry
block of the loki/loki.pkr.hcl
template. The latest image iteration is on the right.
Click on Iterations in the left navigation.
This page displays each build iteration published to the bucket. Click on the iteration version at the top of the list.
The Builds section lists details about the images published in this iteration. The amazon-ebs.base
image matches the image defined in the source
block in the Loki Packer template. Click on the us-east-2 link to find information about the image published to the us-east-2
region, including the AMI ID.
Create channel for Loki image
HCP Packer registry channels let you reference a specific build iteration in Packer or Terraform. This reduces errors from hardcoding image IDs and allows both Packer and Terraform to automatically retrieve the most recent image.
Click Channels in the left navigation.
Create a new channel for the Loki bucket by clicking on New Channel.
Enter production
for the Channel name, select the v1
iteration from the Assign to an image iteration dropdown, and click the Create channel button.
Initialize the Loki instance with Terraform
Now use Terraform to deploy the Loki image to an AWS instance. First, change into the terraform
directory.
Initialize your Terraform configuration.
You have initialized your Terraform configuration and created your learn-hcp-packer-golden-image
workspace. You will now associate the run task you created earlier with this workspace to verify that images referenced in runs have not been revoked.
Add credentials to workspace
In Terraform Cloud, open the learn-hcp-packer-golden-image
workspace.
Go to the Variables page and create the following variables with your specific values.
Variable Name | Value | Category | Sensitive |
---|---|---|---|
AWS_ACCESS_KEY_ID | Your AWS access key ID | environment | |
AWS_SECRET_ACCESS_KEY | Your AWS secret access key | environment | yes |
HCP_CLIENT_ID | Your HCP client ID | environment | |
HCP_CLIENT_SECRET | Your HCP client secret | environment | yes |
Note
Set a variable for your AWS_SESSION_TOKEN
if your organization requires it.
Enable run tasks in workspace
Click on the workspace Settings, then Run Tasks.
Under Available Run Tasks, click on HCP-Packer.
Select the Mandatory enforcement level, then click Create.
The Run Task page now displays the run task for HCP Packer. This run task scans your Terraform configuration for resources that use hard-coded machine image IDs and checks if the image is tracked by HCP Packer. If the image is associated with an image iteration, the run task will warn users if it is a revoked iteration. It will also prompt users to use the HCP Packer data sources instead of hard-coded image IDs to better track and manage machine images.
Deploy the Loki instance with Terraform
Apply your configuration. Respond yes
to the prompt to confirm the operation.
Once Terraform builds the Loki instance, it prints the loki_ip
output value, the Loki instance's public IP address. You will reference this IP address in your parent image configuration to direct log forwarding to the Loki instance.
Verify image validation
In Terraform Cloud, open the latest run and expand the Tasks passed box.
The run task passed, which means that HCP Packer is tracking the Loki image you referenced in the Terraform Configuration, and that the image is not revoked.
Build golden image
To forward the Docker container logs to the Loki instance, you need to update two files with the Loki instance's IP address.
First, verify that you are in the terraform
directory.
In golden/docker-daemon.json
, replace LOKI_URL
with your Loki public IP address.
In golden/promtail.yaml
, replace LOKI_URL
with your Loki public IP address.
Change to the golden
directory.
Initialize the Packer build.
Build the golden image with Packer and the golden/golden.pkr.hcl
template.
Create channel for golden image
In HCP Packer, navigate to the learn-packer-hcp-golden-base-image
bucket page, create a new channel named production
, and select the latest iteration.
Build and deploy HashiCups image
Since the golden image is already configured to send container logs to Loki, and the HashiCups image is built on top of the golden one, you do not need to modify the HashiCups image configuration.
Use Packer to build the HashiCups image. Change to the hashicups
directory.
Initialize the Packer build.
Run the Packer build.
Create channel for HashiCups image and schedule revocation
In HCP Packer, navigate to the learn-packer-hcp-hashicups-image
bucket page.
The Ancestry table shows that this image is up to date with it's parent, the learn-packer-hcp-golden-base-image
image.
Now, create a new channel named production
, and select the latest iteration.
Test HCP image validation
If an image becomes outdated or a security risk, you can revoke it to prevent consumers from accessing its metadata and using it to build artifacts. Schedule a revocation for the current iteration.
- Go to the Iterations page
- Click ...
- Click Revoke iteration
- Select Revoke at a future date
- Enter the time for 1 minute from your current time. The time is in UTC (current time in UTC). For example, if it is currently
10:00
, enter10:01
- Enter
Assign image channel to revoked iteration
for the revocation reason - Click Revoke Iteration to revoke the iteration
You are setting a short revocation window so that your image channel uses a revoked image to test validation workflows. This is for the educational purposes of the tutorial.
Next, attempt to deploy the revoked HashiCups image with Terraform.
Change to the terraform
directory.
Add the following configuration to the end of terraform/main.tf
. This configuration defines EC2 instances in the us-east-2
and us-west-2
regions.
Save your changes.
The ami
values reference values from the HCP Packer data sources instead of hard-coded AMI IDs.
Add the following configuration to the bottom of terraform/outputs.tf
to display the IP addresses of the provisioned HashiCups instances.
Save your changes.
In your terminal, apply your configuration. After Terraform creates the plan, the run will return an error because the run task failed.
In Terraform Cloud, open the latest run to review the details. Click the Tasks failed box.
The run task detected that the aws_instance
resource references the hcp_packer_image
data source. Since the data source retrieved a revoked iteration, the run task failed.
If the run task had found a newer iteration version, it would have suggested that you use it. As an image maintainer always make sure to replace revoked images in channels.
Restore image iteration
Click on the Details link in the run task output to visit the HCP Packer dashboard. Click the learn-packer-hcp-hashicups-image
bucket and select the revoked iteration. Click Manage, then Restore iteration to restore the revoked iteration.
Confirm the action by clicking on Restore iteration.
Deploy HashiCups
Apply your configuration.
Once Terraform finishes provisioning the HashiCups instances, use cURL to query the HashiCups API using the hashicups_east_ip
address, port 19090
, and the /coffees
path.
The endpoint will return a list of coffees you could order using the HashiCups app. This shows that the application is running on the instances Terraform deployed.
Note
If you do not get a similar response, please wait a couple of minutes before trying again. It may take several minutes for the EC2 instance to finish running the set up scripts.
Verify HashiCups logs in Grafana
Add Loki as a data source to retrieve logs in Grafana. Since Grafana is running on the Loki instance, you can access it at the same IP, on port 3000
.
Use the loki_ip
output value to determine the Grafana endpoint.
In your browser, navigate to the Grafana endpoint. Login with the default credentials of admin:admin
and ignore the prompt to update the password by clicking on the Skip link at the bottom of the form. Then, click on the settings icon in the left navigation menu, then Data sources. Click on the Add data source button and then on the Loki option.
In the URL
form field, enter the loki_ip
address from the Terraform output and port 3100
. Scroll down and click the Save & test button. Grafana will display a confirmation message stating that the data source is connected.
To view the HashiCups logs, click on the compass icon in the left navigation and then click Explore.
From the dropdown menu at the top left of the page, choose Loki and then click on the blue Log browser button below it.
Loki uses several labels for the log data it receives and you can choose which logs you want to see by selecting a label and values from the provided list. Select the compose_service
label and then both api
and db
to see logs from the HashiCups API and database services. Notice that the resulting selector query updates as you make selections. Click the Show logs button to save the query.
Click the Live button on the upper right corner to have the output stream automatically.
Run the terraform/hashicups-query.sh
script to generate requests to HashiCups and watch as the output updates. The latest messages appear at the bottom of the output area.
If you want to update the golden image, rebuild it with Packer and update the bucket channel in HCP to the latest iteration. When you rebuild the HashiCups image, Packer will automatically retrieve the latest golden image as the base.
Similarly, if you wanted to update the HashiCups image, rebuild it with Packer and update the HashiCups bucket channel to the latest iteration. Then, when you re-run your Terraform configuration, Terraform will automatically deploy an instance with the latest HashiCups image.
Clean up resources
Now that you completed the tutorial, destroy the resources you created with Terraform. Enter yes
to confirm the destruction process.
Your AWS account still has AMIs and their S3-stored snapshots, which you may be charged for depending on your other usage. Destroy the AMIs and snapshots stored in your S3 buckets in both the us-east-2
and us-west-2
regions.
Tip
Remember to delete both the golden
and hashicups
images and snapshots.
In your us-east-2
AWS account, deregister the AMIs by selecting the AMIs, then click on the Actions button and the Deregister option. Delete the snapshots by selecting the snapshots, then click on the Actions button and the Delete option.
In your us-west-2
AWS account, deregister the AMIs by selecting the AMIs, then click on the Actions button and the Deregister option. Delete the snapshots by selecting the snapshots, then click on the Actions button and the Delete option.
Next steps
In this tutorial, you used Packer and the HCP Packer registry to create a golden image pipeline, allowing you to create a reusable parent image on top of which to build other AMIs. You validated the images using a Terraform Cloud run task.
You learned how to use HCP Packer registry buckets and channels to control which parent images downstream applications build upon and how to integrate them into both Packer and Terraform configurations. This workflow lets your organization build machine images for its services while reducing the overhead of managing system requirements and manually tracking image IDs.
For more information on topics covered in this tutorial, check out the following resources.
- Read more about the HCP Packer announcement
- Browse the Packer and HCP Packer documentation
- Browse the HCP Packer API documentation
- Visit the HCP Discuss forum to leave feedback or engage in discussion