Continuous validation

HCP Terraform can perform automatic health assessments in a workspace to assess whether its real infrastructure matches the requirements defined in its Terraform configuration. Health assessments include the following types of evaluations:

Drift Detection: Think of your Terraform configuration as the blueprint for your infrastructure. Drift detection automatically spots if the real thing doesn't match the plan anymore. This helps you fix problems before they break things.
Continuous validation: Terraform doesn't just build your infrastructure – it enforces your standards over time. Set up rules for security, cost, or anything important, and Continuous Validation checks that they're always being met. For more details on Continuous Validation, refer to Docs for more details on Continuous Validation.

Note: Health assessments are available in HCP Terraform Plus. Refer to HCP Terraform pricing for details. In Terraform Enterprise, health assessments are part of the Governance & Policy module.

Before reviewing the best practices please review the official documentation for Health Assessments in HCP Terraform here.

Drift Detection

Drift detection in HCP Terraform is designed to ensure the integrity and consistency of your infrastructure. This feature identifies when the actual state of your infrastructure deviates from the expected state defined in your Terraform configurations, a scenario commonly referred to as configuration drift.

It’s important to understand the differences between configuration drift and state drift. Configuration drift is a change in the configuration of an application or infrastructure. State drift is a configuration state where a resource's actual configuration differs from its expected configuration.

Configuration drift is a change in the configuration of an application or infrastructure. This occurs when changes are made directly to the infrastructure outside of Terraform's managed processes, leading to discrepancies between the live state and the code-defined desired state. Common causes include manual interventions, emergency fixes, or changes applied through other automation tools.
Configuration drift differs from state drift. Drift detection does not detect state drift. State drift occurs when external changes affecting remote objects do not invalidate your infrastructure configuration.

Limitations

HCP Terraform's drift detection is a powerful tool, but it is important to understand its limitations. While it excels at identifying configuration drift for resources it actively manages, there are certain limitations to keep in mind:

Unmanaged attributes: Drift detection focuses on attributes explicitly managed by Terraform. Changes made directly to resource attributes outside of Terraform's control will not be flagged. For example, if you manually modify a security setting on a virtual machine instance, drift detection wouldn't detect this change.
External additions: HCP Terraform can't detect resources that are added to your infrastructure entirely outside of Terraform's purview. For instance, if you manually create new IAM users within your cloud environment, drift detection wouldn't identify these additions.

Prescriptive Guidance

With Organization owner permissions you can enforce health assessments for all workspaces or allow the setting to be controlled at the individual workspace level. Enforcing health assessments at an organization level overrides workspace-level settings. HashiCorp recommends that Health assessments feature be enabled for all workspaces. This global setting is available under the main Settings -> Health page.

The Platform Team admin optionally can elect to enable Health Assessments on specific workspaces in the following circumstances:

Rate limiting from the provider API is triggered - requiring reduction in API calls.
Terraform Enterprise (Self-hosted) - If load in your Terraform Enterprise instance or its agents is too great and vertically scaling the compute layer is undesirable or impossible (TFE admins have the ability to adjust the frequency health assessments are performed)
Remediating every drift can lead to an increase in operational tasks.

Terraform Enterprise administrators can modify their installation's assessment frequency and number of maximum concurrent assessments from the admin settings console.

We recommend that Workspace notifications is enabled and the Workspace Admin/Owners are notified (e.g via Slack, Email) when a workspace has a health assessment issue. This will help the relevant team take immediate action to resolve issues with the infrastructure.

Drift Resolution workflow

Once a drift is detected, the workspace can notify the application team and it is their responsibility to decide the best way to resole the drift.

The Platform team can also use HCP Terraform explorer to review which workspaces have drift and contact the application teams to resolve the drift.

On the organization's workspaces page, HCP Terraform displays a health warning status for workspaces with infrastructure drift or failed continuous validation checks.

On the right of a workspace’s overview page, HCP Terraform displays a Health bar that summarizes the results of the last health assessment.

The Drift summary shows the total number of resources in the configuration and the number of resources that have drifted.
The Checks summary shows the number of passed, failed, and unknown statuses for objects with continuous validation checks.
Overwrite Drift: For undesired changes, initiate a new Terraform plan and apply it to revert resources to their configuration-defined state.
Update Terraform Configuration: If you want the drift's changes, modify your Terraform configuration to include the changes and push a new configuration version. This prevents Terraform from reverting the drift during the next apply.

In HCP Terraform, refresh state and update Terraform configuration are two distinct operations that play critical roles in infrastructure management, especially in the context of drift detection and remediation. Understanding the differences between these operations and knowing when to use each is vital for effective infrastructure as code practices.

Refresh state: Refreshing the state is the process of updating Terraform's internal state file to match the actual state of the infrastructure as it exists in the cloud or on-premises environments. This operation does not modify the infrastructure; instead, it updates Terraform's record (the state file) to reflect any changes that have occurred outside of Terraform's management. The primary purpose of refreshing the state is to ensure that Terraform's understanding of the infrastructure is accurate. This accuracy is essential for identifying drift, the divergence between the desired state defined in Terraform configurations and the actual state of the infrastructure.
Update Terraform configuration: Updating a Terraform configuration version involves submitting updated Terraform configuration files to HCP Terraform and executing a plan and apply operation based on these updates. This process changes the infrastructure to match the desired state defined in the new configuration version.
- Update Terraform configurations: Once drift is identified, update your Terraform configurations to either realign the infrastructure to the desired state or update the desired state to incorporate intentional changes made outside of Terraform.
- Review and apply changes: Submit the updated configurations as a new configuration version in HCP Terraform. Take advantage of HCP Terraform's planning and review process to ensure all team members understand the changes and their impacts before applying them.
- Monitor and validate: After applying the new configuration version, monitor the apply operation's outcome to ensure the intended changes were successfully implemented and validate that the infrastructure state aligns with the desired configuration.

Additional guidance

Continuous integration/continuous deployment (CI/CD): Integrate both the state refresh and apply operations into your CI/CD pipelines to facilitate continuous monitoring and timely remediation of drift.
Version control and code reviews: Use version control systems for managing Terraform configurations. Implement a code review process for changes to Terraform files to ensure accuracy and intent of changes before they are applied.
Change management: Incorporate these Terraform operations within your organization's change management framework to ensure changes are tracked, audited, and aligned with organizational policies and compliance requirements.
Documentation and training: Document your workflow and operations, including state refresh and configuration updates, and provide training to your team. This ensures a consistent approach to managing and remediating drift across your organization.

For a detailed example, refer to the manage resource drift tutorial.

Continuous validation

We recommend that organizations use continuous validation for their workspaces especially to track the health of critical resources.

Continuous validation is the second component of the health check feature in HCP Terraform and TFE. The purpose of health checks is that by running regularly, they allow the infrastructure engineer to detect (and proactively fix) issues before they impair the next Terraform apply operation.

While drift detection flags out-of-band changes to the managed infrastructure that can affect a Terraform apply operation, continuous validation addresses use cases where more customizable detection rules are necessary. Some examples of those use cases are:

To flag issues with infrastructure elements managed outside of the workspace, but that affect the health of this workspace’s resources.
To identify if cloud services or third-party tools have detected issues with the managed infrastructure resources.

Please review official documentation on continuous validation before implementing.

Why Continuous Validation?

Failed infrastructure changes can be costly to the organization as they may introduce project delays and could expose the organization to operational or security risks. Adopting continuous validation gives infrastructure teams advance notice of issues preventing successful changes in configuration. These issues can then be addressed and failed infrastructure changes avoided.

Best Practice Recommendation

When a new workspace is created to manage infrastructure, continuous validation should be enabled (either explicitly at the workspace level or implicitly at the organization level).

The infrastructure as code engineer should also include in the Terraform configuration code the necessary logic to validate important components of the infrastructure whose health may change over time and prevent a successful Terraform Apply run the next time an infrastructure configuration change is necessary.

In addition, if infrastructure changes do fail in the future because of a condition that was not checked, an engineer should update the Terraform configuration to incorporate this new validation. If applicable, this new pattern should be applied to existing infrastructure code and added to the checklist for future Terraform configurations.

Rule of thumb on which resources should have continuous validation applied:

Check the status of any critical resource that can fail (e.g VM)
1. This is not necessary for certain resources such as S3 buckets which are native to the cloud provider.
Validity of resources such as certificates that have user defined time frames but whose failure can have an impact on the application stack.

Implementation guidance

In this section, we’ll go over the steps and recommendations to implement continuous validation.

Requirements

Continuous validation requires the use of Terraform language features that aren’t available in older versions of the runtime. The table below lists the language feature and the minimum version of the Terraform runtime that supports it.

Language feature	Version requirement	Useful links
Preconditions and postconditions	1.2 and later	Terraform 1.2 Improves Exception Handling and Updates to the CLI-driven Workflow Preconditions and postconditions
Check block	1.5 and later	Terraform 1.5 brings config-driven import and checks Checks with assertions

Permissions required

To configure continuous validation, you’ll need the following permissions:

To change organization health settings, you must be a member of the owners team.
To change a workspace’s health settings or trigger an on-demand health assessment, you must be an administrator for that workspace.

To view the continuous validation status:

To view health status for a workspace, you need read access to that workspace.
To view the status of all workspaces in the HCP Terraform Explorer, you need to be a member of the owners team or have the “View all workspaces” permission or better.

Enabling health assessments

Continuous validation being a component of the health assessment feature, to use continuous validation you must enable the feature.

You may enable health assessments at the organization level or at the workspace level, using the WebUI, the API or using Terraform code. Whenever possible we recommend using Terraform code to configure your HCP Terraform or Terraform Enterprise instance.

Enable health assessments at the …	Using the WebUI	Using the API	Using Terraform
Organization level	Managing Settings section	Organizations API	TFE provider: tfe_organization resource
Workspace level	Enable health assessments section	Workspaces API	TFE provider: tfe_workspace resource

Before adopting continuous validation, review your inventory of workspaces and specifically the Terraform version they are configured to use. Also review your code base and any Terraform required version constraint. If one or more workspaces do not meet the minimum requirements, they will not fully benefit from health assessments: Drift detection may be available (support was added in Terraform 0.15.4) but not the continuous validation health assessments.

If you decide to enable health assessments at the workspace level and you are using Terraform code to configure your HCP Terraform or Terraform Enterprise instance, you can use the flexibility of the Terraform language to selectively activate health assessments on workspaces that meet the requirements.

Defining custom assertions and checks

While continuous validation will evaluate preconditions, postconditions, and check blocks as part of an assessment, we recommend using check blocks for post-apply monitoring. Check blocks do not stop a Terraform run when they fail (contrary to preconditions and postconditions) which makes them ideal tools for monitoring.

Here is a list of useful resources to learn more about the check block:

Checks (documentation)
Use checks to validate infrastructure (tutorial)
Health checks with Terraform Cloud continuous validation (demo)

Here is a list of useful resources showcasing examples using the check block:

Authentication considerations

Health checks require valid credentials in the workspace/run context - If you inject temporary credentials via external pipelines, which then are expired when health/drift runs happen, we recommend adopting “Dynamic provider credentials” which provide the same benefits of temporary credentials while being compatible with health checks

Viewing health status and getting notified

You can view the workspace(s) health status:

From the workspace summary page (the Health panel on the right side of the interface).
From the workspace continuous validation status page (from the workspace page, go to Health > Continuous validation).
From the Projects & Workspaces view.
From the HCP Terraform Explorer.

You can also configure notifications to get alerted on a number of events, including:

When a continuous validation check returns unknown or failed.
When a health assessment cannot be completed successfully.

Notifications can be sent to a number of different supported destinations:

Email
Slack
Microsoft Teams
Webhooks (most flexible)

This feature is a workspace-level configuration, which means that you’ll need to configure it for every workspace that you need to monitor and get a notification. If you are using Terraform code to configure your HCP Terraform or Terraform Enterprise instance, you can use the flexibility of the Terraform language to easily configure notifications on workspaces that need it.