Observability setup

This page covers initial observability setup for Terraform Enterprise. Recurring analysis, alerting, and response belong in the Administration Guide.

Configuring data collection

In this section we'll cover how to set the data collection for:

1. Collecting metrics and logs (including audit trail logs) on Terraform Enterprise
2. Audit trail logs on HCP Terraform
3. Collecting metrics and logs on HCP Terraform agents (applicable for HCP Terraform and Terraform Enterprise)

Terraform Enterprise metrics and logs

Configuring metrics collection on Terraform Enterprise

Explicitly enable Terraform Enterprise metrics collection as it is not enabled by default. Do this by setting the TFE_METRICS_ENABLE parameter to true. Next, configure your monitoring tool of choice to periodically query the Terraform Enterprise metrics endpoint to collect and store this information.

Capture metrics in two formats using the metrics endpoint URL. The table below provides the available options as well as the URL to use. We recommend capturing metrics over an encrypted connection.

Terraform Enterprise computes an aggregate metric value from a 5-second sample. Terraform Enterprise keeps this value in memory for 15 seconds before flushing it. This means that if the monitoring tool pooling frequency is greater than 15 seconds (for example every 60 seconds), you may be missing information necessary to detect short-lived issues.

If you are running multiple Terraform Enterprise instances, you must collect the metrics from each deployed Terraform Enterprise instance and aggregate the information to have a global view of the Terraform Enterprise service.

Configuring log collection on Terraform Enterprise

Terraform Enterprise emits logs to standard output and standard error. We recommend collecting Terraform Enterprise logs in a central location, preferably using a specialized tool that provides searching and alerting capabilities, although we support sending logs to object storage for example.

There is a limited number of supported log destinations.

Implementing audit trail on Terraform Enterprise

Terraform Enterprise generates audit logs alongside its service logs. Use the official Terraform Enterprise logs documentation(opens in new tab) as the source of truth for supported destinations, forwarding options, event types, and the current audit log format.

When forwarding Terraform Enterprise logs to a Security Information and Event Management solution (SIEM) or another centralized logging platform, parse the documented audit fields. The current documentation identifies the audit log component as terraform-enterprise.audit and documents event types such as authentication success, authentication failure, and CSRF violations.

HCP Terraform audit trail logs

HCP Terraform features an Audit Log API endpoint that you must use to collect the audit events and store them in the appropriate system. To implement this solution, you need the following.

• A method to schedule and automate the audit events collection,
• A secure storage solution to store the audit events, and
• A data lifecycle solution to correctly dispose of the audit events once they are no longer required.

If you use a Security Information and Event Management system (SIEM), this must be the destination for those audit events. Suppose you are not using a SIEM but instead are using a centralized log management solution (Datadog, New Relic, Elastic, and so on). In that case, you must send those audit events to your centralized log management system. If neither of these solutions is available, still collect those audit events and store them securely using an object storage solution, such as AWS S3.

Metrics and logs on HCP Terraform agents

Configuring HCP Terraform agent metrics collection

The HCP Terraform agent binary exposes telemetry data using the OpenTelemetry protocol. This behavior allows the user to use a standard OpenTelemetry collector to push the metrics to a monitoring solution that supports the protocol, such as Prometheus or Datadog.

Because of that, to collect telemetry data from the agent, you need to have:

• A way to deploy and operate OpenTelemetry collector(s)
• A monitoring system that can integrate with OpenTelemetry collectors

Details about the selection of such a monitoring system or the operations of OpenTelemetry collectors are beyond the scope of this document. However, we provide some guidelines regarding integrating OpenTelemetry collectors with HCP Terraform agents.

Because OpenTelemetry is a push system, you must start the collector before the HCP Terraform agent. Conversely, shut down the collector after you have stopped all HCP Terraform agents using that collector. We recommend having a one to one ratio of HCP Terraform agent instance and OpenTelemetry collector for long running HCP Terraform agents as this simplifies management.

The OpenTelemetry integration tags metrics with a number of useful fields, including the agent pool ID (agent_pool_id) and the agent name (agent_name). If you do not already have a naming convention for your HCP Terraform agents, then we recommend building one, as it helps you organize your dashboard with the metrics collected from HCP Terraform agents. You can then set the agent's name at startup time using the TFC_AGENT_NAME environment variable or the -name command line option.

Configuring HCP Terraform agent log collection

If you are using HCP Terraform agents in your Terraform enter deployment then configure log collection for agents.