HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream
Terraform
Terraform Home

Perform diagnostics on your Terraform Enterprise deployment

This topic provides instructions on how to perform diagnostic tasks to identify and resolve errors with your Terraform Enterprise deployment.

Note

The `/_health_check` endpoint is now deprecated as of 1.2.0.

Run a diagnostics check

Terraform Enterprise provides a diagnostics API endpoint to perform comprehensive health checks on all subsystems. The endpoint returns detailed status information for each component, including warnings and errors.

To run diagnostics, make a GET request to the /api/v1/diagnostics endpoint. Authentication is required.

Sample request

curl \
  --header "Authorization: Bearer $TOKEN" \
  --request GET \
  "https://tfe.example.com:8443/api/v1/diagnostics"

You can customize the timeout and filter specific checks using query parameters. Refer to the Diagnostics API reference for detailed information on parameters, responses, and examples.

Diagnostic Registry Checks

SubsystemDescription
LicenseValidates Terraform Enterprise license file format and ensures it is not expired
ConfigValidates semantic correctness of Terraform Enterprise configuration including operational mode settings, database configuration, and object storage configuration
TLSValidates TLS configuration including CA bundle file existence and validity
CloudValidates cloud object storage authentication and writability
ArchivistTests ability to write to backend storage
AtlasValidates Atlas Rails application health endpoint connectivity
VaultValidates that Vault is initialized, sealed or unsealed, and active
DatabaseTests PostgreSQL connectivity, validates version is supported, verifies required extensions are installed or user has permission to install them and checks database user has required permissions
RedisTests Redis connectivity, validates configuration, ensures Redis version is supported and verifies user has required Redis permissions
Task WorkerChecks status of all task execution drivers, validates connectivity to external services task worker depends on
DiskTests ability to write to and read from disk path. Disk operational mode only
Explorer DatabaseTests connectivity to Explorer database when configured

Generate a support bundle

A support bundle is a collection of logs and other information about your installation that you can then send to HashiCorp Support for further troubleshooting.

You can generate the support bundle using tfectl or the admin console

To generate a support bundle using the admin console:

  1. Go to https://<your-tfe-hostname>:<admin_console_port> in a browser window.
  2. Authenticate using your admin API token
  3. In the sidebar navigation, select Support bundles.
  4. Click Generate support bundle.
  5. Wait for the bundle generation to complete.
  6. Download the generated bundle for your active nodes.

The console generates a separate support bundle for each actively running node in your deployment. You can download the support bundles once all nodes have finished generating them.

Support bundle contents

Support bundles contain the following information:

  • Logs for all Terraform Enterprise services.
  • License information for the installation.
  • The Terraform Enterprise environment configuration with redacted secrets.
  • Additional diagnostic information about the container, such as the contents of /etc/hosts, disk and memory usage, and network configuration.
  • A comprehensive diagnostics check of Terraform Enterprise and all its subsystems

Logs are available within the bundle under the host directory. All other information is available within the results.json file located at the root of the bundle.

Check service status

To check the status of the services, execute the following command within Terraform Enterprise container.

$ supervisorctl status

Terraform Enterprise lists all services and their status in the console. Refer to the Terraform Enterprise services reference for information about each service. 

$ supervisorctl status

logs                             RUNNING   pid 39, uptime 1:38:49
postgres                         RUNNING   pid 103, uptime 1:38:48
redis                            RUNNING   pid 77, uptime 1:38:49
tfe:archivist                    RUNNING   pid 199, uptime 1:38:46
tfe:atlas                        RUNNING   pid 200, uptime 1:38:46
tfe:atlas-ui                     RUNNING   pid 201, uptime 1:38:46
tfe:backup-restore               RUNNING   pid 203, uptime 1:38:46
tfe:licensing                    RUNNING   pid 205, uptime 1:38:46
tfe:metrics                      RUNNING   pid 211, uptime 1:38:46
tfe:nginx                        RUNNING   pid 215, uptime 1:38:46
tfe:outbound-http-proxy          RUNNING   pid 220, uptime 1:38:46
tfe:sidekiq                      RUNNING   pid 238, uptime 1:38:46
tfe:slug-ingress                 RUNNING   pid 248, uptime 1:38:46
tfe:task-worker                  RUNNING   pid 257, uptime 1:38:46
tfe:terraform-registry-api       RUNNING   pid 265, uptime 1:38:46
tfe:terraform-registry-worker    RUNNING   pid 280, uptime 1:38:46
tfe:terraform-state-parser       RUNNING   pid 291, uptime 1:38:46
tfe:tfe-health-check             RUNNING   pid 298, uptime 1:38:46
tfe:vault                        RUNNING   pid 309, uptime 1:38:46
tfe-next                         RUNNING   pid 40, uptime 1:38:49

Inspect logs

To inspect the logs for a particular service, execute the following command within the Terraform Enterprise container where SERVICE_NAME is the name of a Terraform Enterprise service.

$ cat /var/log/terraform-enterprise/SERVICE_NAME.log

For example, we can see why tfe:licensing exited.

$ cat /var/log/terraform-enterprise/licensing.log
{"@level":"info","@message":"initializing database","@module":"tfe-licensing","@timestamp":"2023-05-10T20:46:26.379084Z"}
{"@level":"error","@message":"error opening database connection","@module":"tfe-licensing","@timestamp":"2023-05-10T20:46:26.399064Z","error":"failed to connect to `host=/var/run/postgresql user=terraform-enterprise database=`: server error (FATAL: role \"terraform-enterprise\" does not exist (SQLSTATE 28000))"}

Run Kubernetes in debug mode

Terraform Enterprise dispatches plans and applies via jobs when running inside Kubernetes. These jobs are removed immediately after their execution, which can make it hard to understand if a job failed due to cluster-specific errors.

To make troubleshooting easier in these scenarios, it is possible to keep the kubernetes jobs alive for a limited period of time, after which they will get garbage collected by the cluster. To enable this, provide the following environment variables to the deployment, either via the env.variables entry in the values.yaml override, or via the ConfigMap attached to the deployment holding all of the environment variables.

Edit this page on GitHub