Terraform Cloud Agents log helpful messages that tell operators about agent behavior, including communication with Terraform Cloud APIs, specific commands run, actions taken, and runtime management.
Terraform Cloud Agents write log messages directly to stdout/stderr. This lets the operator capture the logs in a variety of different logging systems, gives CLI users a way to see a log of the agent's behavior directly in their terminal. By default, the agent does not automatically persist the log output in any way. The user must write the logs to a file or collect them with a process or container supervisor if persistence is required.
The volume of logs and the level of detail they contain are controlled by log levels. There are 5 levels supported by the agent:
errorcontains only critical error messages. The agent process is otherwise silent.
warncontains all error-level messages, as well as informational messages such as system messages directly from the Terraform Cloud platform.
infocontains all warn-level messages and high-level information about the agent and the workflows it is executing. In normal circumstances, this is the safest and most helpful log level for day to day operation.
debugcontains all info-level messages, plus additional informational messages which provide further context about behavior, data, and events.
tracecontains all debug-level messages, plus verbose process logs such as the line-by-line output of the
By default, Terraform Cloud Agents emit log lines in a human-friendly text format. This is convenient for running the TFC Agent locally and streaming the logs directly to a terminal, or for use in log systems where raw logs are consumed directly by operators. The default text format looks something like the following:
2023-04-14T17:12:43.002Z [INFO] core: Job received: job_type=plan job_id=run-xxx
It is also possible to configure the Terraform Cloud Agent to produce JSON-
formatted logs. This format will cause each log line to be serialized as an
individual JSON object, and is more ideal for logging systems which are capable
of parsing and performing post-processing on log data for each line. JSON
logging mode is enabled by passing the
-log-json CLI flag, or setting the
TFC_AGENT_LOG_JSON=1. The JSON format contains additional
verbose information in each log message, and looks something like this:
Log levels have a progressive level of data sensitivy. The
error levels are considered generally safe for log collection and don't
include sensitive information. The
debug log level may include internal
system details, such as specific commands and arguments including paths to user
data on the local filesystem. The
trace log level is the most sensitive and
may include personally identifiable information, secrets, pre-authorized
internal URLs, and other sensitive material.
Flash Messages are a type of log that HashiCorp may send to agents from time to time. These messages may be used to communicate important or breaking changes to the agent. Flash Messages will be emitted when HashiCorp adds a new one, or when starting up an agent for the first time. Their output looks something like:
2021-09-22T15:20:59.269Z [WARN] notice: A breaking change is incoming.
Flash Messages are version specific, and may only apply to the specific version of the agent you are running.
Adding monitoring and alerting for these
notice messages may help you operate Terraform Cloud Agents more easily.