Terraform
Agent logging
HCP Terraform Agents log helpful messages that tell operators about agent behavior, including communication with HCP Terraform APIs, specific commands run, actions taken, and runtime management.
Log output
HCP Terraform Agents write log messages directly to stdout/stderr. This lets the operator capture the logs in a variety of different logging systems, gives CLI users a way to see a log of the agent's behavior directly in their terminal. By default, the agent does not automatically persist the log output in any way. The user must write the logs to a file or collect them with a process or container supervisor if persistence is required.
Log verbosity and levels
The volume of logs and the level of detail they contain are controlled by log levels. There are 5 levels supported by the agent:
errorcontains only critical error messages. The agent process is otherwise silent.warncontains all error-level messages, as well as informational messages such as system messages directly from the HCP Terraform platform.infocontains all warn-level messages and high-level information about the agent and the workflows it is executing. In normal circumstances, this is the safest and most helpful log level for day to day operation.debugcontains all info-level messages, plus additional informational messages which provide further context about behavior, data, and events.tracecontains all debug-level messages, plus verbose process logs such as the line-by-line output of theterraformcommand.
Data format
By default, HCP Terraform Agents emit log lines in a human-readable text format. This is convenient for running the HCP Terraform Agent locally and streaming the logs directly to a terminal, or for use in log systems where raw logs are consumed directly by operators. The default text format looks something like the following:
2023-04-14T17:12:43.002Z [INFO] core: Job received: job_type=plan job_id=run-xxx
It is also possible to configure the HCP Terraform Agent to produce JSON-
formatted logs. This format will cause each log line to be serialized as an
individual JSON object, and is more ideal for logging systems which are capable
of parsing and performing post-processing on log data for each line. JSON
logging mode is enabled by passing the -log-json CLI flag, or setting the
environment variable TFC_AGENT_LOG_JSON=1. The JSON format contains additional
verbose information in each log message, and looks something like this:
{"@level":"info","@message":"Job received","@module":"core","@timestamp":"2023-04-13T23:35:28.653853Z","agent_id":"agent-xxx","agent_name":"name","agent_pool_id":"apool-xxx","agent_version":"1.7.1","job_id":"run-xxx","job_type":"plan"}
Log data sensitivity
Log levels have a progressive level of data sensitivy. The info, warn, and
error levels are considered generally safe for log collection and don't
include sensitive information. The debug log level may include internal
system details, such as specific commands and arguments including paths to user
data on the local filesystem. The trace log level is the most sensitive and
may include personally identifiable information, secrets, pre-authorized
internal URLs, and other sensitive material.
Flash Messages
Flash Messages are a type of log that HashiCorp may send to agents from time to time. These messages may be used to communicate important or breaking changes to the agent. Flash Messages will be emitted when HashiCorp adds a new one, or when starting up an agent for the first time. Their output looks something like:
2021-09-22T15:20:59.269Z [WARN] notice: A breaking change is incoming.
Flash Messages are version specific, and may only apply to the specific version of the agent you are running.
Adding monitoring and alerting for these notice messages may help you operate HCP Terraform Agents more easily.