Consul-Terraform-Sync architecture

This page describes Consul-Terraform-Sync (CTS), a tool for managing network infrastructure in near real-time.

CTS runs as a daemon and integrates your Consul service network with your Terraform network infrastructure. CTS automatically scales both services and their underlying infrastructure up and down in response to network events such as traffic spikes and physical datacenter outages.

Overview

The following diagram shows the CTS workflow as it monitors the Consul service catalog for updates.

1. CTS monitors the state of Consul’s service catalog and its KV store.
2. CTS detects a change.
3. CTS prompts Terraform to update the state of the infrastructure.

Watcher and views

CTS uses Consul's blocking query functionality to monitor Consul for updates. If an endpoint does not support blocking queries, CTS uses polling to watch for changes instead. These mechanisms are referred to in CTS as watchers.

The watcher maintains a separate thread for each value monitored, and runs any tasks that depend on the watched value when it detects an update. These threads are referred to as views. For example, a view may run a task to update a proxy when the watcher detects that an instance has become unhealthy.

Tasks

A task is the action that is triggered by the updated data. For example, a task can automatically update a firewall security policy rule managed with Terraform to include IP addresses of Consul services as soon as the watcher detects the change.

Drivers

A driver encapsulates the resources required to communicate the updates to the network infrastructure. CTS supports two drivers:

• Terraform driver
• HCP Terraform driver
  Enterprise

Each driver includes a set of providers that enable support for a wide variety of infrastructure applications.

State storage and persistence

CTS stores associated Terraform state data as well as CTS task and event data.

Terraform state information

By default, CTS stores Terraform state data in the Consul KV. You can specify where this information is stored by configuring the backend setting in the Terraform driver configuration. The data persists if CTS stops and the backend is configured to a remote location.

CTS task and event data

By default, CTS stores task and event data in memory. This data is transient and does not persist.

If you configure CTS to run with high availability enabled, CTS stores the data in the Consul KV. High availability is an Enterprise feature that promotes CTS resiliency. When high availability is enabled, CTS stores and persists task changes and events that occur when an instance stops.

The data stored when operating in high availability mode includes task changes made using the task API or CLI. Examples of task changes include creating a new task, deleting a task, and enabling or disabling a task. You can empty the leader’s stored state information by starting CTS with the -reset-storage flag.

Instance compatibility checks for high availability

If you run CTS with high availability enabled, CTS performs instance compatibility checks to ensure that all instances in the cluster behave consistently. Consistent instance behavior enables CTS to properly perform automations configured in the state storage.

The CTS instance compatibility check reports an error if the task's module is configured with a local module, but the module does not exist on the CTS instance. Refer to the Terraform documentation for additional information about module sources. Example log:

[ERROR] ha.compat: error="compatibility check failure: stat ./example-module: no such file or directory"

Refer to CTS error messages for additional information.

CTS instances perform a compatibility check on start-up based on the stored state, and then every five minutes after starting. If the check detects an incompatible CTS instance, it generates a log so that an operator can address it.

CTS logs the error message and continues to run when it finds an incompatibility. CTS can still elect an incompatible instance to be the leader, but tasks affected by the incompatibility do not run successfully. This can happen when all active CTS instances enter once-mode and run the tasks once when initially elected.

Security guidelines

We recommend following the network security guidelines described in the Secure Consul-Terraform-Sync for Production tutorial. The tutorial contains a checklist of best practices to secure your CTS installation for a production environment.