HashiConf Last chance to register for our cloud conference October 10-12. Register today
Consul
Network Automation with CTS

Secure Consul-Terraform-Sync for production

  • 13min
  • |
  • Consul
  • Terraform

HashiCorp's Network Infrastructure Automation Integration Program allows network administrators to build integrations that automatically apply network and security infrastructure changes initiated by changes in the Consul service catalog.

Network Infrastructure Automation is carried out by Consul-Terraform-Sync, a multi-platform tool, that is able to connect to the Consul catalog to monitor changes in the services' state and health. The tool leverages Terraform as the underlying automation tool and utilizes the Terraform provider ecosystem to drive relevant change to your network infrastructure.

In this tutorial, you will learn the recommended practices to secure your Consul-Terraform-Sync instances and prepare them for a production environment including how to:

  • Set up a dedicated instance
  • Install Consul-Terraform-Sync
  • Create a dedicated user with specific permissions
  • Create a Consul ACL token
  • Secure the Terraform provider

Prerequisites

This tutorial assumes you have an HashiCorp Cloud Platform (HCP) Consul cluster deployed in your HCP account, and that you have correctly configured an AWS peering connection between your HVN to your AWS VPC. The peering connection allows the Consul servers deployed in your HCP Consul cluster to communicate with clients that you deployed in your AWS VPC. Your Consul clients must have a valid L3 route configured that allows them to communicate with your HCP Consul cluster's private IP. They cannot use a public IP even if you have one configured.

Review the manual deployment tutorial or the automatic deployment tutorial for guidance on creating the necessary network configuration. To complete this tutorial, we expect that you have already provisioned the following resources.

  • An HCP HashiCorp Virtual Network (HVN)
  • An HCP Consul cluster
  • An AWS VPC
  • An AWS peering connection between your HVC and your VPC
  • An AWS EC2 instance deployed to the peered VPC with a Consul client installed

If you do not yet have an EC2 instance provisioned in the target VPC, refer to the official AWS Documentation for guidance on how to create an EC2 instance that you can use with this tutorial. Afterwards, refer to the connect a Consul client to HCP Consul tutorial in order to configure a Consul client on your Consul-Terraform-Sync EC2 instance.

Note

In this tutorial, the AWS virtual machine (EC2) instance that you will run Consul-Terraform-Sync on must be deployed to the same VPC that you have peered with your HVN.

Use a dedicated instance

The Consul-Terraform-Sync daemon could have access to critical secrets for your environment's network infrastructure, depending on your configuration and use case. Using a hardened, dedicated host, for supporting these sensitive operations is highly recommended. Workload orchestrators, such as HashiCorp Nomad, also provide benefits of ensuring uptime and isolation.

While we recommend to run Consul-Terraform-Sync on the same node as a Consul agent, this is not a strict requirement. The daemon can run on any highly available node which has connectivity to the agents in Consul datacenter and the downstream network infrastructure.

For performance and availability reasons, you should not configure Consul-Terraform-Sync to connect directly to a server node nor to a node already running other services that are critical for your infrastructure.

We recommend that you run Consul-Terraform-Sync on a dedicated instance with a Consul client installed.

Install Consul-Terraform-Sync

You can install Consul-Terraform-Sync manually either by using a pre-compiled binary or by building it yourself from source.

In this tutorial, you will use an HCP Consul cluster. The HCP servers are running the Enterprise version of Consul. You can therefore use the Enterprise version of Consul-Terraform-Sync without the need for additional license. Learn more about the Enterprise functions of Consul-Terraform-Sync here.

You will install Consul-Terraform-Sync on an Ubuntu EC2 instance. For more information regarding installation on different distributions and platforms, please refer to the Consul-Terraform-Sync installation guide.

HashiCorp officially maintains and signs packages for the Ubuntu Linux distribution.

Install the following packages on your machine so you can install the CTS binary.

$ echo $PATH
$ apt-get install curl gnupg lsb-release

Add the HashiCorp GPG key.

$ curl --fail --silent --show-error --location https://apt.releases.hashicorp.com/gpg | \
      gpg --dearmor | \
      sudo dd of=/usr/share/keyrings/hashicorp-archive-keyring.gpg

Add the official HashiCorp Linux repository.

$ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
 sudo tee -a /etc/apt/sources.list.d/hashicorp.list

Update the list of available packages and their versions.

$ sudo apt-get update

Find available enterprise versions to install.

$ sudo apt-cache policy consul-terraform-sync-enterprise

consul-terraform-sync-enterprise:
  Installed: (none)
  Candidate: 0.6.0+ent
  Version table:
     0.6.0+ent 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages
     0.5.2+ent 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages
     0.5.1+ent 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages
     0.5.0+ent 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages
     0.4.3+ent 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages
     0.4.2+ent 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages
     0.4.1+ent-2 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages
     0.4.1+ent 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages
     0.4.0+ent-2 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages
     0.4.0+ent 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages
     0.3.0+ent 500
        500 https://apt.releases.hashicorp.com bullseye/main amd64 Packages

Install a specific version.

$ sudo apt-get install consul-terraform-sync-enterprise=0.4.1+ent

Install latest version.

$ sudo apt-get install consul-terraform-sync-enterprise

Once installed, verify the installation works by prompting the help option.

$ consul-terraform-sync -h

Usage of consul-terraform-sync:
  -config-dir
        A directory to load files for configuring Sync. Configuration files require an .hcl or .json file extention in order to specify their format. This option can be specified multiple times to load different directories.
  -config-file
        A file to load for configuring Sync. Configuration file requires an .hcl or .json extension in order to specify their format. This option can be specified multiple times to load different configuration files.
  -inspect false
        Run Sync in Inspect mode to print the proposed state changes for all tasks, and then exits. No changes are applied in this mode.
  -inspect-task
        Run Sync in Inspect mode to print the proposed state changes for the task, and then exits. No changes are applied in this mode.
  -once false
        Render templates and run tasks once. Does not run the process as a daemon and disables buffer periods.
  -version false
        Print the version of this daemon.

Setup user account and file permissions

The Consul-Terraform-Sync daemon does not require root or other administrative privileges to operate.

To ensure privilege isolation, we recommend to create a dedicated user and group, with limited permissions, to run Consul-Terraform-Sync. With a dedicated user and group you can appropriately scope directory and file permissions for your operating environment.

You will create a dedicated consul-nia system user, along with the supporting directories and configuration file; then you will secure those permissions using chown and chmod.

Create dedicated user and group.

$ sudo useradd --system --home /etc/consul-nia.d --shell /bin/false consul-nia

Create data and configuration directory.

$ sudo mkdir -p /opt/consul-nia && sudo mkdir -p /etc/consul-nia.d

Setup appropriate permissions for the directories.

$ sudo chown --recursive consul-nia:consul-nia /opt/consul-nia && \
  sudo chmod -R 0750 /opt/consul-nia && \
  sudo chown --recursive consul-nia:consul-nia /etc/consul-nia.d && \
  sudo chmod -R 0750 /etc/consul-nia.d

Configure ACL privileges for Consul-Terraform-Sync

Consul-Terraform-Sync needs to access different data in the Consul catalog and if you're using Consul as Terraform backend, you'll also need privileges to store the Terraform state into Consul's KV store.

In production environments, it is expected for Consul to be configured with Access Control Lists (ACLs) enabled. In order for Consul-Terraform-Sync to access information from Consul, a token needs to be passed to its configuration.

The two following sections review the privileges required by the token to accomplish the two different operations, catalog monitoring and Terraform state management.

Depending on the configuration choices you can select for one of the following options:

  • combine the policies in one single token for the Consul-Terraform-Sync instance used for both the catalog monitoring and the Terraform backend
  • create two different tokens for the same instance of Consul-Terrafomr-Sync and use one in the consul block of the Consul-Terraform-Sync configuration and one for the backend section
  • create one token for the Consul-Terraform-Sync instance to be monitored using rules mentioned in the Consul catalog privileges section below and one using the rules for the KV and session and use them for the two different parts of the configuration.

Consul CTS service privileges

CTS automatically registers itself as a service with Consul, which requires write permissions for the CTS service. The name of the CTS service defaults to Consul-Terraform-Sync. See the consul.service_registration configuration for options to change this name or to disable this feature.

In the example below, the policy grants write permissions for the CTS service, using the default service name.

service "Consul-Terraform-Sync" {
  policy = "write"
}

Consul catalog privileges

To retrieve information about services registered with Consul, Consul-Terraform-Sync needs access to the Consul catalog.

For this reason you will need a token with the right privileges to read all the selected services to be monitored, as well as the namespaces for the services in case you have them implemented in Consul Enterprise.

You must have a policy for your token before you can create a token. Click on the following tab(s) for policy examples with different levels of security.

A basic configuration for giving Consul-Terraform-Sync read permissions over the whole Consul catalog is to provide it with a token with read permissions over all services and all nodes.
service_prefix "" {
  policy = "read"
}

node_prefix "" {
  policy = "read"
}

Consul KV privileges

CTS requires read permissions to any Consul KV pairs that are monitored by tasks. This is only applicable if any of your tasks are configured with a Consul KV condition or Consul KV module input.

This is a basic example that provides read permissions for all Consul KV entries.

key_prefix "" {
  policy = "read"
}

Consul KV privileges when used as the Terraform backend

When Terraform uses Consul as a backend, it will also need to have the necessary privileges to store the state inside the Consul KV store and to use sessions to ensure locking during the Terraform state changes.

// Consul KV backend default prefix for state files
key_prefix "consul-terraform-sync/" {
  policy = "write"
}

session_prefix "" {
  policy = "write"
}

By default Consul-Terraform-Sync will use the consul-terraform-sync KV path to store the Terraform state. However, you can configure the path in the driver.terraform.backend section of the configuration.

The example below provides privileges for all possible sessions names. Once the infrastructure is defined, it is possible to reduce the number of privileges to be limited to the name of the Consul node running Consul-Terraform-Sync.

Note

Consul is the default backend for Consul-Terraform-Sync. It is important to note that it does not support encryption. In a production environment we recommend you to use a Terraform backend that supports encryption.

Generate a Consul token for Consul-Terraform-Sync

Now you will create a policy and a token associated with it. Take the selected policy content from the last few steps that apply to your environment and put it in a file. The name of the file does not matter, but in this example it is called cts-policy.hcl. The name of the policy can also be anything - in this example, the name is the same.

$ consul acl policy create -name cts-policy -rules @cts-policy.hcl
ID:           f54a91f8-4d6c-5603-5843-dd30177310c6
Name:         cts-policy
Description:
Datacenters:
Rules:
## ...

After you create the policy, generate a token associated to this policy. Take note of your token's SecretID for later. The value in the example output below has been truncated.

$ consul acl token create -description 'consul-terraform-sync policy' -policy-name cts-policy
AccessorID:       TRUNCATED
SecretID:         TRUNCATED
Description:      consul-terraform-sync policy
Local:            false
Create Time:      2022-05-20 12:38:05.813941 +0300 EEST
Policies:
   f54a91f8-4d6c-5603-5843-dd30177310c6 - cts-policy

Secure Terraform providers configuration

To effectively communicate with your infrastructure, the Terraform provider you use might require the setting of sensitive data such as tokens or authentication parameters, or to define environment variables that might change depending on the different task that is being executed. This information should be present in the Consul-Terraform-Sync configuration file in order to have your integration completely working.

To help you avoid the use of secrets or otherwise sensitive data in your configuration file, the terraform_provider block supports dynamically loading arguments and environment variables from external sources.

Available sources for these information are environment variables, Consul KV, or Vault.

Using the task_env meta-argument or the template syntax below, you can avoid exposing sensitive values or credentials in plain text within configuration files for Consul-Terraform-Sync.

Note

task_env and the template syntax for dynamic values are only supported within the terraform_provider block.

Task environment setting

By default, Consul-Terraform-Sync enables all Terraform workspaces to inherit from its environment. This means that variables defined for the shell session used to run Consul-Terraform-Sync will be exposed to the providers too.

The task_env block is a meta-argument available for the terraform_provider block that can be used to rename or scope the available environment to a selected set of variables. Passing sensitive values as environment variables will scope the values to only the tasks that require the provider.

For example, imagine that you have a provider, named example, that allows you to pass a token using the environment variable PROVIDER_TOKEN or using the field token in the configuration. You can avoid having the token in plain text into the configuration using task_env in your Consul-Terraform-Sync configuration.

terraform_provider "example" {
  // Direct assignment of provider arguments are rendered in plain-text within
  // the Consul-Terraform-Sync configuration and the generated providers.tfvars
  // file for the corresponding Terraform workspaces.
  // token = "<token value>"

  // Dynamically assign the task's environment from the shell env, Consul KV, or
  // Vault.
  task_env {
    "PROVIDER_TOKEN" = "{{ env \"CTS_FOO_TOKEN\" }}"
  }
}

Refer to the official provider docs hosted on the Terraform Registry to find the supported environment variables for the provider you are using.

Load dynamic values

Terraform providers often require sensitive or dynamic data to be present in the configuration to successfully interact with your infrastructure providers. These variables might not be static, requiring you to change them before being able to successfully run Consul-Terraform-Sync daemon, or might contain secrets that you do not want to leave in plain text inside your configuration files.

As a best practice you should store those values elsewhere, and use one of the following approaches to populate sensitive fields in your terraform_provider block.

Environment variables

You can use environment variables to populate the filed using env.

terraform_provider "example" {
  address = "{{ env \"NODE_IP_ADDRESS\" }}"
}

Consul keys

You can query the Consul KV store, for the Consul datacenter configured in the consul block using key.

terraform_provider "example" {
  value = "{{ key \"path/example/key\" }}"
}

Note

The ACL token used for the Consul datacenter will require read privileges to the path specified in this section to be able to retrieve the values from Consul.

Vault secrets

You can use a Vault instance and its KV secrets engine using with secret.

vault {
  address = "vault.example.com"
}

terraform_provider "example" {
  token = "{{ with secret \"secret/my/path\" }}{{ .Data.data.foo }}{{ end }}"
}

Vault is an optional source that requires you to configure the Vault client with a vault block. You can access the secret using template dot notation Data.data.<secret_key>.

Warning

Consul-Terraform-Sync does not prevent sensitive values from being written to Terraform state files. We recommend securing state files in addition to securely configuring Terraform providers. Options for securing state files can be set within driver.backend based on the backend used. For example, Consul KV is the default backend and can be secured with ACLs for KV path. For other backends, we recommend enabling encryption, when applicable.

Configure Consul-Terraform-Sync process

We recommend running Consul-Terraform-Sync on a dedicated machine. In addition, you should configure CTS to launch at startup, so it can survive machine reboots.

Tip

We recommend to create a user for running Consul-Terraform-Sync. Earlier in this tutorial, you created the consul-nia user. Alternatively, if you completed the Deployment Guide, you would have created a consul user account in your system. You can reuse the consul user account for Consul-Terraform-Sync.

Configure systemd

Systemd uses documented reasonable defaults so only non-default values must be set in the configuration file.

Create a Consul service file at /usr/lib/systemd/system/consul-terraform-sync.service.

$ sudo touch /usr/lib/systemd/system/consul-terraform-sync.service

Add this configuration to the Consul-Terraform-Sync service file.

Warning

If you are using Consul-Terraform-Sync version older than v0.6.0, the start parameter is not supported when starting in daemon mode. Remove it from line 12 of the SystemD unit file below.

/usr/lib/systemd/system/consul-terraform-sync.service
1 2 3 4 5 6 7 8 9 1011121314151617181920[Unit]
Description="HashiCorp Consul-Terraform-Sync - A Network Infrastructure Automation solution"
Documentation=https://www.consul.io/docs/nia
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul-nia.d/config.hcl

[Service]
EnvironmentFile=/etc/consul-nia.d/consul-nia.env
User=consul-nia
Group=consul-nia
ExecStart=/usr/bin/consul-terraform-sync start -config-dir=/etc/consul-nia.d/
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

The service file configures the system to run Consul-Terraform-Sync using the ExecStart command and using a dedicated User and Group.

For more information on the available systemd.service settings, refer to the systemd documentation.

Create configuration and environment files

In the service file, you configured Consul-Terraform-Sync to use /etc/consul-nia.d/ as configuration directory.

Specifically the service will need two files:

  • /etc/consul-nia.d/config.hcl - The main configuration file. At startup, the service will verify that the file exists and that it's not empty.

  • /etc/consul-nia.d/consul-nia.env - The environment file used to setup environment variables for the Consul-Terraform-Sync process. The file needs to exist in order for the service to start, but it can be empty.

/usr/lib/systemd/system/consul-terraform-sync.service
[Unit]
Description="HashiCorp Consul-Terraform-Sync - A Network Infrastructure Automation solution"
Documentation=https://www.consul.io/docs/nia
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/consul-nia.d/config.hcl

[Service]
EnvironmentFile=/etc/consul-nia.d/consul-nia.env
User=consul-nia
Group=consul-nia
ExecStart=/usr/bin/consul-terraform-sync start -config-dir=/etc/consul-nia.d/
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGTERM
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Earlier you created the configuration folder /etc/consul-nia.d. Now, create the configuration files.

$ sudo touch /etc/consul-nia.d/config.hcl

Use the following as a template for writing a configuration that fits your environment. Refer to Consul-Terraform-Sync documentation for a list of all available parameters. In the consul block, set token to the SecretID value of the token you created earlier in the Generate a Consul token for Consul-Terraform-Sync step.

/etc/consul-nia.d/config.hcl
# Consul Block
consul {
  address = "127.0.0.1:8500"
  token = "<YOUR-CTS-TOKEN>"
}

## Global Config
log_level = "INFO"

port = 8558

working_dir = "/opt/consul-nia"

syslog {  
  enabled = "true"
  facility = "local0"
  name = "consul-terraform-sync"
}

buffer_period {
  enabled = true
  min = "5s"
  max = "20s"
}

# Driver "terraform" block
driver "terraform" {
 # version = "0.14.0"
  path = "/opt/consul-nia"
  log = false
  persist_log = false
}

# Task Block
task {
 name        = "learn-cts-example"
 description = "Example task with two services"
 module      = "findkim/print/cts"
 version     = "0.1.0"
 condition "services" {
  names = ["web", "api"]
 }
}

Create the file to set up the environment variables for the CTS process.

$ sudo touch /etc/consul-nia.d/consul-nia.env

Then grant the consul-nia user permissions to access the file.

$ sudo chown --recursive consul-nia:consul-nia /etc/consul-nia.d

Set up the Enterprise license for Consul-Terraform-Sync (optional)

Note

If you are using Consul-Terraform-Sync enterprise version lower than 0.6.0, you will need to manually set up the Enterprise license. However, Consul-Terraform-Sync enterprise version 0.6.0 and higher support automatic retrieval of the Consul license from your own Consul enterprise cluster.

In this tutorial, you are using an HCP Consul cluster that already comes with an Enterprise license that you can also use with Consul-Terraform-Sync.

Set up an automatic license retrieval configuration for Consul-Terraform-Sync.

/etc/consul-nia.d/license.hcl
123456license {
  path = "/opt/consul-nia/license.lic"
  auto_retrieval {
    enabled = true
  }
}

Start the Consul-Terraform-Sync process

Enable and start Consul-Terraform-Sync using the systemctl command responsible for controlling systemd managed services. Check the status of the Consul-Terraform-Sync service using systemctl.

First, make sure the SystemD reloads its unit file configurations.

$ sudo systemctl daemon-reload

Then, enable the CTS system process.

$ sudo systemctl enable consul-terraform-sync

Then, start the CTS system process.

$ sudo systemctl start consul-terraform-sync

Finally, check the status for the process. The output should indicate the service is running.

$ sudo systemctl status consul-terraform-sync

● consul-terraform-sync.service - "HashiCorp Consul-Terraform-Sync - A Network Infrastructure Automation solution"
     Loaded: loaded (/lib/systemd/system/consul-terraform-sync.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-11-11 16:48:36 UTC; 1s ago
       Docs: https://www.consul.io/docs/nia
   Main PID: 12909 (consul-terrafor)
      Tasks: 24 (limit: 226)
     Memory: 63.3M
        CPU: 993ms
     CGroup: /system.slice/consul-terraform-sync.service
             ├─12909 /usr/bin/consul-terraform-sync start -config-dir=/etc/consul-nia.d/
             ├─12947 /opt/consul-nia/terraform init -no-color -force-copy -input=false -lock-timeout=0s -backend=true -get=true -upgrade=false -lock=true -get-plugins=true ->
             └─12952 /opt/consul-nia/terraform init -no-color -force-copy -input=false -lock-timeout=0s -backend=true -get=true -upgrade=false -lock=true -get-plugins=true ->
## ...

Additional best practice recommendations

  • Protect NIA Daemon API Endpoint - Any network endpoints provided by, or exposed to the Consul-Terraform-Sync daemon should be protected using appropriate firewall rules or configured to use Consul service mesh

  • Audit selected Terraform providers - The underlying Terraform providers that are configured with the Consul-Terraform-Sync daemon should be audited to ensure you are only using providers from sources that you trust.

  • Audit selected Terraform modules - The underlying Terraform modules that are configured with the Consul-Terraform-Sync daemon should be audited to ensure you are only using modules from sources that you trust.

  • Configure syslog - Configure Consul-Terraform-Sync to use syslog for logging.

  • Use a centralized logging solution - Export log entries within syslog generated from the Consul-Terraform-Sync daemon to a centralized logging solution.

Next steps

In this tutorial you learned how to secure your Consul-Terraform-Sync and prepare it to run in a production environment. You learned about the privileges required by the daemon to successfully interact with a fully secured Consul and best practices on how to avoid having sensitive data in the configuration.

To learn more about Network Infrastructure Automation with Consul-Terraform-Sync check the full documentation for it on the Consul website.

You can refer to Production Readiness Checklist to learn more on how to prepare your Consul datacenter for production.

If you need guidance on how to deploy your services to Consul service mesh, refer to our tutorials for getting started with service mesh on Kubernetes or VMs.

To learn even more about operating, observing, and monitoring your Consul service mesh, check out the following tutorials and collections.

Previous

Build a custom CTS module

 Next

CTS and A10 ADC