Secure Consul-Terraform-Sync for production
HashiCorp's Network Infrastructure Automation Integration Program allows network administrators to build integrations that automatically apply network and security infrastructure changes initiated by changes in the Consul service catalog.
Network Infrastructure Automation is carried out by Consul-Terraform-Sync, a multi-platform tool, that is able to connect to the Consul catalog to monitor changes in the services' state and health. The tool leverages Terraform as the underlying automation tool and utilizes the Terraform provider ecosystem to drive relevant change to your network infrastructure.
In this tutorial, you will learn the recommended practices to secure your Consul-Terraform-Sync instances and prepare them for a production environment including how to:
- Set up a dedicated instance
- Install Consul-Terraform-Sync
- Create a dedicated user with specific permissions
- Create a Consul ACL token
- Secure the Terraform provider
Prerequisites
This tutorial assumes you have an HashiCorp Cloud Platform (HCP) Consul cluster deployed in your HCP account, and that you have correctly configured an AWS peering connection between your HVN to your AWS VPC. The peering connection allows the Consul servers deployed in your HCP Consul cluster to communicate with clients that you deployed in your AWS VPC. Your Consul clients must have a valid L3 route configured that allows them to communicate with your HCP Consul cluster's private IP. They cannot use a public IP even if you have one configured.
Review the manual deployment tutorial or the automatic deployment tutorial for guidance on creating the necessary network configuration. To complete this tutorial, we expect that you have already provisioned the following resources.
- An HCP HashiCorp Virtual Network (HVN)
- An HCP Consul cluster
- An AWS VPC
- An AWS peering connection between your HVC and your VPC
- An AWS EC2 instance deployed to the peered VPC with a Consul client installed
If you do not yet have an EC2 instance provisioned in the target VPC, refer to the official AWS Documentation for guidance on how to create an EC2 instance that you can use with this tutorial. Afterwards, refer to the connect a Consul client to HCP Consul tutorial in order to configure a Consul client on your Consul-Terraform-Sync EC2 instance.
Note
In this tutorial, the AWS virtual machine (EC2) instance that you will run Consul-Terraform-Sync on must be deployed to the same VPC that you have peered with your HVN.
Use a dedicated instance
The Consul-Terraform-Sync daemon could have access to critical secrets for your environment's network infrastructure, depending on your configuration and use case. Using a hardened, dedicated host, for supporting these sensitive operations is highly recommended. Workload orchestrators, such as HashiCorp Nomad, also provide benefits of ensuring uptime and isolation.
While we recommend to run Consul-Terraform-Sync on the same node as a Consul agent, this is not a strict requirement. The daemon can run on any highly available node which has connectivity to the agents in Consul datacenter and the downstream network infrastructure.
For performance and availability reasons, you should not configure Consul-Terraform-Sync to connect directly to a server node nor to a node already running other services that are critical for your infrastructure.
We recommend that you run Consul-Terraform-Sync on a dedicated instance with a Consul client installed.
Install Consul-Terraform-Sync
You can install Consul-Terraform-Sync manually either by using a pre-compiled binary or by building it yourself from source.
In this tutorial, you will use an HCP Consul cluster. The HCP servers are running the Enterprise version of Consul. You can therefore use the Enterprise version of Consul-Terraform-Sync without the need for additional license. Learn more about the Enterprise functions of Consul-Terraform-Sync here.
You will install Consul-Terraform-Sync on an Ubuntu EC2 instance. For more information regarding installation on different distributions and platforms, please refer to the Consul-Terraform-Sync installation guide.
HashiCorp officially maintains and signs packages for the Ubuntu Linux distribution.
Install the following packages on your machine so you can install the CTS binary.
Add the HashiCorp GPG key.
Add the official HashiCorp Linux repository.
Update the list of available packages and their versions.
Find available enterprise versions to install.
Install a specific version.
Install latest version.
Once installed, verify the installation works by prompting the help option.
Setup user account and file permissions
The Consul-Terraform-Sync daemon does not require root or other administrative privileges to operate.
To ensure privilege isolation, we recommend to create a dedicated user and group, with limited permissions, to run Consul-Terraform-Sync. With a dedicated user and group you can appropriately scope directory and file permissions for your operating environment.
You will create a dedicated consul-nia system user, along with the supporting
directories and configuration file; then you will secure those permissions
using chown
and chmod
.
Create dedicated user and group.
Create data and configuration directory.
Setup appropriate permissions for the directories.
Configure ACL privileges for Consul-Terraform-Sync
Consul-Terraform-Sync needs to access different data in the Consul catalog and if you're using Consul as Terraform backend, you'll also need privileges to store the Terraform state into Consul's KV store.
In production environments, it is expected for Consul to be configured with Access Control Lists (ACLs) enabled. In order for Consul-Terraform-Sync to access information from Consul, a token needs to be passed to its configuration.
The two following sections review the privileges required by the token to accomplish the two different operations, catalog monitoring and Terraform state management.
Depending on the configuration choices you can select for one of the following options:
- combine the policies in one single token for the Consul-Terraform-Sync instance used for both the catalog monitoring and the Terraform backend
- create two different tokens for the same instance of Consul-Terrafomr-Sync and use one in the
consul
block of the Consul-Terraform-Sync configuration and one for the backend section - create one token for the Consul-Terraform-Sync instance to be monitored using rules mentioned in the Consul catalog privileges section below and one using the rules for the KV and session and use them for the two different parts of the configuration.
Consul CTS service privileges
CTS automatically registers itself as a service with Consul, which requires
write permissions for the CTS service. The name of the CTS service defaults to
Consul-Terraform-Sync
. See the
consul.service_registration
configuration for options to change this name or to disable this feature.
In the example below, the policy grants write permissions for the CTS service, using the default service name.
Consul catalog privileges
To retrieve information about services registered with Consul, Consul-Terraform-Sync needs access to the Consul catalog.
For this reason you will need a token with the right privileges to read all the selected services to be monitored, as well as the namespaces for the services in case you have them implemented in Consul Enterprise.
You must have a policy for your token before you can create a token. Click on the following tab(s) for policy examples with different levels of security.
Consul KV privileges
CTS requires read permissions to any Consul KV pairs that are monitored by tasks. This is only applicable if any of your tasks are configured with a Consul KV condition or Consul KV module input.
This is a basic example that provides read permissions for all Consul KV entries.
Consul KV privileges when used as the Terraform backend
When Terraform uses Consul as a backend, it will also need to have the necessary privileges to store the state inside the Consul KV store and to use sessions to ensure locking during the Terraform state changes.
By default Consul-Terraform-Sync will use the consul-terraform-sync
KV path to
store the Terraform state. However, you can configure the path in the
driver.terraform.backend
section of the configuration.
The example below provides privileges for all possible sessions names. Once the infrastructure is defined, it is possible to reduce the number of privileges to be limited to the name of the Consul node running Consul-Terraform-Sync.
Note
Consul is the default backend for Consul-Terraform-Sync. It is important to note that it does not support encryption. In a production environment we recommend you to use a Terraform backend that supports encryption.
Generate a Consul token for Consul-Terraform-Sync
Now you will create a policy and a token associated with it. Take the selected
policy content from the last few steps that apply to your environment and put
it in a file. The name of the file does not matter, but in this example it is
called cts-policy.hcl
. The name of the policy can also be anything - in this
example, the name is the same.
After you create the policy, generate a token associated to this policy. Take
note of your token's SecretID
for later. The value in the example output
below has been truncated.
Secure Terraform providers configuration
To effectively communicate with your infrastructure, the Terraform provider you use might require the setting of sensitive data such as tokens or authentication parameters, or to define environment variables that might change depending on the different task that is being executed. This information should be present in the Consul-Terraform-Sync configuration file in order to have your integration completely working.
To help you avoid the use of secrets or otherwise sensitive data in your
configuration file, the terraform_provider
block supports dynamically loading
arguments and environment variables from external sources.
Available sources for these information are environment variables, Consul KV, or Vault.
Using the task_env
meta-argument or the template syntax below, you can avoid
exposing sensitive values or credentials in plain text within configuration
files for Consul-Terraform-Sync.
Note
task_env
and the template syntax for dynamic values are only
supported within the terraform_provider
block.
Task environment setting
By default, Consul-Terraform-Sync enables all Terraform workspaces to inherit from its environment. This means that variables defined for the shell session used to run Consul-Terraform-Sync will be exposed to the providers too.
The task_env
block is a meta-argument available for the terraform_provider
block that can be used to rename or scope the available environment to a
selected set of variables. Passing sensitive values as environment variables
will scope the values to only the tasks that require the provider.
For example, imagine that you have a provider, named example
, that allows you to pass
a token using the environment variable PROVIDER_TOKEN
or using the field
token
in the configuration.
You can avoid having the token in plain text into the configuration using
task_env
in your Consul-Terraform-Sync configuration.
Refer to the official provider docs hosted on the Terraform Registry to find the supported environment variables for the provider you are using.
Load dynamic values
Terraform providers often require sensitive or dynamic data to be present in the configuration to successfully interact with your infrastructure providers. These variables might not be static, requiring you to change them before being able to successfully run Consul-Terraform-Sync daemon, or might contain secrets that you do not want to leave in plain text inside your configuration files.
As a best practice you should store those values elsewhere, and use
one of the following approaches to populate sensitive fields in your
terraform_provider
block.
Environment variables
You can use environment variables to populate the filed using env
.
Consul keys
You can query the Consul KV store, for the Consul datacenter configured in the
consul
block using key
.
Note
The ACL token used for the Consul datacenter will require read privileges to the path specified in this section to be able to retrieve the values from Consul.
Vault secrets
You can use a Vault instance and its KV secrets engine using with secret
.
Vault is an optional source that requires you to configure the Vault client
with a vault
block.
You can access the secret using template dot notation Data.data.<secret_key>
.
Warning
Consul-Terraform-Sync does not prevent sensitive values
from being written to Terraform state files. We recommend securing state files
in addition to securely configuring Terraform providers. Options for securing
state files can be set within driver.backend
based on the backend used. For
example, Consul KV is the default backend and can be secured with ACLs for KV
path. For other backends, we recommend enabling encryption, when applicable.
Configure Consul-Terraform-Sync process
We recommend running Consul-Terraform-Sync on a dedicated machine. In addition, you should configure CTS to launch at startup, so it can survive machine reboots.
Tip
We recommend to create a user for running Consul-Terraform-Sync.
Earlier in this tutorial, you created the consul-nia
user. Alternatively, if
you completed the Deployment Guide, you
would have created a consul
user account in your system. You can reuse the
consul
user account for Consul-Terraform-Sync.
Configure systemd
Systemd uses documented reasonable defaults so only non-default values must be set in the configuration file.
Create a Consul service file at /usr/lib/systemd/system/consul-terraform-sync.service
.
Add this configuration to the Consul-Terraform-Sync service file.
Warning
If you are using Consul-Terraform-Sync version older than
v0.6.0, the start
parameter is not supported when starting in daemon mode.
Remove it from line 12 of the SystemD unit file below.
1 2 3 4 5 6 7 8 9 1011121314151617181920
The service file configures the system to run Consul-Terraform-Sync using the
ExecStart
command and using a dedicated
User
and Group
.
For more information on the available systemd.service
settings, refer to the
systemd
documentation.
Create configuration and environment files
In the service file, you configured Consul-Terraform-Sync to use /etc/consul-nia.d/
as configuration directory.
Specifically the service will need two files:
/etc/consul-nia.d/config.hcl
- The main configuration file. At startup, the service will verify that the file exists and that it's not empty./etc/consul-nia.d/consul-nia.env
- The environment file used to setup environment variables for the Consul-Terraform-Sync process. The file needs to exist in order for the service to start, but it can be empty.
Earlier you created the configuration folder /etc/consul-nia.d
. Now, create the configuration files.
Use the following as a template for writing a configuration that fits your
environment. Refer to
Consul-Terraform-Sync documentation
for a list of all available parameters. In the consul
block, set token
to
the SecretID
value of the token you created earlier in the
Generate a Consul token for Consul-Terraform-Sync step.
Create the file to set up the environment variables for the CTS process.
Then grant the consul-nia
user permissions to access the file.
Set up the Enterprise license for Consul-Terraform-Sync (optional)
Note
If you are using Consul-Terraform-Sync enterprise version lower than 0.6.0, you will need to manually set up the Enterprise license. However, Consul-Terraform-Sync enterprise version 0.6.0 and higher support automatic retrieval of the Consul license from your own Consul enterprise cluster.
In this tutorial, you are using an HCP Consul cluster that already comes with an Enterprise license that you can also use with Consul-Terraform-Sync.
Set up an automatic license retrieval configuration for Consul-Terraform-Sync.
123456
Start the Consul-Terraform-Sync process
Enable and start Consul-Terraform-Sync using the systemctl command responsible for controlling systemd managed services. Check the status of the Consul-Terraform-Sync service using systemctl.
First, make sure the SystemD reloads its unit file configurations.
Then, enable the CTS system process.
Then, start the CTS system process.
Finally, check the status for the process. The output should indicate the service is running.
Additional best practice recommendations
Protect NIA Daemon API Endpoint - Any network endpoints provided by, or exposed to the Consul-Terraform-Sync daemon should be protected using appropriate firewall rules or configured to use Consul service mesh
Audit selected Terraform providers - The underlying Terraform providers that are configured with the Consul-Terraform-Sync daemon should be audited to ensure you are only using providers from sources that you trust.
Audit selected Terraform modules - The underlying Terraform modules that are configured with the Consul-Terraform-Sync daemon should be audited to ensure you are only using modules from sources that you trust.
Configure syslog - Configure Consul-Terraform-Sync to use syslog for logging.
Use a centralized logging solution - Export log entries within syslog generated from the Consul-Terraform-Sync daemon to a centralized logging solution.
Next steps
In this tutorial you learned how to secure your Consul-Terraform-Sync and prepare it to run in a production environment. You learned about the privileges required by the daemon to successfully interact with a fully secured Consul and best practices on how to avoid having sensitive data in the configuration.
To learn more about Network Infrastructure Automation with Consul-Terraform-Sync check the full documentation for it on the Consul website.
You can refer to Production Readiness Checklist to learn more on how to prepare your Consul datacenter for production.
If you need guidance on how to deploy your services to Consul service mesh, refer to our tutorials for getting started with service mesh on Kubernetes or VMs.
To learn even more about operating, observing, and monitoring your Consul service mesh, check out the following tutorials and collections.