HashiConf Our community conference is taking place in San Francisco and online October 10-12. Register now
Consul
Associate Tutorials

Network Infrastructure Automation with Consul-Terraform-Sync Intro

  • 17min

  • Terraform
  • Consul

HashiCorp's Network Infrastructure Automation Integration Program allows your network administrators to build integrations that automatically apply network and security infrastructure changes reacting to changes in the Consul service catalog.

Network Infrastructure Automation is carried out by Consul-Terraform-Sync, a multi-platform tool, that is able to connect to Consul catalog and monitor changes in the services' state and health. The tool leverages Terraform as the underlying automation tool, and utilizes the Terraform provider ecosystem to drive relevant change to your network infrastructure.

Consul-Terraform-Sync can be configured to execute one or more automation tasks that use variables based on the content of the Consul service catalog. Each task consists of a runbook automation written as a compatible Terraform module using resources and data sources for the underlying network infrastructure.

NIA CTS architecture diagram

In this tutorial, you will learn how to configure Consul-Terraform-Sync to communicate with your Consul datacenter, react to service changes, and execute an example task.

Prerequisites

This tutorial features two learning paths:

  1. HashiCorp Cloud Platform (HCP) Consul deployment, AWS Elastic Kubernetes Service (EKS), and AWS EC2 instance
  2. Self-managed Kubernetes Consul deployment on AWS Elastic Kubernetes Service (EKS), and AWS EC2 instance

Select your learning path by clicking one of the following tabs.

Clone GitHub repository

Clone the GitHub repository containing the configuration files and resources.

$ git clone https://github.com/hashicorp-education/learn-consul-cts-intro.git

$ git clone git@github.com:hashicorp-education/learn-consul-cts-intro.git

Using the following GitHub repository, you will provision the following resources:

  • An HCP HashiCorp Virtual Network (HVN)
  • An HCP Consul server Cluster
  • An AWS VPC
  • An AWS Elastic Kubernetes Service (EKS)
  • An AWS key pair
  • An AWS EC2 instance with a Consul client and CTS installed

Change into the directory that contains the complete configuration files for this tutorial.

$ cd learn-consul-cts-intro/hcp

Deploy your infrastructure

The sub-directories inside are responsible for the following:

Deploy your HCP, VPC and EKS

You will now deploy the basic infrastructure that consists of an HCP cluster, a VPC and an EKS cluster. The following command downloads the necessary providers and initializes the backend.

$ terraform -chdir=infrastructure/ init

Initializing the backend...

Initializing provider plugins...
##...

Terraform has been successfully initialized!
##...

Deploy the resources using the apply command. Confirm the run by entering yes.

$ terraform -chdir=infrastructure/ apply

##...

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

Once you confirm, it will take a few minutes to complete the deploy. If the deploy was successful you should get the following output.

Apply complete! Resources: 68 added, 0 changed, 0 destroyed.

Outputs:

##...

vpc_cluster_id = "learn-hcp-consul-ec2-client-af5y"
region = "eu-central-1"
hcp_consul_cluster_id = "learn-hcp-consul-ec2-client-af5y"
hcp_consul_security_group = "sg-0eb88e0427b2a759e"
vpc_public_subnets = "subnet-044a7a789bad6ee2e"
vpc_cidr_block = "10.0.0.0/16"
vpc_id = "vpc-0677cdc5f5f9a7d7c"

You basic infrastructure is now deployed, which includes your AWS VPC, AWS EKS, HCP HVN, as well as your HCP Consul cluster. Next, configure the kubectl tool to communicate with your EKS Kubernetes cluster.

$ aws eks --region $(terraform -chdir=infrastructure/ output -raw region) update-kubeconfig --name $(terraform -chdir=infrastructure/ output -raw cluster_id)

Prepare configuration to deploy Consul

Terraform has automatically retrieved the client configuration information from HCP that you need to connect your EKS cluster client agents to your Consul cluster. These are two files in your infrastructure directory - a default client configuration and a certificate. Both should be considered secrets, and in production environments should be kept in a secure location.

Use ls to confirm that both the client_config.json and ca.pem files are available.

$ ls infrastructure/client_config.json infrastructure/ca.pem
infrastructure/ca.pem           infrastructure/client_config.json

Next, export the Consul root token into a shell variable.

$ export CONSUL_HTTP_TOKEN="$(terraform -chdir=infrastructure/ output -raw hcp_consul_root_token)"

Create a consul namespace in your Kubernetes cluster. Your Consul secrets and resources will be created in this namespace.

$ kubectl create namespace consul
namespace/consul created

Consul Service on HCP is secure by default. This means that you will need to configure client agents with the gossip encryption key, the Consul CA cert, and a root ACL token. You need to store these secrets in the Kubernetes secrets engine so that the Helm chart can reference and retrieve them during installation.

  1. Use the ca.pem file in the current directory to create a Kubernetes secret to store the Consul CA certificate.

    $ kubectl create secret generic "consul-ca-cert" --from-file='tls.crt=./infrastructure/ca.pem' --namespace consul
secret/consul-ca-cert created

  2. The Consul gossip encryption key is embedded in the client_config.json file that you downloaded and extracted into your current directory. Issue the following command to create a Kubernetes secret that stores the Consul gossip key encryption key. The following command uses jq to extract the value from the client_config.json file.

    $ kubectl create secret generic "consul-gossip-key" --from-literal="key=$(jq -r .encrypt infrastructure/client_config.json)" --namespace consul
secret/consul-gossip-key created

  3. The last secret you need to add is an ACL bootstrap token. You can use the one you set to your CONSUL_HTTP_TOKEN environment variable earlier. Issue the following command to create a Kubernetes secret to store the bootstrap ACL token.

    $ kubectl create secret generic "consul-bootstrap-token" --from-literal="token=${CONSUL_HTTP_TOKEN}" --namespace consul
secret/consul-bootstrap-token created

Note

If you are configuring a production environment, you should create a client token with a minimum set of privileges. For an in depth review of how to configure ACLs for Consul, refer to the Secure Consul with Access Control Lists tutorial or the official documentation.

Next, extract some more configuration values from the client_config.json file and set them to environment variables that can be used to generate your Helm values file. Issue the following command to set the DATACENTER environment variable.

$ export DATACENTER=$(jq -r .datacenter infrastructure/client_config.json)

Extract the private server URL from the client config so that it can be set in the Helm values file as the externalServers:hosts entry. This value will be passed as the retry_join option to the Consul clients.

$ export RETRY_JOIN=$(jq -r --compact-output .retry_join infrastructure/client_config.json)

Extract the public server URL from the client config so that it can be set the Helm values file as the k8sAuthMethodHost entry.

Note

The following script relies on your cluster matching your current-context name. If you have created an alias for your context, or the current-context name does not match the cluster name for any other reason, you must manually set the KUBE_API_URL to the API server URL of your EKS cluster. You can use kubectl config view to view your cluster, and retrieve API server URL.

$ export KUBE_API_URL=$(kubectl config view -o jsonpath="{.clusters[?(@.name == \"$(kubectl config current-context)\")].cluster.server}")

Validate that all of your environment variables have been set.

$ echo $DATACENTER && \
  echo $RETRY_JOIN && \
  echo $KUBE_API_URL

Example output:

learn-hcp-consul-ec2-client-876w
["learn-hcp-consul-ec2-client-876w.private.consul.223350f5-f0b9-4d89-959d-aa8eab3184e5.aws.hashicorp.cloud"]
https://0EAF417B60845332241616950BF462A6.gr7.eu-central-1.eks.amazonaws.com

Note

If any of these environment variables are not correctly set, the following script will generate an incomplete Helm values file, and the Consul Helm installation will not succeed.

Note

The value for the global.name configuration must be unique for each Kubernetes cluster where Consul clients are installed and configured to join Consul as a shared service, such as HCP Consul. You can change the global name through the global.name value in the Helm chart.

Generate the Consul Helm chart with the following command.

1 2 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031323334353637$ cat > helm/consul-values.yaml << EOF
global:
  name: consul
  image: hashicorp/consul-enterprise:1.12.8-ent
  enabled: false
  datacenter: ${DATACENTER}
  acls:
    manageSystemACLs: true
    bootstrapToken:
      secretName: consul-bootstrap-token
      secretKey: token
  gossipEncryption:
    secretName: consul-gossip-key
    secretKey: key
  tls:
    enabled: true
    enableAutoEncrypt: true
    caCert:
      secretName: consul-ca-cert
      secretKey: tls.crt
  enableConsulNamespaces: true
externalServers:
  enabled: true
  hosts: ["${RETRY_JOIN}"]
  httpsPort: 443
  useSystemRoots: true
  k8sAuthMethodHost: ${KUBE_API_URL}
client:
  enabled: true
  join: ${RETRY_JOIN}
connectInject:
  enabled: true
controller:
  enabled: true
dns:
  enabled: true
EOF

Last, validate that the config file is populated correctly.

$ more helm/consul-values.yaml

Deploy Consul in your EKS cluster

Make sure that the consul-k8s tool is the correct version for this deployment - v0.44.0.

$ consul-k8s version
    consul-k8s v0.44.0

Deploy Consul to your AWS EKS platform using the consul-k8s tool. Confirm the run by entering y.

$ consul-k8s install -config-file helm/consul-values.yaml

==> Checking if Consul can be installed
 ✓ No existing Consul installations found.
 ✓ No existing Consul persistent volume claims found
 ✓ No existing Consul secrets found.

==> Consul Installation Summary
    Name: consul
    Namespace: consul
    
    Helm value overrides
    -------------------
    ##...

    Proceed with installation? (y/N)

Once you confirm, it will take a few minutes to complete the deploy. If the deploy was successful you should get a similar output.

==> Installing Consul
 ✓ Downloaded charts
 --> creating 1 resource(s)
 --> creating 45 resource(s)
 --> beginning wait for 45 resources with timeout of 10m0s
 ✓ Consul installed in namespace "consul".

Verify that Consul is running by inspecting the output of the consul members command.

$ kubectl exec -ti $(kubectl get pods --namespace consul | grep -Eo -m1 'consul-client-(\w+)') --namespace consul  -- /bin/consul members
Node                                        Address         Status  Type    Build   Protocol  DC   Partition  Segment
consul-server-0                             10.0.1.73:9301  alive   server  1.12.8  2         dc1  default    <all>
ip-10-0-1-6.eu-central-1.compute.internal   10.0.1.6:8301   alive   client  1.12.8  2         dc1  default    <default>
ip-10-0-1-73.eu-central-1.compute.internal  10.0.1.73:8301  alive   client  1.12.8  2         dc1  default    <default>
ip-10-0-2-71.eu-central-1.compute.internal  10.0.2.71:8301  alive   client  1.12.8  2         dc1  default    <default>

Next, you will update your existing coredns ConfigMap in the kube-system namespace to include a forward definition for Consul that points to the cluster IP of the Consul DNS service. The following command adds the correct section to the K8s configuration.

$ kubectl -n kube-system get configmap coredns -o YAML | sed "s/^    }/    }\\n    consul {\\n      errors\\n      cache 30\\n      forward . $(kubectl -n consul get svc consul-dns --output jsonpath='{.spec.clusterIP}')\\n    }/" | kubectl apply -f -
configmap/coredns configured

Trigger a restart of the CoreDNS deployment to apply your changes.

$ kubectl rollout restart -n kube-system deployment/coredns

Verify that DNS forwarding works by resolving the consul.service.consul domain.

$ kubectl exec -ti $(kubectl get pods --namespace consul | grep -Eo -m1 'consul-client-(\w+)') --namespace consul -- /usr/bin/nslookup consul.service.consul
Defaulted container "consul" out of: consul, client-acl-init (init)
Server:         172.20.0.10
Address:        172.20.0.10:53


Name:   consul.service.consul
Address: 10.0.1.73

Deploy the EC2 instance for Consul-Terraform-Sync

Deploy an EC2 instance that will run the Consul-Terraform-Sync service.

First, use the Terraform output values from the infrastructure deployment as input variables for the ec2-instance-cts deployment. The following command does that for you automatically.

cat <<EOF >ec2-instance-cts/terraform.tfvars 
aws_region="$(terraform -chdir=infrastructure/ output -raw region)"
vpc_id="$(terraform -chdir=infrastructure/ output -raw vpc_id)"
vpc_cidr_block="$(terraform -chdir=infrastructure/ output -raw vpc_cidr_block)"
subnet_id="$(terraform -chdir=infrastructure/ output -raw vpc_public_subnets)"
cluster_id="$(terraform -chdir=infrastructure/ output -raw cluster_id)"
hcp_consul_security_group_id="$(terraform -chdir=infrastructure/ output -raw hcp_consul_security_group_id)"
hcp_consul_root_token="$(terraform -chdir=infrastructure/ output -raw hcp_consul_root_token)"
EOF

You are now ready to deploy the EC2 instance running Consul and Consul-terraform-sync. Initialize Terraform, and then apply the Terraform configuration.

$ terraform -chdir=ec2-instance-cts/ init

Initializing the backend...

Initializing provider plugins...
...

Terraform has been successfully initialized!
...

Deploy the resources using the apply command. Confirm the run by entering yes.

$ terraform -chdir=ec2-instance-cts/ apply

...
Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value:

Once you confirm, it will take a few minutes to complete the deploy. If the deploy was successful you should get the following output.

Apply complete! Resources: 5 added, 0 changed, 0 destroyed.

Outputs:

consul_root_token = <sensitive>
ec2_client = "3.71.107.133"

To further interact with the Consul cluster, you will need an access token. Get the root token and make note of it for later.

$ terraform -chdir=ec2-instance-cts/ output -raw consul_root_token
00000000-0000-0000-0000-000000000000

Note

This tutorial uses the Consul root token which is not recommended for production use. Examine the Secure Consul with Access Control Lists (ACLs) tutorial to learn how to correctly create a token just for the Consul client agent on the EC2 instance.

You EC2 instance is now deployed. The Consul client agent is currently being installed by the provisioning scripts. After a couple of minutes make sure that the Consul service has started by running the consul members command.

$ ssh ubuntu@$(terraform -chdir=ec2-instance-cts output -raw ec2_client) -i ./ec2-instance-cts/consul-client.pem "consul members"
Node                                         Address             Status  Type    Build       Protocol  DC                                Partition  Segment
ip-172-25-42-185                             172.25.42.185:8301  alive   server  1.12.8+ent  2         learn-hcp-consul-ec2-client-mmew  default    <all>
ip-10-0-1-103                                10.0.1.103:8301     alive   client  1.12.8+ent  2         learn-hcp-consul-ec2-client-mmew  default    <default>
ip-10-0-4-103.eu-central-1.compute.internal  10.0.4.143:8301     alive   client  1.12.8      2         learn-hcp-consul-ec2-client-mmew  default    <default>
ip-10-0-4-201.eu-central-1.compute.internal  10.0.4.205:8301     alive   client  1.12.8      2         learn-hcp-consul-ec2-client-mmew  default    <default>
ip-10-0-6-220.eu-central-1.compute.internal  10.0.6.32:8301      alive   client  1.12.8      2         learn-hcp-consul-ec2-client-mmew  default    <default>

Deploy HashiCups to Kubernetes

Deploy the example app HashiCups to your K8s cluster.

$ kubectl apply -f hashicups-v1.0.2/
service/frontend created
serviceaccount/frontend created
servicedefaults.consul.hashicorp.com/frontend created
##...

Forward a port to the K8s cluster to verify that HashiCups has been deployed correctly. 

$ kubectl port-forward deploy/nginx 8082:80

Forwarding from 127.0.0.1:8082 -> 80
Forwarding from [::1]:8082 -> 80

Open the HashiCups app by pointing your browser to http://localhost:8082. You should find the HashiCups welcome page.

HashiCups web application in a browser

Next, stop the K8s port forwarding by entering CTRL+C to return to your terminal session.

Finally, make sure that the Consul catalog contains the HashiCups services and their sidecar proxies.

$ ssh ubuntu@$(terraform -chdir=ec2-instance-cts output -raw ec2_client) -i ./ec2-instance-cts/consul-client.pem "consul catalog services"
consul
frontend
frontend-sidecar-proxy
nginx
nginx-sidecar-proxy
payments
payments-sidecar-proxy
postgres
postgres-sidecar-proxy
products-api
products-api-sidecar-proxy
public-api
public-api-sidecar-proxy

Configure Consul-Terraform-Sync

The Consul-Terraform-Sync daemon is configured using configuration files and supports HashiCorp Configuration Language (HCL) and JSON file formats.

A configuration file for Consul-Terraform-Sync is composed by several blocks, this section will guide you through the different blocks and provide you with example values. For the full list of available options check the documentation.

Access the EC2 instance that you previously deployed by running the following command.

$ ssh ubuntu@$(terraform -chdir=ec2-instance-cts output -raw ec2_client) -i ./ec2-instance-cts/consul-client.pem

Create a file named cts-config.hcl.

$ touch cts-config.hcl

Paste the following content into the cts-config.hcl file. Make sure to edit the token on line 15 to match with your root token.

cts-config.hcl
The content of the Terraform configuration file
log_level   = "INFO"
working_dir = "sync-tasks"
port        = 8558

syslog {}

buffer_period {
  enabled = true
  min     = "5s"
  max     = "20s"
}

consul {
  address = "localhost:8500"
  token = "00000000-0000-0000-0000-000000000000"
}

driver "terraform" {
  # version = "0.14.0"
  # path = ""
  log         = false
  persist_log = false

  backend "consul" {
    gzip = true
  }
}

task {
 name        = "learn-cts-example"
 description = "Example task with two services"
 module      = "findkim/print/cts"
 version     = "0.1.0"
 condition "services" {
  names = ["frontend", "public-api"]
 }
}

Global configs

Top level options are reserved for configuring the Consul-Terraform-Sync daemon.

log_level   = "INFO"
working_dir = "sync-tasks"
port        = 8558

syslog {}

buffer_period {
  enabled = true
  min     = "5s"
  max     = "20s"
}

In this section you can configure the log level to use for Consul-Terraform-Sync logging as well as the port used by the demon to offer the API interface.

Other notable sections are:

  • The buffer_period section configures the default buffer period for all tasks to mitigate the affects of flapping services to downstream network devices.

  • The syslog section specifies the syslog server for logging. This section can be useful when Consul-Terraform-Sync is configured as a daemon, for example in Linux using systemd.

Consul block

The consul block is used to configure Consul-Terraform-Sync connection with a Consul agent to perform queries to the Consul catalog and Consul KV pertaining to task execution.

You can use this block to configure connection parameters, such as the Consul address, and security parameters, like TLS certificates or ACL token, to secure the connection with Consul and make sure you provide the Consul-Terraform-Sync daemon with the right privileges to perform the required operations.

Configure CTS to communicate to your HCP Consul cluster using the Consul root token that you retrieved earlier.

consul {
  address = "localhost:8500"
  token = "00000000-0000-0000-0000-000000000000"
}

Note

In a fully secured environment, with mTLS and ACLs enabled, you can use this section to include the certificates, in addition to the token required by Consul-Terraform-Sync to securely communicate with Consul. You can also specify the token using environment variables, such as CONSUL_HTTP_TOKEN, to avoid having sensitive data in your configuration file. Furthermore, the token used here should have only the minumum level of access needed. Refer to the Secure Consul-Terraform-Sync for Production guide for more information on how to do that.

Performance considerations

As shown in the architectural diagram above it is recommended to run Consul-Terraform-Sync on a dedicated node running a Consul agent. This permits to have dedicated resources for network automation and to fine tune security and privilege separation between the network administrators and the other Consul agents.

Driver “terraform” block

The driver block configures the subprocess used by Consul-Terraform-Sync to propagate infrastructure change. The Terraform driver is a required configuration for Consul-Terraform-Sync to relay provider discovery and installation information to Terraform.

driver "terraform" {
  # version = "0.14.0"
  # path = ""
  log         = false
  persist_log = false

  backend "consul" {
    gzip = true
  }
}

Using this section you can define the version of Terraform you want to use, in case you want to define strict requirements for it, the path where to find/install the Terraform binary or if you want the logs for Terraform to be included in the Consul-Terraform-Sync logs or persisted on disk. Finally, you can also define the working directory for Terraform to operate in.

Terraform state

By default, Consul-Terraform-Sync uses Consul to store Terraform state files. If no options are specified the same Consul instance configured in the consul block is used. In case you want to use a different backend for Terraform or to specify a different Consul datacenter as backend, you can use the backend section of the configuration to define it. All standard backends supported by Terraform are supported by Consul-Terraform-Sync. Check Terraform backend documentation to learn about available options.

Deprecated

The driver.terraform.working_dir option is marked for deprecation in Consul-Terraform-Sync 0.3.0. Use the global working_dir option instead.

Task block

A task is executed when any change of information for services the task is configured for is detected from the Consul catalog. Execution could include one or more changes to service values, like IP address, added or removed service instance, or tags.

task {
 name        = "learn-cts-example"
 description = "Example task with two services"
 module      = "findkim/print/cts"
 version     = "0.1.0"
 condition "services" {
  names = ["frontend", "public-api"]
 }
}

You can check the full list of values that would cause a task to run in the Task Execution documentation.

Consul-Terraform-Sync will attempt to execute each task once upon startup to synchronize infrastructure with the current state of Consul. The daemon will stop and exit if any error occurs while preparing the automation environment or executing a task for the first time.

A task block configures the task to run as automation for the defined services. The services can be explicitly defined in the task's condition block.

You can specify multiple task blocks in case you need to configure multiple tasks.

For example, in the code snippet above, you are defining a task block that will monitor and react to changes on frontend and public-api services and run the module defined in the module parameter when any change is detected.

For this tutorial, you will be using the Consul Print Module to configure Consul-Terraform-Sync. This Terraform module creates text files on your local machine containing Consul service information. It is a useful module to use to familiarize yourself with Consul Terraform Sync, without requiring deployed infrastructure and credentials.

Start Consul-Terraform-Sync

Once the configuration file is created it is possible to start Consul-Terraform-Sync using the consul-terraform-sync binary. The binary requires the configuration to be passed either using the -config-file or -config-dir.

Consul-Terraform-Sync provides different running modes, including some that can be useful to safely test your configuration and the changes that are going to be applied.

The default mode is named daemon mode, in this mode Consul-Terraform-Sync passes through a once-mode phase, in which it will try to run all the tasks once, and then turns into a long running process. During the once-mode phase, the daemon will exit with a non-zero status if it encounters an error. After successfully passing through once-mode phase, errors will be logged and the process is not expected to exit.

You may also start Consul-Terraform-Sync as a systemd process. To learn how to configure Consul-Terraform-Sync as a systemd process, checkout the Secure Consul-Terraform-Sync for Production tutorial.

Warning

If you are using Consul-Terraform-Sync version older than v0.6.0, the start parameter is not supported and must be removed.

$ consul-terraform-sync start -config-file cts-config.hcl
[INFO]  cli: v0.6.0 (7f0d7ba)
[INFO]  cli: running controller in daemon mode
[INFO]  cli: setting up controller: type=readwrite
[INFO]  ctrl: initializing Consul client and testing connection
[INFO]  driver.terraform: install terraform: install_path=/
[INFO]  driver.terraform: successfully installed terraform
[INFO]  cli: initializing controller
[INFO]  ctrl: initializing driver
[INFO]  ctrl: initializing all tasks
[INFO]  client.terraformcli: Terraform output is muted
[INFO]  driver.terraform: retrieved 0 Terraform handlers for task: task_name=learn-cts-example
[INFO]  ctrl: drivers initialized
[INFO]  ctrl: executing all tasks once through
[INFO]  ctrl: executing task: task_name=learn-cts-example
[INFO]  ctrl: task completed: task_name=learn-cts-example
[INFO]  ctrl: all tasks completed once
[INFO]  api: starting server: port=8558
...

When running in daemon mode Consul-Terraform-Sync will keep running in the foreground and prevent you from performing other operations in the same terminal. Also it will get terminated in case the terminal session is closed. Consider configuring it as a service and to run it at system startup to be able to survive machine reboots.

Review files created by Consul-Terraform-Sync daemon

After startup, Consul-Terraform-Sync will run Terraform inside the working_directory defined in the configuration. If no directory is defined, it creates a folder, named sync-tasks, in the directory from which the binary is started.

Inside that folder, Terraform will create a workspace for each task defined in the configuration.

$ tree sync-tasks/
sync-tasks/
└── learn-cts-example
    ├── main.tf
    ├── variables.tf
    ├── terraform.tfvars
    ├── terraform.tfvars.tmpl
    ├── frontend.txt
    ├── public-api.txt
    └── providers.auto.tfvars

1 directory, 7 files

Here are the files of particular interest in this folder:

  • The main.tf file contains the Terraform block, provider blocks, and a module block calling the module configured for the task.
  • The variables.tf file contains the services input variable which determines module compatibility with Consul-Terraform Sync and optionally the intermediate variables used to dynamically configure providers.
  • The terraform.tfvars file is where the services input variable is assigned values from the Consul catalog. It is periodically updated to reflect the current state of the configured set of services for the task.
  • The terraform.tfvars.tmpl file is used to template the information retrieved from Consul catalog into the terraform.tfvars file.

For example, running Consul-Terraform-Sync against the current deployment will produce the following terraform.tfvars:

terraform.tfvars
# This file is generated by Consul Terraform Sync.
#
# The HCL blocks, arguments, variables, and values are derived from the
# operator configuration for Sync. Any manual changes to this file
# may not be preserved and could be overwritten by a subsequent update.
#
# Task: learn-cts-example
# Description: Example task with two services

services = {
  "frontend-68978f9797-wvdhj-frontend.ip-10-0-2-110.eu-central-1.compute.internal.dc1" = {
    id      = "frontend-68978f9797-wvdhj-frontend"
    name    = "frontend"
    kind    = ""
    address = "10.0.2.101"
    port    = 3000
    meta = {
      k8s-namespace    = "default"
      k8s-service-name = "frontend"
      managed-by       = "consul-k8s-endpoints-controller"
      pod-name         = "frontend-68978f9797-wvdhj"
    }
    tags            = []
    namespace       = ""
    status          = "passing"
    node            = "ip-10-0-2-110.eu-central-1.compute.internal"
    node_id         = "8d3af2cc-ffc7-10e1-dc14-f16d3a9ae61d"
    node_address    = "10.0.2.110"
    node_datacenter = "dc1"
    node_tagged_addresses = {
      lan      = "10.0.2.110"
      lan_ipv4 = "10.0.2.110"
      wan      = "10.0.2.110"
      wan_ipv4 = "10.0.2.110"
    }
    node_meta = {
      consul-network-segment = ""
      host-ip                = "10.0.2.110"
      pod-name               = "consul-client-qvctc"
    }
    cts_user_defined_meta = {}
  },
  "public-api-778b9c8447-x4lcc-public-api.ip-10-0-1-101.eu-central-1.compute.internal.dc1" = {
    id      = "public-api-778b9c8447-x4lcc-public-api"
    name    = "public-api"
    kind    = ""
    address = "10.0.1.161"
    port    = 8080
    meta = {
      k8s-namespace    = "default"
      k8s-service-name = "public-api"
      managed-by       = "consul-k8s-endpoints-controller"
      pod-name         = "public-api-778b9c8447-x4lcc"
    }
    tags            = []
    namespace       = ""
    status          = "passing"
    node            = "ip-10-0-1-101.eu-central-1.compute.internal"
    node_id         = "2c4a7f11-06b7-a8d9-c5f3-0cbc75f62909"
    node_address    = "10.0.1.101"
    node_datacenter = "dc1"
    node_tagged_addresses = {
      lan      = "10.0.1.101"
      lan_ipv4 = "10.0.1.101"
      wan      = "10.0.1.101"
      wan_ipv4 = "10.0.1.101"
    }
    node_meta = {
      consul-network-segment = ""
      host-ip                = "10.0.1.101"
      pod-name               = "consul-client-jbr72"
    }
    cts_user_defined_meta = {}
  },
}

Note

Generated template and Terraform configuration files are crucial to the automation of tasks. Any manual changes to the files may not be preserved and could be overwritten by a subsequent update.

Review automation results

In this example the module prints the addresses of the matched services into one file for each service matched. You will find two of those files - frontend.txt and public-api.txt in the task workspace folder learn-cts-example.

In a real life scenario your module will react on service changes to trigger Terraform runs and deploy changes to your network infrastructure.

Scale-up the deployments of both the frontend and public-api services and observe the changes that CTS does to the related text files. First inspect the current content of both files.

$ for x in sync-tasks/learn-cts-example/*.txt; do echo "$x: "; cat $x; echo; done
sync-tasks/learn-cts-example/frontend.txt:
10.0.2.110
sync-tasks/learn-cts-example/public-api.txt:
10.0.1.101

Keep the current session to the EC2 instance open (CTS is running in the background), and open a new terminal session on your own machine. Then scale-up the K8s deployment of the frontend service to two instances.

$ kubectl scale deployment frontend --replicas=2
deployment.apps/frontend scaled

Return to your EC2 instance session and you will find that while CTS was running in the background, it reacted to the scale-up of the frontend deployment.

2022-07-29T11:36:25.557Z [INFO]  tasksmanager: executing task: task_name=learn-cts-example
2022-07-29T11:36:26.141Z [INFO]  tasksmanager: task completed: task_name=learn-cts-example
2022-07-29T11:36:47.855Z [INFO]  tasksmanager: executing task: task_name=learn-cts-example
2022-07-29T11:36:48.592Z [INFO]  tasksmanager: task completed: task_name=learn-cts-example

Press enter to clear your terminal and then check out the content of the text files for changes.

$ for x in sync-tasks/learn-cts-example/*.txt; do echo "$x: "; cat $x; echo; done
sync-tasks/learn-cts-example/frontend.txt:
10.0.1.100
10.0.2.110
sync-tasks/learn-cts-example/public-api.txt:
10.0.1.101

By scaling-up the frontend deployment, CTS has reacted to the change and refreshed the content of the text file to include the IP address of the second instance of the service. Meanwhile, the public-api service is unchanged.

Clean-up

Exit the session to your EC2 instance.

Destroy resources

Use the terraform destroy command to clean up the resources you created. First, destroy the CTS EC2 instance.

$ terraform -chdir=ec2-instance-cts/ destroy

...
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value:

Remember to confirm by entering yes.

Once you confirm, it will take a few minutes to complete the removal. If the command was successful you should get the following output.

Destroy complete! Resources: 5 destroyed.

Next, uninstall Consul with the help of the consul-k8s tool. Confirm the uninstall action and then the deletion of Consul data when prompted.

$ consul-k8s uninstall

==> Checking if Consul demo application can be uninstalled
    No existing Consul demo application installation found.

==> Checking if Consul can be uninstalled
 ✓ Existing Consul installation found.

==> Consul Uninstall Summary
    Name: consul
    Namespace: consul
    Proceed with uninstall? (y/N) y

##...

 ✓ Successfully uninstalled Consul Helm release.

==> Other Consul Resources
 * WARNING: Proceed with deleting PVCs, Secrets, Service Accounts, Roles, Role Bindings, Jobs, Cluster Roles, and Cluster Role Bindings for the following installation?

   Name: consul
   Namespace: consul

   Only approve if all data from this installation can be deleted. (y/N) y
 ✓ Deleted PVC => data-consul-consul-server-0
 ✓ PVCs deleted.
 ✓ Deleted Secret => consul-bootstrap-acl-token
 ✓ Consul secrets deleted.
 ✓ No Consul service accounts found.
 ✓ No Consul roles found.
 ✓ No Consul rolebindings found.
 ✓ No Consul jobs found.
 ✓ No Consul cluster roles found.
 ✓ No Consul cluster role bindings found.

Next, destroy the AWS VPC and the EKS cluster.

$ terraform -chdir=infrastructure destroy

...
Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value:

Remember to confirm by entering yes.

Once you confirm, it will take a few more minutes to complete the removal. Once the remove is successful, you should get the following output.

Destroy complete! Resources: 53 destroyed.

Next steps

In this tutorial you learned the basics of Network Infrastructure Automation with Consul-Terraform-Sync. You got an architectural overview of all the logical blocks involved in the process and what are the advantages of automating Day-2 operations for network administrative tasks. Finally, you tried first hand the Consul-Terraform-Sync binary with an example module that retrieved information from Consul service catalog and printed it on a text file in your working directory.

Specifically you:

  • Installed Consul-Terraform-Sync
  • Created a working configuration file for the binary
  • Started Consul-Terraform-Sync
  • Checked the files and data pulled from Consul-Terraform-Sync into the Terraform workspace

In the next tutorial you will learn the different run modes for Consul-Terraform-Sync and how to inspect task status using the REST API, and Terraform state when using Consul as backend for Terraform.

For more information on topics covered in this tutorial, check out the following resources.

To learn even more about operating, observing, and monitoring your Consul service mesh, check out the following tutorials and collections.

Previous
Next

This tutorial also appears in:

  • 16 tutorials
    Use Cases for Terraform
    Use Terraform to perform common operations with other technologies, including Consul, Vault, Packer, and Kubernetes.
    • Terraform
  • 5 tutorials
    Network Applications with Terraform
    Use Terraform to control your Networking infrastructure, or interact with Networking tools like HashiCorp Consul. Update firewall rules based on Consul service registration. Use Terraform to register services.
    • Terraform
  • 16 tutorials
    HashiCorp Products - Better Together
    Use Terraform with other Hashicorp products including Vault, Boundary, Consul, Packer, and Hashicorp Cloud Platform.
    • Terraform
  • 8 tutorials
    Network Automation with CTS
    Consul-Terraform-Sync helps you automate infrastructure changes to react to changes in your Consul service catalog.
    • Consul