Build a custom Consul-Terraform-Sync module

20min
|
Terraform
Consul

Network Infrastructure Automation relies on a declarative workflow and service driven network automation architecture. Changes in configuration are carried out by Consul-Terraform-Sync. Consul-Terraform-Sync leverages Terraform as the underlying automation tool, and leverages the Terraform provider ecosystem to apply relevant changes to your network infrastructure.

NIA CTS architecture diagram

This usage of Terraform providers, in conjunction with Consul-Terraform-Sync, allows for Consul to act as the source of truth for your network health state and resources. Consul-Terraform-Sync can update existing configuration by automatically reacting to changes.

This tutorial guides you through the steps necessary to write your own Consul-Terraform-Sync compatible module and publish it on the Terraform registry.

Prerequisites

GitHub account and basic git familiarity
Terraform binary and some familiarity with the tool
Terraform registry account to publish your module
(Optional) Consul-Terraform-Sync binary to test run your module in an automated scenario.

For this tutorial, you will use an existing example module for Consul-Terraform-Sync, terraform-aws-listener-rule, that creates a listener rule to be added to an AWS application load balancer. The module files will be presented during the tutorial as an implementation example for a working Consul-Terraform-Sync compatible module.

Find the right Terraform provider

The first step in creating your Consul-Terraform-Sync compatible module is to choose an infrastructure management task you want to automate.

Use one or more existing providers

A Terraform provider is a plugin that is responsible for managing API interactions with underlying infrastructure, such as public cloud services (AWS, GCP, Azure), a PaaS service (Heroku), a SaaS service (DNSimple, CloudFlare), or on-prem resources (vSphere).

Terraform provides you with a large number of providers for different target environments, which are available on the Terraform registry. This should help you get started writing your module without having to write an integration specific to your infrastructure.

Create your own provider

If you want to create a custom provider to integrate a custom API in your network infrastructure, you can always write your own provider using the available tutorials. Once your provider is ready, follow Terraform documentation on publishing providers to publish it to the registry.

Tip

Terraform providers should follow the specification defined in the Terraform Provider Development Program. Make sure to review that document before publishing your provider.

Terraform modules

Once you have determined the appropriate Terraform provider or providers you need for your integration, you can begin writing the module that will perform the network automation.

This tutorial provides you with resources and examples of the required structures and file content for your module.

If you are not already familiar with building Terraform modules, you should first review the Terraform modules documentation or our tutorial on building a module.

Consul-Terraform-Sync compatible modules requirements

Testing Terraform is complex because it can create or change real infrastructure. Testing Terraform automation with Consul-Terraform-Sync introduces more complexity. One approach to test your Consul-Terraform-Sync module is to simplify the development environment by testing only the Terraform configuration file and not the full end-to-end workflow. This can be done by manually emulating changes in your infrastructure, changing the values in a mock variables file, and execute Terraform manually.

Note

You do not need Consul-Terraform-Sync or a running Consul agent to test the integration, but the test requires familiarity with Terraform.

Create a Consul-Terraform-Sync compatible module

Compatible modules for Consul-Terraform-Sync follow the Terraform standard module structure. Modules can use syntax supported by Terraform version 0.13 and newer.

Template repository to clone

To simplify the creation of new modules, we have provided a template repository to use as a model for Consul-Terraform-Sync compatible modules.

Clone the repository locally using HTTPS.

git clone https://github.com/hashicorp/consul-terraform-sync-template-module.git

Tip

Once you have cloned the repository locally, copy the files into a new folder for your module, and rename the file README.tmpl.md to README.md.

Clone the repository locally using SSH.

git clone git@github.com:hashicorp/consul-terraform-sync-template-module.git

Tip

Once you have cloned the repository locally, copy the files into a new folder for your module, and rename the file README.tmpl.md to README.md.

Consul-Terraform-Sync compatibility requirements

There are two required project elements for Consul-Terraform-Sync compatibility, and one optional element that can be used for local testing of the module.

Root module - main.tf is the recommended filename for the main file where resources are created. It should contain the references to all the providers used by your integration, as well as any extra resources necessary.
Services input variable - Consul-Terraform-Sync requires all compatible modules to declare a services variable within the module. This is done by having a file, named variables.tf that contains a map of objects representing the response object from the Consul catalog API. This file defines network information to be consumed by the module. The same file can be extended to include other variables you might need for your module.
(Optional) Mock input variables - To simplify testing without a Consul datacenter or the Consul-Terraform-Sync binary, you can create a file that mocks the service information from the Consul catalog using example services. This file must be named terraform.tfvars.

Root module

Create a file named main.tf and populate it with the content from the main.tf template file.

Note

Content should resemble the example below. This example is not guaranteed to be up to date. Always refer to the template file provided in the repository.

main.tf

terraform {
  required_providers {
    # Provider source is used for Terraform discovery and installation of
    # providers. Declare source for all providers required by the module.
    myprovider = {
      source  = "namespace/myprovider"
      version = "~>1.1.0"
    }
  }
}

#
# Declare resource blocks to describe module behaviors for the infrastructure.
#
# The example block below describes creating an address group for each Consul
# service, and applies an existing policy to the group.
#
resource "myprovider_address_group" "consul_service" {
  for_each = local.consul_services

  name     = "${var.address_group_prefix}${each.key}"
  tags     = var.address_group_tags
  policies = [each.value.cts_user_defined_meta["policy_name"]]
}

#
# You can utilize the locals block to transform the var.services variable
# into a data structure for your module. For more examples of common data
# transformations, visit the project wiki.
# https://github.com/hashicorp/consul-terraform-sync-template-module/wiki
#
# The example below converts var.services to a map of service names to a list
# of service instances.
locals {
  # Create a map of service names to instance IDs to then build
  # a map of service names to instances
  consul_service_ids = transpose({
    for id, s in var.services : id => [s.name]
  })

  # Group service instances by service name
  # consul_services = {
  #   "app" = [
  #     {
  #       "id" = "app-id-01"
  #       "name" = "app"
  #       "node_address" = "192.168.10.10"
  #     }
  #   ]
  # }
  consul_services = {
    for name, ids in local.consul_service_ids :
    name => [for id in ids : var.services[id]]
  }
}

For any provider used by the module, add the provider source and version in the terraform.required_providers block. This is a Terraform 0.13 feature.

Note

Consul-Terraform-Sync relies on passing providers implicitly through inheritance to the child module. Do not define providers in the child module Read more on defining providers within modules in the Terraform documentation

Declare resource blocks to describe module behaviors for the infrastructure. You can either use the main.tf file for this, or create a different file to collect all the extra resources needed by your module.

Note

When deciding which resources to declare, keep in mind that you want your module to react to changes in the catalog as quickly as possible. For this reason, you should not have all of your infrastructure declared. Instead, only include resources that require configuration changes (i.e., load balancer listener or firewall rules). Any other infrastructure (i.e., load balancers or firewalls) should ideally be declared outside of the module.

Declare local blocks to perform variable interpolation for the module. You can either use the main.tf file for this or create a different file to collect all the data manipulation needed by your module.
Note that the services input variable is passed to the module. This satisfies the module spec requirement for Consul-Terraform-Sync.
Pass any variables in the module block to set input variables for the module to test. The variables sources might vary. Some service-specific variables might be defined in cts_user_defined_meta, while others might be interpolated using locals or could even be input variables.
If you want to define requirements for the Terraform version to be used by the module, you can do so by using the required_version setting in the terraform block.

Click here for an example from existing module

In the example module, the information is divided across two different files providers.tf and targets.tf.

providers.tf contains only the reference for the AWS provider and its version.

providers.tf

terraform {
  required_version = ">=0.14"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">=3.23"
    }
  }
}

targets.tf contains any additional resources needed by the module.

targets.tf

resource "aws_lb_target_group" "canary" {
  name        = "${local.name.0}-${local.datacenter.0}"
  port        = local.port.0
  protocol    = "HTTP"
  vpc_id      = var.vpc_id
  target_type = "ip"

  health_check {
    enabled = var.enable_health_check
    path    = var.health_check_path
    matcher = var.health_check_matcher
  }
}

resource "aws_lb_target_group_attachment" "app_canary" {
  for_each          = local.ip_addresses
  target_group_arn  = aws_lb_target_group.canary.arn
  target_id         = each.value
  port              = local.port.0
  availability_zone = "all"
}

resource "aws_lb_listener_rule" "app_canary" {
  listener_arn = var.listener_arn
  priority     = var.listener_rule_priority

  action {
    type = "forward"
    forward {
      target_group {
        arn    = var.blue_target_group_arn
        weight = 100 - local.weight.0
      }

      target_group {
        arn    = aws_lb_target_group.canary.arn
        weight = local.weight.0
      }

      stickiness {
        enabled  = var.enable_stickiness
        duration = var.stickiness_duration
      }
    }
  }

  condition {
    host_header {
      values = [local.host.0]
    }
  }
}

As mentioned above, observe that the targets.tf does not declare the Amazon Load Balancer (ALB), but instead refers to an existing one. The module only defines the rule changes.

The variables used by the targets.tf are defined in the variables.tf file showed in the next block.

Services input variable

Create a file named variables.tf and populate it with the contents from the variables.tf template file.

Note

The contents should resemble the example below. This example is not guaranteed to be not up to date. Always refer to the template file provided in the repository.

variables.tf

variable "services" {
  description = "Consul services monitored by Consul Terraform Sync"
  type = map(
    object({
      id        = string
      name      = string
      kind      = string
      address   = string
      port      = number
      meta      = map(string)
      tags      = list(string)
      namespace = string
      status    = string

      node                  = string
      node_id               = string
      node_address          = string
      node_datacenter       = string
      node_tagged_addresses = map(string)
      node_meta             = map(string)

      cts_user_defined_meta = map(string)
    })
  )
}

This is the services variable declared in the root module and is expected to be compatible with the services variable declared in the module for testing. If there are any additional input variables referenced in main.tf specific to your test, you can add the variables declarations below var.services.

Terraform variables when passed as module arguments can be lossy for object types. This allows Consul-Terraform-Sync to declare the full variable with every object attribute in the generated root module, and pass the variable to a child module that contains a subset of these attributes for its variable declaration. If your module does not require the whole set of information on the services, you can simplify the services variable within the module by omitting unused attributes.

For example, the following services variable has four attributes with the rest omitted.

variable "services" {
  description = "Consul services monitored by Consul-Terraform-Sync"
  type = map(
    object({
      id           = string
      name         = string
      node_address = string
      port         = number
      status       = string
    })
  )
}

Click here for an example from existing module

In the example module, the variables.tf file contains the services variable to honor the compatibility contract with Consul-Terraform-Sync. It also contains all the variables needed to interact with the AWS provider and resources.

Notice the module uses variables such as listener_arn and blue_target_group_arn to refer to the existing ALB instead of creating a new one.

variables.tf

variable "services" {
  description = "Consul services monitored by Consul-Terraform-Sync"
  type = map(
    object({
      id        = string
      name      = string
      kind      = string
      address   = string
      port      = number
      meta      = map(string)
      tags      = list(string)
      namespace = string
      status    = string

      node                  = string
      node_id               = string
      node_address          = string
      node_datacenter       = string
      node_tagged_addresses = map(string)
      node_meta             = map(string)

      cts_user_defined_meta = map(string)
    })
  )
}

variable "listener_arn" {
  type        = string
  description = "ARN of listener to update"
}

variable "blue_target_group_arn" {
  type        = string
  description = "ARN of blue target group (existing target group with working application)"
}

variable "vpc_id" {
  type        = string
  description = "ID of the VPC for the target group"
}

variable "green_weight" {
  type        = number
  description = "Percentage of traffic to send to green deployment"
  default     = 0
}

variable "enable_health_check" {
  type        = bool
  description = "Enable health checking for target group"
  default     = true
}

variable "health_check_path" {
  type        = string
  description = "Path of health check for applications"
  default     = "/health"
}

variable "health_check_matcher" {
  type        = string
  description = "HTTP codes for health check"
  default     = "200"
}

variable "enable_stickiness" {
  type        = bool
  description = "Enable stickiness"
  default     = false
}

variable "stickiness_duration" {
  type        = number
  description = "Duration of stickness in seconds"
  default     = 600
}

variable "listener_rule_priority" {
  type        = number
  default     = 1
  description = "Priority of listener rule between 1 to 50000"
  validation {
    condition     = var.listener_rule_priority > 0 && var.listener_rule_priority < 50000
    error_message = "The priority of listener rule must between 1 to 50000."
  }
}

variable "service_kind" {
  type        = string
  description = "Kind of Consul service. Can be ingress-gateway, terminating-gateway."
  default     = ""
}

locals {
  name = distinct([
    for service, service_data in var.services :
    service_data.name if service_data.kind == var.service_kind
  ])

  ip_addresses = toset([
    for service, service_data in var.services :
    replace(replace(split(".", service_data.node)[0], "ip-", ""), "-", ".") if service_data.kind == var.service_kind
  ])

  port = distinct([
    for service, service_data in var.services :
    service_data.port if service_data.kind == var.service_kind
  ])

  host = coalescelist(distinct([
    for service, service_data in var.services :
    service_data.cts_user_defined_meta.host if service_data.kind == ""
  ]))

  weight = distinct([
    for service, service_data in var.services :
    lookup(service_data.meta, "weight", 0) if service_data.kind == ""
  ])

  datacenter = distinct([
    for service, service_data in var.services :
    service_data.node_datacenter if service_data.kind == var.service_kind
  ])
}

Mock input variables

During its execution, Consul-Terraform-Sync will create a file in the task workspace named terraform.tfvars. This file will contain information retrieved from the Consul catalog about the services that match the task definition.

To test your module without using a Consul datacenter, and without using the Consul-Terraform-Sync binary, you can manually create a file, named terraform.tfvars, and fill it with mock values for the services input variable.

The mock values file below represents service information from the Consul catalog for two services named "api" and "web", with two instances of the service "web".

As long as you retain the same variable structure during your tests, you can modify the values to test changes in your Consul catalog and verify the module behaves properly.

Click here for an example of mock values for module testing

terraform.tfvars

services = {
  "api" : {
    address = "172.17.0.1"
    id      = "api"
    name    = "api"
    kind    = ""
    port    = 80
    meta = {}
    tags            = []
    namespace       = null
    status          = "passing"
    node_id         = "node_a"
    node            = "foobar"
    node_address    = "192.168.10.10"
    node_datacenter = "dc1"
    node_tagged_addresses = {
      lan = "192.168.10.10"
      wan = "10.0.10.10"
    }
    node_meta = {}
    cts_user_defined_meta = {}
  },
  "web_1" : {
    address = "172.17.0.3"
    id      = "web_1"
    name    = "web"
    kind    = ""
    port    = 5000
    meta = {
      foobar_meta_value = "baz"
    }
    tags            = ["tacos"]
    namespace       = null
    status          = "passing"
    node_id         = "node_a"
    node            = "foobar"
    node_address    = "192.168.10.10"
    node_datacenter = "dc1"
    node_tagged_addresses = {
      lan = "192.168.10.10"
      wan = "10.0.10.10"
    }
    node_meta = {
      somekey = "somevalue"
    }
    cts_user_defined_meta = {}
  },
  "web_2" : {
    address = "172.17.0.3"
    id      = "web_2"
    name    = "web"
    kind    = ""
    port    = 5000
    meta = {
      foobar_meta_value = "baz"
    }
    tags            = ["burrito"]
    namespace       = null
    status          = "passing"
    node_id         = "node_b"
    node            = "foobarbaz"
    node_address    = "192.168.10.11"
    node_datacenter = "dc1"
    node_tagged_addresses = {
      lan = "192.168.10.11"
      wan = "10.0.10.10"
    }
    node_meta = {}
    cts_user_defined_meta = {}
  }
}

Documentation guidelines

The repository contains a template for a README.md file that can be used to document your module.

All Consul-Terraform-Sync compatible modules follow the naming convention terraform-<PROVIDER>-<NAME>-nia. Module repositories must use this four-part name format, where <PROVIDER> is the Terraform Provider being used, <NAME> reflects the type of infrastructure the module manages, and ends with the suffix -nia to represent that this module is designed for Network Infrastructure Automation using Consul-Terraform-Sync.

Read more on this on the Network Infrastructure Automation Integration Program page.

Test your module

To configure Consul-Terraform-Sync to use your module you have to configure your task to locate the module and, if needed, to specify the required input variables for it.

Configure your task's module

Depending on your internal policies, or your internal development process, your module might not be hosted on the Terraform registry when you are testing it.

Consul-Terraform-Sync permits you to define different modules in your task definition using the module field.

Allowed module sources for tasks are:

local paths - this is probably the first configuration you will use while developing your module. A local path must begin with either ./ or ../ to indicate that a local path is intended. This distinguishes it from a module registry address.

## ...
task {
  ## ...
  module = "../example/module"
  ## ...
}
## ...

GitHub repository - if you are developing your module to be published on the Terraform registry, you will be using a GitHub repository. It is possible to use the GitHub repository as the module for your task until the module gets published.

## ...
task {
  ## ...
  module = "github.com/hashicorp/example-module"
  ## ...
}
## ...

By default, Terraform will clone and use the default branch referenced by HEAD in the selected repository. You can override this using the ref argument.

Note

You will need to configure credentials to access private repositories.

Terraform registry - Once you publish the module, you can use the Terraform registry as the module for the task. Using the registry as the source of the module also allows you to specify the version for the module.

## ...
task {
  ## ...
  module  = "joatmon08/listener-rule-nia/aws"
  version = "0.2.1"
  ## ...
}
## ...

Terraform Cloud and Terraform Enterprise private registry - You can also use a private registry hosted on an enterprise or cloud instance of Terraform as the source for your module.

Note

You will need to configure credentials to access private module registries.

For the full list of available sources for Terraform modules check the documentation.

Configure required input variables

If your module requires extra variables outside the ones defined in the services object, you can pass them to the task using the variable_files field.

This is a list of paths to files containing variables for the task. These are used as Terraform variable definitions (.tfvars) for the Terraform driver files and contain only variable name assignments.

##...
task {
  name        = "example"
  ## ...
  variable_files = [
    "/home/user/test-module/datacenter.module.tfvars",
    "/home/user/test-module/service.tfvars"
  ]
}

Module publishing guide

Remember to follow the publishing guidelines for Terraform modules during development.

Once your deployment is complete, and you decide to publish your module to the Terraform registry, you can follow the NIA program steps listed in the Consul documentation.

Terraform registry modules

Here is a list of available Consul-Terraform-Sync compatible modules already published on the Terraform registry.

You can use them in your environment or use them as an example to develop your own module.

Next steps

In this tutorial, you learned how to create a customized Consul-Terraform-Sync module for your Network Infrastructure Automation. You learned the basic structure for a Terraform module to be compatible with Consul-Terraform-Sync, and how to customize that basic structure to include variables and objects relative to your specific Terraform providers.

Please engage in the review process once one or two sample modules have been developed. Begin the process by emailing nia-integration-dev@hashicorp.com with a URL to the public GitHub repo containing the code.

In the next tutorial, you will learn how to secure the Consul-Terraform-Sync instance and other best practices to integrate it in a production environment.

To learn more about Network Infrastructure Automation with Consul-Terraform-Sync, check the full documentation for it on the Consul website.

CTS and Terraform Enterprise/Cloud

CTS and A10 ADC