Build a custom Consul-Terraform-Sync module
Network Infrastructure Automation relies on a declarative workflow and service driven network automation architecture. Changes in configuration are carried out by Consul-Terraform-Sync. Consul-Terraform-Sync leverages Terraform as the underlying automation tool, and leverages the Terraform provider ecosystem to apply relevant changes to your network infrastructure.
This usage of Terraform providers, in conjunction with Consul-Terraform-Sync, allows for Consul to act as the source of truth for your network health state and resources. Consul-Terraform-Sync can update existing configuration by automatically reacting to changes.
This tutorial guides you through the steps necessary to write your own Consul-Terraform-Sync compatible module and publish it on the Terraform registry.
Prerequisites
- GitHub account and basic git familiarity
- Terraform binary and some familiarity with the tool
- Terraform registry account to publish your module
- (Optional) Consul-Terraform-Sync binary to test run your module in an automated scenario.
For this tutorial, you will use an existing example module for Consul-Terraform-Sync, terraform-aws-listener-rule, that creates a listener rule to be added to an AWS application load balancer. The module files will be presented during the tutorial as an implementation example for a working Consul-Terraform-Sync compatible module.
Find the right Terraform provider
The first step in creating your Consul-Terraform-Sync compatible module is to choose an infrastructure management task you want to automate.
Use one or more existing providers
A Terraform provider is a plugin that is responsible for managing API interactions with underlying infrastructure, such as public cloud services (AWS, GCP, Azure), a PaaS service (Heroku), a SaaS service (DNSimple, CloudFlare), or on-prem resources (vSphere).
Terraform provides you with a large number of providers for different target environments, which are available on the Terraform registry. This should help you get started writing your module without having to write an integration specific to your infrastructure.
Create your own provider
If you want to create a custom provider to integrate a custom API in your network infrastructure, you can always write your own provider using the available tutorials. Once your provider is ready, follow Terraform documentation on publishing providers to publish it to the registry.
Tip
Terraform providers should follow the specification defined in the Terraform Provider Development Program. Make sure to review that document before publishing your provider.
Terraform modules
Once you have determined the appropriate Terraform provider or providers you need for your integration, you can begin writing the module that will perform the network automation.
This tutorial provides you with resources and examples of the required structures and file content for your module.
If you are not already familiar with building Terraform modules, you should first review the Terraform modules documentation or our tutorial on building a module.
Consul-Terraform-Sync compatible modules requirements
Testing Terraform is complex because it can create or change real infrastructure. Testing Terraform automation with Consul-Terraform-Sync introduces more complexity. One approach to test your Consul-Terraform-Sync module is to simplify the development environment by testing only the Terraform configuration file and not the full end-to-end workflow. This can be done by manually emulating changes in your infrastructure, changing the values in a mock variables file, and execute Terraform manually.
Note
You do not need Consul-Terraform-Sync or a running Consul agent to test the integration, but the test requires familiarity with Terraform.
Create a Consul-Terraform-Sync compatible module
Compatible modules for Consul-Terraform-Sync follow the Terraform standard module structure. Modules can use syntax supported by Terraform version 0.13 and newer.
Template repository to clone
To simplify the creation of new modules, we have provided a template repository to use as a model for Consul-Terraform-Sync compatible modules.
Clone the repository locally using HTTPS.
Tip
Once you have cloned the repository locally, copy the files into a new folder for your module, and rename the file README.tmpl.md to README.md.
Consul-Terraform-Sync compatibility requirements
There are two required project elements for Consul-Terraform-Sync compatibility, and one optional element that can be used for local testing of the module.
Root module -
main.tf
is the recommended filename for the main file where resources are created. It should contain the references to all the providers used by your integration, as well as any extra resources necessary.Services input variable - Consul-Terraform-Sync requires all compatible modules to declare a
services
variable within the module. This is done by having a file, namedvariables.tf
that contains a map of objects representing the response object from the Consul catalog API. This file defines network information to be consumed by the module. The same file can be extended to include other variables you might need for your module.(Optional) Mock input variables - To simplify testing without a Consul datacenter or the Consul-Terraform-Sync binary, you can create a file that mocks the service information from the Consul catalog using example services. This file must be named
terraform.tfvars
.
Root module
Create a file named main.tf
and populate it with the content from the
main.tf
template file.
Note
Content should resemble the example below. This example is not guaranteed to be up to date. Always refer to the template file provided in the repository.
- For any provider used by the module, add the provider
source and version in the
terraform.required_providers
block. This is a Terraform 0.13 feature.
Note
Consul-Terraform-Sync relies on passing providers implicitly through inheritance to the child module. Do not define providers in the child module Read more on defining providers within modules in the Terraform documentation
- Declare
resource
blocks to describe module behaviors for the infrastructure. You can either use themain.tf
file for this, or create a different file to collect all the extra resources needed by your module.
Note
When deciding which resources to declare, keep in mind that you want your module to react to changes in the catalog as quickly as possible. For this reason, you should not have all of your infrastructure declared. Instead, only include resources that require configuration changes (i.e., load balancer listener or firewall rules). Any other infrastructure (i.e., load balancers or firewalls) should ideally be declared outside of the module.
Declare
local
blocks to perform variable interpolation for the module. You can either use themain.tf
file for this or create a different file to collect all the data manipulation needed by your module.Note that the
services
input variable is passed to the module. This satisfies the module spec requirement for Consul-Terraform-Sync.Pass any
variables
in the module block to set input variables for the module to test. The variables sources might vary. Some service-specific variables might be defined incts_user_defined_meta
, while others might be interpolated usinglocals
or could even be input variables.If you want to define requirements for the Terraform version to be used by the module, you can do so by using the
required_version
setting in theterraform
block.
In the example module, the information is divided across two different files providers.tf
and targets.tf
.
providers.tf
contains only the reference for the AWS provider and its version.
targets.tf
contains any additional resources needed by the module.
As mentioned above, observe that the targets.tf
does not declare the Amazon Load Balancer (ALB),
but instead refers to an existing one. The module only defines the rule changes.
The variables used by the targets.tf
are defined in the variables.tf
file
showed in the next block.
Services input variable
Create a file named variables.tf
and populate it with the contents from the
variables.tf
template file.
Note
The contents should resemble the example below. This example is not guaranteed to be not up to date. Always refer to the template file provided in the repository.
This is the services
variable declared in the root module and is expected to
be compatible with the services
variable declared in the module for testing.
If there are any additional input variables referenced in main.tf
specific to
your test, you can add the variables declarations below var.services
.
Terraform variables when passed as module arguments can be
lossy for object types. This allows Consul-Terraform-Sync to
declare the full variable with every object attribute in the generated root
module, and pass the variable to a child module that contains a subset of these
attributes for its variable declaration. If your module does not require the
whole set of information on the services, you can simplify the services
variable within the module by omitting unused attributes.
For example, the following services variable has four attributes with the rest omitted.
In the example module, the variables.tf
file contains the services
variable to honor the compatibility contract with
Consul-Terraform-Sync. It also contains all the variables needed to interact with the
AWS provider and resources.
Notice the module uses variables such as listener_arn
and blue_target_group_arn
to refer to the existing ALB instead of creating a new one.
Mock input variables
During its execution, Consul-Terraform-Sync will create a file in the task
workspace named terraform.tfvars
. This file will contain information retrieved
from the Consul catalog about the services that match the task definition.
To test your module without using a Consul datacenter, and without using the
Consul-Terraform-Sync binary, you can manually create a file, named terraform.tfvars
,
and fill it with mock values for the services
input variable.
The mock values file below represents service information from the Consul catalog for two services named "api" and "web", with two instances of the service "web".
As long as you retain the same variable structure during your tests, you can modify the values to test changes in your Consul catalog and verify the module behaves properly.
Documentation guidelines
The repository contains a template for a README.md
file that can be used to
document your module.
All Consul-Terraform-Sync compatible modules follow the naming convention
terraform-<PROVIDER>-<NAME>-nia
. Module repositories must use this four-part
name format, where <PROVIDER>
is the Terraform Provider being used, <NAME>
reflects the type of infrastructure the module manages, and ends with the suffix
-nia
to represent that this module is designed for Network Infrastructure
Automation using Consul-Terraform-Sync.
Read more on this on the Network Infrastructure Automation Integration Program page.
Test your module
To configure Consul-Terraform-Sync to use your module you have to configure your task to locate the module and, if needed, to specify the required input variables for it.
Configure your task's module
Depending on your internal policies, or your internal development process, your module might not be hosted on the Terraform registry when you are testing it.
Consul-Terraform-Sync permits you to define different modules in your task
definition using the module
field.
Allowed module sources for tasks are:
- local paths - this is probably the first configuration you will use while
developing your module. A local path must begin with either
./
or../
to indicate that a local path is intended. This distinguishes it from a module registry address.
- GitHub repository - if you are developing your module to be published on
the Terraform registry, you will be using a GitHub repository. It
is possible to use the GitHub repository as the
module
for your task until the module gets published.
By default, Terraform will clone and use the default branch referenced by
HEAD
in the selected repository. You can override this using the ref
argument.
Note
You will need to configure credentials to access private repositories.
- Terraform registry - Once you publish the module, you can use the
Terraform registry as the
module
for the task. Using the registry as the source of the module also allows you to specify the version for the module.
- Terraform Cloud and Terraform Enterprise private registry - You can also use a private registry hosted on an enterprise or cloud instance of Terraform as the source for your module.
Note
You will need to configure credentials to access private module registries.
For the full list of available sources for Terraform modules check the documentation.
Configure required input variables
If your module requires extra variables outside the ones defined in the
services
object, you can pass them to the task using the variable_files
field.
This is a list of paths to files containing variables for the task. These are
used as Terraform variable definitions (.tfvars
) for the Terraform driver
files and contain only variable name assignments.
Module publishing guide
Remember to follow the publishing guidelines for Terraform modules during development.
Once your deployment is complete, and you decide to publish your module to the Terraform registry, you can follow the NIA program steps listed in the Consul documentation.
Terraform registry modules
Here is a list of available Consul-Terraform-Sync compatible modules already published on the Terraform registry.
You can use them in your environment or use them as an example to develop your own module.
- A10 Networks Thunder ADC Service Group
- Check Point Software Technologies Dynamic Objects
- Cisco ACI PBR Service-Graph
- F5 BIG-IP Application
- Palo Alto Networks PAN-OS Dynamic Address Group (DAG) Tags
- Palo Alto Networks PAN-OS Address Group and Dynamic Address Group (DAG) Tags
Next steps
In this tutorial, you learned how to create a customized Consul-Terraform-Sync module for your Network Infrastructure Automation. You learned the basic structure for a Terraform module to be compatible with Consul-Terraform-Sync, and how to customize that basic structure to include variables and objects relative to your specific Terraform providers.
Please engage in the review process once one or two sample modules have been developed. Begin the process by emailing nia-integration-dev@hashicorp.com with a URL to the public GitHub repo containing the code.
In the next tutorial, you will learn how to secure the Consul-Terraform-Sync instance and other best practices to integrate it in a production environment.
To learn more about Network Infrastructure Automation with Consul-Terraform-Sync, check the full documentation for it on the Consul website.