Connect services between Consul datacenters with cluster peering
Service meshes provide secure communication across your services within and across your infrastructure, including on-premises and cloud environments. As your organization scales, it may need to deploy services in multiple cloud providers in different regions. Cluster peering enables you to connect multiple Consul clusters, letting services in one cluster securely communicate with services in the other.
Cluster peering removes some of the administrative burdens associated with WAN federation. Because there is no primary cluster, administrative boundaries are clearly separated per cluster since changes in one Consul cluster does not affect peered clusters. For more information on differences between WAN federation and cluster peering, refer to the cluster peering documentation.
In this tutorial you will:
- Deploy two managed Kubernetes environments with Terraform
- Deploy Consul in each Kubernetes cluster
- Deploy the microservices from HashiCups, a demo application, in both Kubernetes cluster
- Peer the two Consul clusters
- Connect the services across the peered service mesh
Scenario overview
HashiCups is a coffee-shop demo application. It has a microservices architecture and uses Consul service mesh to securely connect the services. In this tutorial, you will deploy HashiCups services on Kubernetes clusters in two different AWS regions. By peering the Consul clusters, the frontend services in one region will be able to communicate with the API services in the other.
HashiCups uses the following microservices:
- The
nginx
service is an NGINX instance that routes requests to thefrontend
microservice and serves as a reverse proxy to thepublic-api
service. - The
frontend
service provides a React-based UI. - The
public-api
service is a GraphQL public API that communicates with theproducts-api
and thepayments
services. - The
product-api
service stores the core HashiCups application logic, including authentication, coffee (product) information, and orders. - The
postgres
service is a Postgres database instance that stores user, product, and order information. - The
payments
service is a gRCP-based Java application service that handles customer payments.
Prerequisites
If you are not familiar with Consul's core functionality, refer to the Consul Getting Started tutorials collection first.
For this tutorial, you will need:
- An AWS account configured for use with Terraform
- aws-cli v2.0 or later
This tutorial uses Terraform automation to deploy the demo environment. You do not need to know Terraform to successfully complete this tutorial.
Clone example repository
Clone the GitHub repository containing the configuration files and resources.
Change into the directory with the newly cloned repository.
This repository has the following:
- The
dc1
directory contains Terraform configuration to deploy an EKS cluster inus-east-2
. - The
dc2
directory contains Terraform configuration to deploy an EKS cluster ineu-west-2
. - The
k8s-yamls
directory contains YAML configuration files that support this tutorial. - The
hashicups-v1.0.2
directory contains YAML configuration files for deploying HashiCups.
Deploy Kubernetes clusters and Consul
In this section, you will create a Kubernetes cluster on each datacenter, and install Consul to provide service mesh functionality.
Initialize the Terraform configuration for dc1
to download the necessary providers and modules.
Open a new terminal window and initialize the Terraform configuration for dc2
.
Then, deploy the resources for dc1
. Confirm the run by entering yes
. This will take about 15 minutes to deploy your infrastructure.
Deploy the resources for dc2
. Confirm the run by entering yes
.
Configure kubectl
Now that you have deployed the two datacenters, configure the kubectl
tool to interact with the Kubernetes cluster in the first datacenter.
Notice that this command stores the cluster connection information in the dc1
alias.
Configure the kubectl
tool to interact with the Kubernetes cluster in the second datacenter.
Notice that this command stores the cluster connection information in the dc2
alias.
Deploy Consul on both Kubernetes clusters
You will now deploy Consul on your Kubernetes platforms with the Consul K8S CLI. By default, Consul deploys into its own dedicated namespace (consul
). The Consul installation will use the Consul Helm chart file in the k8s-yaml
directory. Cluster peering requires Consul v1.13.1+ and the global.peering.enabled
parameter set to true
. Deploying Consul on each Kubernetes cluster should only take a few minutes.
Deploy Consul on the Kubernetes cluster in the first datacenter. Confirm the installation with a y
. Notice that this command sets global.datacenter
to dc1
.
Warning
Make sure to run the correct version of consul-k8s
otherwise the deployment will fail. This tutorial uses the 1.1.x release train. Refer to the consul-k8s CLI documentation on how to install a specific version on your system.
Verify that you have installed Consul in dc1
by inspecting the Kubernetes pods in the consul
namespace.
Then, deploy Consul on the Kubernetes cluster in the second datacenter. Confirm the installation with a y
. Notice that this command sets global.datacenter
to dc2
.
Verify that you have installed Consul in dc2
by inspecting the Kubernetes pods in the consul
namespace.
Deploy HashiCups
You will now deploy the HashiCups microservices on your Kubernetes clusters. The dc1
Kubernetes cluster will host the frontend services, while the dc2
Kubernetes cluster will host the API and database services. Later in this tutorial, you will connect the Consul datacenters to form the complete HashiCups deployment. The following diagram illustrates how HashiCups will be deployed across the two clusters.
Deploy HashiCups on first cluster
Deploy the frontend
, nginx
, public-api
, and payments
services, along with intentions-dc1
, to the dc1
Kubernetes cluster.
You can view the HashiCups frontend, but the demo application will not display any products because products-api
is not deployed.
Verify that you have successfully deployed the services by listing the Kubernetes services.
List the services registered with Consul in dc1
. This command runs consul catalog services
in one of the Consul server agents.
Deploy products-api microservice on second cluster
Deploy the product-api
and postgres
services, along with intentions-dc2
, to the dc2
Kubernetes cluster.
Verify that you have successfully deployed the services by listing the Kubernetes services.
List the services registered with Consul in dc2
.
Explore the Consul UI (optional)
Retrieve the Consul UI address for dc1
and open it in your browser.
Retrieve the Consul UI address for dc2
and open it in your browser.
Explore HashiCups in browser
Open the HashiCups application. First, open a new terminal and port forward the nginx
service locally to port 8080
.
Open localhost:8080 in your browser to view the HashiCups UI. Notice that it displays no products, since there is no instance of product-api
on dc1
.
Configure Consul cluster peering
Tip
Consul cluster peering works on both Enterprise and OSS versions of Consul. On Consul OSS, you can only peer clusters between the default
partitions. On Consul Enterprise, you can peer clusters between any partition.
You will now peer the two data centers to enable services in dc1
to communicate to product-api
in dc2
.
Consul cluster peering works by defining two cluster roles:
- A peering acceptor is the cluster that generates a peering token and accepts an incoming peering connection.
- A peering dialer is the cluster that uses a peering token to make an outbound peering connection with the cluster that generated the token.
Configure cluster peering traffic routing
You can peer Consul clusters by either directly connecting Consul server nodes or connecting the Consul mesh gateways.
Most Kubernetes deployments will not let services connect outside the cluster. This prevents the Consul server pods from communicating to other Kubernetes clusters. Therefore, we recommend configuring the clusters to use mesh gateways for peering. The following file configures the Consul clusters to use mesh gateways for cluster peering:
Configure cluster peering traffic for both dc1
and dc2
to be routed via the mesh gateways.
There are two modes for routing traffic from local services to remote services when cluster peering connections are routed through mesh gateways. In remote
mode, your local services contact the remote mesh gateway in order to reach remote services. In local
mode, your local services contact their local gateway in order to reach remote services. Refer to the modes documentation and well as the Mesh architecture diagram for more information.
We recommend you use local
mode because most Kubernetes deployments do not allow local services to connect outside the cluster. The following configuration specifies local
mode for traffic routed over the mesh gateways:
Configure local
mode for traffic routed over the mesh gateways for both dc1
and dc2
.
Create a peering token
Configuring the peering acceptor role for a cluster generates a peering token and waits to accept an incoming peering connection. The following configuration sets dc1
as the peering acceptor:
Configure a PeeringAcceptor
role for dc1
.
Confirm you successfully created the peering acceptor custom resource definition (CRD).
Confirm that the PeeringAcceptor CRD generated a peering token secret.
Import the peering token generated in dc1
into dc2
.
Establish a connection between clusters
Configuring a peering dialer role for a cluster makes an outbound peering connection towards a peering acceptor cluster using the specified peering token. The following configuration sets dc2
as the peering dialer and peering-token-dc2
as its token.
Configure a PeeringDialer
role for dc2
. This will create a peering connection from the second datacenter towards the first one.
Verify that the two Consul clusters are peered. This command queries the peering
API endpoint on the Consul server agent in dc1
.
Notice the state is Active
, which means that the two clusters are peered successfully.
Export the products-api service
After you peer the Consul clusters, you need to create a configuration entry that defines the services you want to export to other clusters. Consul uses this configuration entry to advertise those services' information and connect those services across Consul clusters. The following configuration exports the products-api
service into the dc1
peer.
In dc2
, apply the ExportedServices
custom resource file that exports the products-api
service to dc1
.
Confirm that the Consul cluster in dc1
can access the products-api
in dc2
. This command queries the services
API endpoint on the Consul server agent in dc1
about the products-api
service from dc2
for its sidecar service ID and the related peer name.
Notice the output contains the products-api
sidecar service ID and the name of the related cluster peering.
Create a cross-cluster service intention
In order for communication from products-api
service in dc1
to reach public-api
in dc2
, you must define a ServiceIntentions
custom resource definition that enables communication from products-api
service in dc1
.
Create a service intention in dc2
that that allows communication from the public-api
service in dc1
to the products-api
service in dc2
.
Set up new upstream for the public-api service
At this point, the public-api
service in dc1
is configured to connect to its local instance of the products-api
service. To use the instance of products-api
hosted in another datacenter, you must define the upstream to point to the products-api
service in dc2
. Consul uses the DNS syntax for service virtual IP lookups to contact services across mesh gateways, so the upstream address is formatted as <service>.virtual[.<namespace>].<peer>.<domain>
. The namespace
segment is only available in Consul Enterprise. In this scenario, the lookup is products-api.virtual.dc2.consul
. The following configuration is for the updated public-api
service in dc1
:
Override the public-api
service definition in dc1
with the updated configuration pointing to upstream in dc2
.
Verify peered Consul services
Port forward the nginx
service locally to port 8080
.
Then, open localhost:8080 in your browser. Notice that it now displays a curated selection of coffee drinks.
Destroy environment
Now that you have peered two Consul clusters, you will now remove the exported service and cluster peering, before destroying the environment.
Remove exported service
Stop the products-api
service from being exported.
Remove cluster peering
To remove a peering connection, delete both the PeeringAcceptor
and PeeringDialer
resources.
First, delete the PeeringDialer
from dc2
.
Then, delete the PeeringAcceptor
from dc1
.
Verify the two clusters are no longer peered by querying the /health
HTTP endpoint in dc1
.
Delete supporting infrastructure
To destroy the environment, first uninstall Consul from both Kubernetes clusters.
First, uninstall Consul from dc1
. Confirm with a y
.
Then, uninstall Consul from dc2
. Confirm with a y
.
Note
Before running the terraform destroy
command, make sure that all the services in the Consul namespace have been terminated. If you try to perform a destroy before that, your Terraform run will fail and you will have to restart it.
Then, destroy the supporting infrastructure in your first datacenter.
Verify that you have removed all services from your second datacenter.
Then, destroy the supporting infrastructure in your second datacenter.
Next steps
In this tutorial, you used the Consul cluster peering to route traffic across service meshes in two Consul clusters. In the process, you learned the benefits of using cluster peering for cluster interconnections with minimal shared administrative overhead.
Feel free to explore these tutorials and collections to learn more about Consul service mesh, microservices, and Kubernetes security.