Secure Service Mesh Communication Across Kubernetes Clusters
You can secure service-to-service communication across multiple Kubernetes clusters with Consul's mesh gateway feature. Mesh gateways enable you to secure cross-datacenter communication that may be sent over the public internet with mTLS.
In this tutorial, you will deploy two Consul datacenters on separate Kubernetes clusters with Consul's service mesh, WAN federation, and mesh gateways configured. You will then deploy two services into the service mesh, one in each Consul datacenter. To securely connect the two services, you will configure the service sidecar proxies to route communication through the mesh gateways. Finally, you will test that the services are able to communicate.
Prerequisites
To successfully complete this tutorial, you will need the following environments provisioned.
- Two Kubernetes clusters running in environments that support external load balancers.
kubectl
installed locally with two contexts, one for each Kubernetes cluster.Helm
or the Consul-K8S CLI 0.43.0+ installed locally.
Security Warning: This tutorial is not for production use. Please refer to the Kubernetes deployment guide to determine how you can secure Consul on Kubernetes in production. Additionally, we recommend you use a properly secured Kubernetes cluster or make sure that you understand and enable the recommended security features.
Deploy Consul
You can deploy a complete Consul datacenter using the official Consul Helm chart or the Consul K8S CLI. Feel free to review the Consul Kubernetes installation documentation to learn more about these installation options.
Create a custom values file
You will learn how to configure each component section of the custom Consul values file, and then examine and apply the complete configuration for each datacenter.
Configure Consul service mesh
By default, all Consul agents will be added to the Consul service mesh and catalog. However, your Kubernetes services will still need sidecar proxies to secure communication.
The first step to enable Consul to automatically add a sidecar proxy to all the service pods is to install the resources necessary on the Kubernetes node. That is achieved by adding the following stanza to the values yaml files that will be applied to your datacenters.
Note
You will still need to configure the service to automatically deploy sidecar proxies at deployment time.
Configure mesh gateways
Next, enable a mesh gateway. A mesh gateway is a proxy that provides an accessible IP address that other datacenters can reach. This also resolves issues with pod IP address ranges overlapping between datacenters. This stanza will also need to be included in the values yaml files that will be applied to your datacenters.
Configure WAN federation
WAN federation connects Consul servers from multiple datacenters into the same WAN gossip pool. WAN federation enables services to discover each other across datacenters. The WAN federation configuration is slightly different for in primary versus secondary datacenters. This yaml snippet shows how to configure the primary data center. Notice the createFederationSecret entry. This should only be set in a primary datacenter. Later in this tutorial you will export the secret from the primary datacenter and inject it into the secondary datacenter. This will allow the secondary datacenter to automatically negotiate WAN federation with the primary.
Note Mesh gateways also require TLS encryption.
Review the complete custom values file
Finish configuring dc1
and the Consul agents. Below is a complete Consul datacenter
configuration file, dc1-values.yaml
.
Warning
By default, the chart will install an insecure configuration of Consul. This provides a less complicated out-of-box experience for new users, but is not appropriate for a production setup. Review the Secure Consul and Registered Services on Kubernetes tutorial for instructions on how to secure your datacenter for production.
Install Consul in your dc1 cluster
You can now deploy a complete Consul datacenter in your Kubernetes cluster using the official Consul Helm chart or the Consul K8S CLI.
Your kubectl
context should be connected to the Kubernetes cluster where you are deploying Consul datacenter dc1
.
Verify that Consul is running in your Kubernetes cluster using kubectl
. Consul
setup is complete when all pods have a status of Running
, as illustrated in the
following output.
Set mesh gateway mode
Mesh gateways can be configured per datacenter in one of three modes: local
,
remote
, or none
. The official documentation has full details on the implications of running in each mode.
For this tutorial, create a file named proxy-defaults.yaml
that contains a CRD
specification that globally configures all proxies to run in local
mode.
Use kubectl
to apply the CRD.
Export secrets
You will need to export the federation secret created with Consul datacenter dc1
to use with Consul datacenter dc2
.
Deploy Consul datacenter dc2
Now that you have deployed Consul datacenter dc1
, you can configure and deploy
dc2
. Connect your kubectl
context to dc2
.
Create the namespace consul
.
Create the federation secret in dc2
.
Now that you have prepared your Kubernetes cluster, finish configuring dc2
and
the Consul agents. The dc2
datacenter will need the following additional options
configured:
caCert
- contains the certificate of the CA to use for TLS communication, which is the Consul federation secret exported from Consul datacenterdc1
.caKey
- contains the private key of the CA to use for TLS communication, which is the Consul federation secret exported from Consul datacenterdc1
.server
config - contains the server information from Consul datacenterdc1
necessary to configure federation.
Create the customized chart dc2-values.yaml
.
Finally, install Consul on dc2
with Helm or the Consul-K8S CLI.
Check that Consul is running in your Kubernetes cluster using kubectl
. Consul
setup is complete when all pods have a status of Running
, as illustrated in the
following output.
Set mesh gateway mode
In dc2
, you must also deploy a ProxyDefaults
CRD that sets the mesh gateway
mode. Use kubectl
and the proxy-defaults.yaml
file you created earlier to
apply the CRD to dc2
.
Verify the datacenters are connected
To verify that the Consul datacenters are connected and WAN federated, you can
use kubectl
to execute a Consul CLI command to query for a list of servers in
the WAN gossip pool. All the servers, from both datacenters, should be listed.
Deploy microservices
Now that you have two connected Consul datacenters, you can deploy a service in
each using kubectl
.
Deploy the static-client
service
The static-client
service in this tutorial represents a frontend service, for
example a website.
Change contexts to communicate with Consul datacenter dc1
.
The service definition includes some Consul-specific annotations:
"consul.hashicorp.com/connect-inject": "true"
- ensures that the service is deployed into the Consul service mesh with a sidecar proxy and automatically registered in the Consul catalog."consul.hashicorp.com/transparent-proxy": "false"
- use explicit upstreams instead of transparent proxy to connect todc2
. You cannot use transparent proxy to dial services across federated Kubernetes clusters. This example disables transparent proxy for all upstream connections. You can omit this annotation to mix and match upstream connections with both transparent proxy and explicit upstreams."consul.hashicorp.com/connect-service-upstreams": "static-server:8080:dc2"
- directs requests to thestatic-server
service indc2
using an explicit upstream. You can accessstatic-server
on port 8080.
First, create a yaml file, static-client.yaml
, to define the static-client
service.
The "consul.hashicorp.com/connect-inject": "true"
annotation causes Consul to
deploy a sidecar proxy alongside the static-client
service. The sidecar proxy
can both accept and establish connections using Consul.
Now, deploy the static-client
service into Consul datacenter dc1
.
Use kubectl
to verify that the pod deployed, and is running successfully.
Add a service intention
In order for the static-client
service to be able to communicate with the
upstream static-server
that is in a different datacenter, you must define a
ServiceIntentions
config entry that allows communication between the two services.
Create a yaml file, service-intentions.yaml
, to contain the ServiceIntentions
CRD YAML.
Use kubectl
to apply the intention.
Deploy the static-server
service
The static-server
service in this tutorial represents a backend service, for
example, a database.
Change contexts to communicate with Consul datacenter dc2
.
First, create a yaml file, static-server.yaml
, to define the static-server
service. Note, the static-server
service also includes the consul.hashicorp.com/connect-inject
annotation.
Now, deploy the static-server
service.
Use kubectl
to check that the pod deployed, and is running successfully.
Discover services across Kubernetes clusters
Servers participate in WAN gossip to share membership information, which allows servers to perform cross datacenter requests. To secure these requests, the WAN gossip is sent through the mesh gateways which encrypt the communication with mTLS. The requests include service queries.
To discover services across your Kubernetes clusters, you can use the Consul UI or CLI to query the available services.
First, connect to one of the servers in Consul datacenter dc2
.
Use kubectl
to execute a Consul CLI command that retrieves a list of all services
in the other datacenter, dc1
. The output will confirm that you are able to request
service information from the WAN connected datacenter.
Route service communication across Kubernetes clusters
Finally, verify that communication is routed through the mesh gateways. Use
curl
to verify that the server in dc2
can retrieve data from the client in dc1
.
Connect to the Kubernetes cluster where Consul datacenter dc1
is running.
Using kubectl
to connect to the client and request data from the server via
the proxy.
Clean up
Connect to Kubernetes cluster for dc2
.
Remove static-server
from dc2
.
Remove proxy-defaults
from dc2
.
Delete the Helm release of Consul in dc2
.
Note You can fully uninstall Consul by removing the volume claims and secrets.
Connect to Kubernetes cluster for dc1
.
Remove static-client
from dc1
.
Remove proxy-defaults
from dc1
.
Delete the Helm release of Consul in dc1
.
Extended concepts
Now that you have securely connected services across multiple Kubernetes clusters with mesh gateways, you can extend the feature for service fail-over, blue/green deployments, and canary testing.
You can deploy multiple instances of the same services across multiple production Kubernetes clusters. When services become unavailable in one cluster, you can route traffic to healthy instances in another cluster.
With the L7 routing and splitting features, you can test service upgrades in stages across multiple Kubernetes clusters.
Next steps
In this tutorial, you enabled two services in the Consul service mesh in separate Kubernetes clusters to securely communicate with each other over mesh gateways. You also secured WAN gossip server communication by routing traffic through the mesh gateways.
If you are ready to deploy Kubernetes into production, review the Reference Architecture and Deployment Guide.