Cluster Peering on Kubernetes
Cluster peering is currently in beta: Functionality associated
with cluster peering is subject to change. You should never use the beta release in secure environments or production scenarios. Features in
beta may have performance issues, scaling issues, and limited support.
Cluster peering is not currently available in the HCP Consul offering.
To establish a cluster peering connection on Kubernetes, you need to enable the feature in the Helm chart and create custom resource definitions (CRDs) for each side of the peering.
The following CRDs are used to create and manage a peering connection:
PeeringAcceptor
: Generates a peering token and accepts an incoming peering connection.PeeringDialer
: Uses a peering token to make an outbound peering connection with the cluster that generated the token.
Prerequisites
You must implement the following requirements to create and use cluster peering connections with Kubernetes:
- Consul version 1.13.1 or later
- At least two Kubernetes clusters
- The installation must be running on Consul on Kubernetes version 0.47.1 or later
Prepare for install
After provisioning a Kubernetes cluster and setting up your kubeconfig file to manage access to multiple Kubernetes clusters, export the Kubernetes context names for future use with
kubectl
. For more information on how to use kubeconfig and contexts, refer to Configure access to multiple clusters on the Kubernetes documentation website.You can use the following methods to get the context names for your clusters:
- Issue the
kubectl config current-context
command to get the context for the cluster you are currently in. - Issue the
kubectl config get-contexts
command to get all configured contexts in your kubeconfig file.
- Issue the
To establish cluster peering through Kubernetes, create a
values.yaml
file with the following Helm values.With these values, the servers in each cluster will be exposed over a Kubernetes Load balancer service. This service can be customized using
server.exposeService
.When generating a peering token from one of the clusters, Consul uses the address(es) of the load balancer in the peering token so that the peering stream goes through the load balancer in front of the servers. For customizing the addresses used in the peering token, refer to
global.peering.tokenGeneration
.values.yaml
Install Consul on Kubernetes
Install Consul on Kubernetes on each Kubernetes cluster by applying
values.yaml
using the Helm CLI.Install Consul on Kubernetes on
cluster-01
Install Consul on Kubernetes on
cluster-02
Create a peering token
To peer Kubernetes clusters running Consul, you need to create a peering token and share it with the other cluster. As part of the peering process, the peer names for each respective cluster within the peering are established by using the metadata.name
values for the PeeringAcceptor
and PeeringDialer
CRDs.
In
cluster-01
, create thePeeringAcceptor
custom resource.acceptor.yamlApply the
PeeringAcceptor
resource to the first cluster.Save your peering token so that you can export it to the other cluster.
Establish a peering connection between clusters
Apply the peering token to the second cluster.
In
cluster-02
, create thePeeringDialer
custom resource.dialer.yamlApply the
PeeringDialer
resource to the second cluster.
Export services between clusters
For the service in "cluster-02" that you want to export, add the following annotation to your service's pods.
backend.yamlIn
cluster-02
, create anExportedServices
custom resource.exportedsvc.yamlApply the service file and the
ExportedServices
resource for the second cluster.
Authorize services for peers
Create service intentions for the second cluster.
intention.ymlApply the intentions to the second cluster.
For the services in
cluster-01
that you want to access the "backend," add the following annotations to the service file. To dial the upstream service from an application, ensure that the requests are sent to the correct DNS name as specified in Service Virtual IP Lookups.frontend.yamlApply the service file to the first cluster.
Run the following command in
frontend
and check the output to confirm that you peered your clusters successfully.
End a peering connection
To end a peering connection, delete both the PeeringAcceptor
and PeeringDialer
resources.
To confirm that you deleted your peering connection, in cluster-01
, query the /health
HTTP endpoint. The peered services should no longer appear.
Recreate or reset a peering connection
To recreate or reset the peering connection, you need to generate a new peering token on the cluster where you created the PeeringAcceptor
(in this example, cluster-01
).
You can do this by creating or updating the annotation
consul.hashicorp.com/peering-version
on thePeeringAcceptor
. If the annotation already exists, update its value to a version that is higher.acceptor.ymlOnce you have done this, repeat the steps in the peering process. This includes saving your peering token so that you can export it to the other cluster. This will re-establish peering with the updated token.
Note: A new peering token is only generated upon manually setting and updating the value of the annotation consul.hashicorp.com/peering-version
. Creating a new token will cause the previous token to expire.