Secure Consul and Registered Services on Kubernetes
When operating pre-production environments it is common to initialize an environment without security features enabled, in order make the development process more transparent. Cleartext network traffic can be a useful debugging tool.
To secure an existing environment, or create an environment that is secured from the beginning, you can apply security controls using the official Consul Helm chart.
In this tutorial, you will:
- Review the types of Consul service mesh traffic
- Install an unsecured Consul service mesh on Kubernetes for development or debugging
- Verify that gossip encryption, TLS, and ACLs are not enabled
- Upgrade the installation to enable gossip encryption, TLS, and ACLs
- Verify that gossip encryption, TLS, and ACLs are enabled
- Deploy two example services to the service mesh
- Configure zero-trust networking using Consul intentions
Intended audience
admins
with privileged permissions to administer local, pre-production, and production clustersdevelopers
with privileged permissions to administer at least local or pre-production clusterscontributors
who wish to contribute to HashiCorp open source projects
Prerequisites
To complete this tutorial in your own environment you will need the following:
- Consul
- consul-helm
- helm
- kubectl (CLI)
- Administrative access to an active Kubernetes cluster
This tutorial was tested using:
Types of Consul service mesh traffic
The diagram below depicts the conceptual model of an active Consul service mesh on Kubernetes. The number of nodes varies based on your configuration, but the fundamental architecture and communication flows are represented.
- Consul uses a
gossip protocol
to manage membership and broadcast messages to the service mesh. - Consul uses the remote procedure call (
RPC
) pattern for communication between client and server nodes. Each server provides an HTTP API that supports read and write operations on the catalog which tracks the status of nodes, services, and other state information. - Consul uses Access Control Lists (
ACLs
) to secure the UI, API, CLI, service communications, and agent communications. At the core, ACLs operate by grouping rules into policies, then associating one or more policies with a token. - Consul uses
intentions
to control which services may establish connections. Intentions can be managed via the API, CLI, or UI.
Create a Kubernetes cluster (optional)
Run a local Kubernetes cluster using kind.
Kind will configure your kubectl
tool to interact with your local K8s cluster.
Verify that kubectl
is configured to interact with the K8s control plane running
on your local machine (127.0.0.1
).
Install an unsecured Consul service mesh
Create an unsecured config file
Now you will create an unsecured Consul Helm configuration file called dc1.yaml
.
Consul server containers in Kubernetes do not run as root by default, and the
Alpine Linux image they are based off of intentionally does not include tools
like tcpdump
. In order to modify the running container, you will need to add
Consul Helm configuration that overrides the default, and does run the Consul
agent containers as root. By doing that, you will be able to run apk
and add
the utilities you need for this tutorial. Specifically, you will be adding a
securityContext
stanza under the server
region as shown in the following
YAML example.
Create the Consul Helm configuration file.
Install Consul in your cluster
You can now deploy a complete Consul datacenter in your Kubernetes cluster using the official Consul Helm chart or the Consul K8S CLI.
Helm chart preparation
Consul on Kubernetes provides a Helm chart to deploy a Consul datacenter on Kubernetes in a highly customized configuration. Review the docs on Helm chart Configuration to learn more about the available configurations.
Add the official HashiCorp Helm repository and download the latest official consul-helm chart now.
Verify chart version
To ensure you have version 0.43.0
of the Helm chart, search your local repo.
Install Consul in your cluster
Now, issue the helm install
command. The following command specifies that the
installation should:
- Use the custom values file
dc1.yaml
you created earlier - Use the
hashicorp/consul
chart - Set your Consul installation name to
consul
- Create Consul resources in the
consul
namespace - Use
consul-helm
chart version0.43.0
Verify installation
Use kubectl get pods
to verify your installation.
Once all pods have a status of Running
, enter CTRL-C
to stop the watch.
At this point, the Consul service mesh has been installed in the Kubernetes cluster, but no security features have been enabled. This means that:
- All gossip traffic between agents is in clear text
- All RPC communications between agents is in clear text
- There are no Access Controls in place
- No intentions have been defined
Verify security not enabled (optional)
To verify that network traffic is in cleartext inspect it.
View server traffic
To view network traffic, connect to the consul-server-0
container with
kubectl
, and observe its traffic using the tcpdump
program.
The container images used by the Consul Helm chart are lightweight alpine images.
They ship with limited tools. Issue the following commands to install tcpdump
:
Now start tcpdump
to view traffic to the server container. This script limits
the port range to the range of ports used by Consul.
Traffic occurs rapidly. The following output is an abbreviated example.
Inspect the output and observe that the traffic is in cleartext. Note the UDP operations, these entries are the gossip protocol at work. This proves that gossip encryption is not enabled.
Next, issue a Consul CLI command to prove two things:
- RPC traffic is currently unencrypted
- ACLs are not enabled
If the previous tcpdump
process is still active, type CTRL-C to end it. This
slightly modified version of the tcpdump
command writes results to a log file.
Start it now so that you can grep for interesting log entries later.
Traffic is now being captured in a log file rather than being output to the terminal.
Next, generate some Consul traffic in a different terminal using kubectl
to exec
a command against the Consul CLI on the client container. This traffic
will originate from the client to the server, and will prove that RPC traffic is
in cleartext.
The command succeeds, but notice that you did not pass a -token
option nor did
you set the CONSUL_HTTP_TOKEN
environment variable. One or the other is required
when ACLs are enabled. This proves that ACLs are not enabled.
Now, from the terminal session on the server container type CTRL-C to stop
the tcpdump
process, and then search the log file for the CLI operation with
the following command:
Note that you are able to inspect the RPC operation in cleartext. This proves that RPC traffic is not encrypted.
Exit the terminal session on the server container.
Upgrade to a secured Consul service mesh
Next, upgrade your Consul service mesh installation to enable gossip encryption, TLS, and ACLs. You can upgrade the service mesh by updating your custom configuration yaml file settings, and then passing the new configuration via Helm or Consul K8s.
The following command will generate the secure-dc1.yaml
file with gossip
encryption, TLS, and ACLs enabled.
Notice that the file now includes a gossipEncryption
stanza.
TLS is enabled by this stanza.
ACLs are enabled by this stanza.
Upgrade the deployment
The config file for the chart has been properly configured, and all necessary secrets have been registered with Kubernetes. Execute the following command to upgrade the installation with these changes. The upgrade may take a minute or two to complete.
Verify the upgrade
Use kubectl get pods
to verify your installation.
Once all pods have a status of Running
you can proceed to the next step.
Verify security enabled
Now, you can verify that gossip encryption and TLS are enabled, and that ACLs are being enforced. Otherwise, you can skip ahead to Configuring Consul intentions.
In a separate terminal, forward port 8501 from the Consul server on Kubernetes so that you can interact with the Consul CLI from the development host.
Note
with TLS enabled, Consul uses port 8501 instead of 8500 by default.
Set the CONSUL_HTTP_ADDR
environment variable to use the HTTPS address/port on
the development host.
Export the CA file from Kubernetes so that you can pass it to the CLI.
Now, execute consul members
and provide Consul with the ca-file option to
verify TLS connections. You will observe a list of all members of the service mesh.
The actions you performed in this section of the tutorial prove that TLS is being enforced.
Set an ACL token
Now, try launching a debug session. The command will fail.
The 403 response proves that ACLs are being enforced. You have not yet supplied
an ACL token, so the command fails. The consul members
command worked because
consul-helm created an anonymous token and set the following policy for it:
Note
This policy is necessary for DNS, more specifically, so that the
Kubernetes DNS server can make DNS queries against Consul. Queries like
consul members
end up working without a client specifically providing a token.
If you don't want to use Consul DNS, you could disable it in the chart by
setting dns.enabled
to false. This will configure the ACL bootstrapping job to
not create a policy for the anonymous token. However, if you want to use Consul
DNS, this policy is required.
In order to perform operations that require a higher level of authority, you
must provide a token with the necessary permissions. In this tutorial you will
set the CONSUL_HTTP_TOKEN
environment variable.
Note
You could alternately pass the token as the -token
option to your
CLI commands.
The consul-helm chart created several secrets during the initialization process and registered them with Kubernetes. For a list of all Kubernetes secrets issue the following command:
Notice that one of the secrets is named consul-bootstrap-acl-token
.
To view the Kubernetes secret, execute the following command:
This secret contains the Consul ACL bootstrap token. The bootstrap token is a full access token that can perform any operation in the service mesh. In a production scenario, you should avoid using the bootstrap token, and instead create tokens with specific permissions. In this tutorial, you will use it for convenience.
The value of interest is the string in the data stanza's token field. That value is a base64 encoded string that contains the bootstrap token generated during the consul-helm ACL init process.
For this tutorial you can retrieve the value, decode it, and set it to the CONSUL_HTTP_TOKEN
environment variable with the following command.
Start a debug session again with an ACL token set.
The command succeeds. This proves that ACLs are being enforced. Type CTRL-C to end the debug session in the terminal.
Verify that network traffic is encrypted
Now that you have proven that ACLs are enabled, and that TLS verification is being enforced, you will prove that all gossip and RPC traffic is encrypted.
Start a shell session on the server container.
Since the containers were recycled during the helm upgrade, you will
have to add tcpdump
again.
Next, start tcpdump
and observe the gossip traffic.
Notice that none of the traffic is in cleartext, as it was before. This proves that gossip traffic is now encrypted.
Type CTRL-C to stop the tcpdump
session. Use the following command
to restart tcpdump
and pipe the results to a log file so that
you can search for cleartext RPC traffic.
From a different terminal, list services with the Consul cli.
There will be only one service - consul
.
Switch back to server, type CTRL-C to stop tcpdump
, and grep log for an RPC entry
Notice that no rows were found this time. This proves that RCP traffic is now encrypted. Exit the terminal session on the server container.
Configure Consul intentions
Now, deploy two sample services, and manage them using Consul intentions.
Deploy example services
To simulate an active environment you will deploy a client, and an upstream backend
service. First, issue the following command to create a file named server.yaml
that will be used to create an http echo server on Kubernetes:
Next, deploy the sample backend service.
Next, create a file named client.yaml
that defines the sample client service.
Next, deploy the sample client.
Finally, ensure all pods/containers having a status of Running
before proceeding to the next section.
With manageSystemACLs
set to true, the Consul Helm chart will, by default,
create a deny all
intention. This means that services will not be able to communicate
until an explicit intention is defined that allows them to communicate.
Issue the following command to validate that the default deny all
intention is enforced.
The command is unable to resolve the service name. This proves the intention is enforced.
Run the following script to create a YAML file containingallow
ServiceIntention
for client to server traffic.
Use kubectl
to apply the ServiceIntention
to the service mesh.
Finally, validate the intention allows traffic from the client to the server. If this fails, wait a few seconds for the intention to be applied, and try again.
This proves the intention is allowing traffic from the client to the server.
Next steps
In this tutorial, you enabled Consul security controls for Consul on Kubernetes using the Consul Helm chart.
Specifically, you:
- Installed an unsecured Consul service mesh on Kubernetes for development or debugging
- Verified that gossip encryption, TLS, and ACLs were not enabled
- Upgraded the installation to enable gossip encryption, TLS, and ACLs
- Verified that gossip encryption, TLS, and ACLs were now enabled
- Deployed two example services to the service mesh
- Configured zero-trust networking using Consul intentions
Next, consider reviewing our L7 observability tutorial to learn more techniques for monitoring or debugging Consul on Kubernetes.