Consul
Access services in your service mesh
In the previous tutorial, you updated your Consul datacenter with a service mesh that secures service-to-service communication for your application.
In this tutorial, you will learn how to add a Consul API Gateway to secure external network access to applications and services running in your Consul service mesh.
In this tutorial, you will:
- Add a new node to your Consul datacenter to host the API gateway
- Generate configurations and routes for the API gateway
- Start the API gateway
- Use service intentions to allow access to applications inside the service mesh
Note
Because this tutorial is part of the Get Started on VMs tutorial collection, the following workflow was designed for education and demonstration. It uses scripts to generate agent configurations and requires you to execute commands manually on different nodes. If you are setting up a production environment you should codify and automate the installation and deployment process according to your infrastructure and networking needs. Refer to the VM production patterns tutorial collection for Consul production deployment considerations and best practices.
Tutorial scenario
This tutorial uses HashiCups, a demo coffee shop application made up of several microservices running on VMs.
At the beginning of the tutorial, you have a fully deployed Consul service mesh with Envoy sidecar proxies running alongside each service.
At the end of this tutorial, you will have Consul API gateway running and configured to permit access to the HashiCups application over port 8443. You will also generate an SSL certificate to be exposed by the application.
Prerequisites
If you completed the previous tutorial, the infrastructure is already in place with all prerequisites.
Log into the bastion host VM
Terraform output provides a series of useful information, including bastion host IP address.
Log into the bastion host with SSH.
$ ssh -i certs/id_rsa.pem admin@`terraform output -raw ip_bastion`
Verify Envoy binary
Verify Envoy is installed on the API gateway node.
Log into the API gateway from the bastion host.
$ ssh -i certs/id_rsa gateway-api-0
Verify the Envoy binary is installed.
$ envoy --version
envoy version: 688c4bbe47f4d05bb8ed268f5172bb026cf03242/1.31.5/Clean/RELEASE/BoringSSL
Use the Envoy and Consul Client Agent compatibility matrix to verify the Envoy version is compatible with the Consul version.
Return to the bastion host by exiting the SSH session.
$ exit
logout
Connection to gateway-api-0 closed.
admin@bastion:~$
Repeat the steps for all VMs you want to add to the Consul service mesh.
Configure environment
The tutorial creates all the files in a destination folder. Export the path where you wish to create the configuration files for the scenario.
$ export OUTPUT_FOLDER=/home/admin/assets/scenario/conf/
Make sure the folder exists.
$ mkdir -p ${OUTPUT_FOLDER}
Source the env-scenario.env file to set the variables in the terminal session.
$ source assets/scenario/env-scenario.env
Configure the Consul CLI to interact with the Consul server.
$ export CONSUL_HTTP_ADDR="https://consul-server-0:8443" \
export CONSUL_HTTP_SSL=true \
export CONSUL_CACERT="${OUTPUT_FOLDER}secrets/consul-agent-ca.pem" \
export CONSUL_TLS_SERVER_NAME="server.${CONSUL_DATACENTER}.${CONSUL_DOMAIN}" \
export CONSUL_HTTP_TOKEN=`cat ${OUTPUT_FOLDER}secrets/acl-token-bootstrap.json | jq -r ".SecretID"`
Verify your Consul CLI can interact with your Consul server.
$ consul members
Node Address Status Type Build Protocol DC Partition Segment
consul-server-0 172.18.0.7:8301 alive server 1.20.2 2 dc1 default <all>
hashicups-api-0 172.18.0.2:8301 alive client 1.20.2 2 dc1 default <default>
hashicups-db-0 172.18.0.4:8301 alive client 1.20.2 2 dc1 default <default>
hashicups-frontend-0 172.18.0.3:8301 alive client 1.20.2 2 dc1 default <default>
hashicups-nginx-0 172.18.0.11:8301 alive client 1.20.2 2 dc1 default <default>
Add API gateway node to Consul datacenter
Consul API Gateway uses the same components as the rest of the service mesh client nodes to join the Consul datacenter. That means you need a Consul agent running on the node and an Envoy proxy instance to act as a proxy for the services you want to expose outside your service mesh.
Generate Consul configuration for API gateway
First, define the Consul node name.
$ export NODE_NAME="gateway-api-0"
Then, generate the Consul configuration for the API Gateway node.
$ ~/ops/scenarios/00_base_scenario_files/supporting_scripts/generate_consul_client_config.sh
[generate_consul_client_config.sh] - - Generate configuration for [gateway-api-0]
+ --------------------
| Parameter Check
+ --------------------
[WARN] Script is running with the following values:
[WARN] ----------
[WARN] CONSUL_DATACENTER = dc1
[WARN] CONSUL_DOMAIN = consul
[WARN] CONSUL_RETRY_JOIN = consul-server-0
[WARN] CONSUL_CONFIG_DIR = /etc/consul.d/
[WARN] CONSUL_DATA_DIR = /opt/consul/
[WARN] ----------
[WARN] Generated configuration will be placed under:
[WARN] OUTPUT_FOLDER = ~/assets/scenario/conf/
[WARN] ----------
+ --------------------
| Generate configuration for Consul agent gateway-api-0
+ --------------------
- Cleaning folder from pre-existing files
[WARN] Removing pre-existing configuration in ~/assets/scenario/conf/
- Generate folder structure
- Copy available configuration
- Generate configuration files
- Validate configuration for gateway-api-0
To complete Consul agent configuration, you need to set up tokens for the client. For this tutorial, you are using the bootstrap token. We recommend using more restrictive tokens for your Consul client agents in production.
$ tee ${OUTPUT_FOLDER}${NODE_NAME}/agent-acl-tokens.hcl > /dev/null << EOF
acl {
tokens {
agent = "${CONSUL_HTTP_TOKEN}"
default = "${CONSUL_HTTP_TOKEN}"
config_file_service_registration = "${CONSUL_HTTP_TOKEN}"
}
}
EOF
Once you have generated your configuration files, your directory should look like the following:
$ tree ${OUTPUT_FOLDER}gateway-api-0
~/assets/scenario/conf/gateway-api-0
|-- agent-acl-tokens.hcl
|-- agent-gossip-encryption.hcl
|-- consul-agent-ca.pem
`-- consul.hcl
1 directory, 4 files
The scripts generated multiple configuration files to separate the configuration so that it is easier to read and tune them for your environment.
The following are the generated files and a description of their purpose:
- The
agent-acl-tokens.hclfile contains tokens for the Consul agent. - The
agent-gossip-encryption.hclfile configures gossip encryption. - The
consul-agent-ca.pemfile is the public certificate for Consul CA. - The
consul.hclfile contains node specific configuration and it is needed, with this specific name, if you want to configure Consul as a systemd daemon.
Refer to the agent configuration documentation to interpret the files or to modify them when applying them to your environment.
After the script generates the client configuration, you will copy these files into the API gateway node.
First, configure the Consul configuration directory.
$ export CONSUL_REMOTE_CONFIG_DIR=/etc/consul.d/
Then, use rsync to copy the service configuration file into the remote node.
$ rsync -av --no-g --no-t --no-p \
-e "ssh -i ~/certs/id_rsa" \
${OUTPUT_FOLDER}gateway-api-0/ \
gateway-api-0:${CONSUL_REMOTE_CONFIG_DIR}
The output is similar to the following:
sending incremental file list
./
agent-acl-tokens.hcl
agent-gossip-encryption.hcl
consul-agent-ca.pem
consul.hcl
sent 3,152 bytes received 95 bytes 6,494.00 bytes/sec
total size is 2,799 speedup is 0.86
Start Consul on API GW
Log into gateway-api-0 from the bastion host.
$ ssh -i certs/id_rsa gateway-api-0
##..
admin@gateway-api-0:~
Define the Consul configuration and data directories.
$ export CONSUL_CONFIG_DIR=/etc/consul.d/ \
export CONSUL_DATA_DIR=/opt/consul/
Ensure your user has write permission to the Consul data directory.
$ sudo chmod g+w ${CONSUL_DATA_DIR}
Finally, start the Consul server process.
$ consul agent -config-dir=${CONSUL_CONFIG_DIR} > /tmp/consul-client.log 2>&1 &
The process starts in background to avoid a lock on the terminal.
Access the Consul server log in the /tmp/consul-client.log file.
Exit the SSH session to return to the bastion host.
$ exit
logout
Connection to gateway-api-0 closed.
admin@bastion:~$
Use the consul members command to verify that the Consul API Gateway successfully joined the datacenter.
$ consul members
Node Address Status Type Build Protocol DC Partition Segment
consul-server-0 172.18.0.7:8301 alive server 1.20.2 2 dc1 default <all>
gateway-api-0 172.18.0.5:8301 alive client 1.20.2 2 dc1 default <default>
hashicups-api-0 172.18.0.2:8301 alive client 1.20.2 2 dc1 default <default>
hashicups-db-0 172.18.0.4:8301 alive client 1.20.2 2 dc1 default <default>
hashicups-frontend-0 172.18.0.3:8301 alive client 1.20.2 2 dc1 default <default>
hashicups-nginx-0 172.18.0.11:8301 alive client 1.20.2 2 dc1 default <default>
Generate API Gateway rules
Now that the Consul agent for the API Gateway successfully joined the datacenter, you can configure how it handles incoming traffic.
Consul API Gateway is configured using Consul's global configuration entries so that you can configure it from a remote node. For this scenario, you will use the bastion host VM to generate, store, and apply the configuration.
To configure a Consul API Gateway you need the following:
- A TLS certificate used by the API Gateway to secure connections to the mesh services.
- An API Gateway configuration entry that defines the listeners the gateway exposes externally and the certificates associated with them.
Generate API Gateway certificate
You can create the certificate using an internal or public CA so your services can be compliant with your internal standards.
For this tutorial, you will use openssl to generate a valid certificate for the HashiCups application.
Define the certificate common name.
$ export COMMON_NAME="hashicups.hashicorp.com"
Create a configuration file for openssl.
$ tee ${OUTPUT_FOLDER}gateway-api-ca-config.cnf > /dev/null << EOF
[req]
default_bit = 4096
distinguished_name = req_distinguished_name
prompt = no
[req_distinguished_name]
countryName = US
stateOrProvinceName = California
localityName = San Francisco
organizationName = HashiCorp
commonName = ${COMMON_NAME}
EOF
Generate a private key.
$ openssl genrsa -out ${OUTPUT_FOLDER}gateway-api-cert.key 4096 2>/dev/null
Create a certificate signing request.
$ openssl req -new \
-key ${OUTPUT_FOLDER}gateway-api-cert.key \
-out ${OUTPUT_FOLDER}gateway-api-csr.csr \
-config ${OUTPUT_FOLDER}gateway-api-ca-config.cnf 2>/dev/null
Finally, sign the certificate and save it to a crt file.
$ openssl x509 -req -days 3650 \
-in ${OUTPUT_FOLDER}gateway-api-csr.csr \
-signkey ${OUTPUT_FOLDER}gateway-api-cert.key \
-out ${OUTPUT_FOLDER}gateway-api-cert.crt 2>/dev/null
The file system certificate is the most secure method to use a TLS certificate for Consul API Gateway on VMs because it references a local file path instead of including sensitive information in the configuration entry itself. File system certificates also include a file system watch that implements certificate and key changes without restarting the gateway.
Populate the configuration file with the desired path for the certificate and key.
$ tee ${OUTPUT_FOLDER}config-gateway-api-fs-certificate.hcl > /dev/null << EOF
Kind = "file-system-certificate"
Name = "api-gw-certificate"
Certificate = "/etc/consul.d/gateway-api-cert.crt"
PrivateKey = "/etc/consul.d/gateway-api-cert.key"
EOF
Copy the certificate to the remote node.
$ rsync -av \
-e "ssh -i ~/certs/id_rsa" \
${OUTPUT_FOLDER}gateway-api-cert* \
gateway-api-0:/etc/consul.d/
The output is similar to the following:
sending incremental file list
gateway-api-cert.crt
gateway-api-cert.key
sent 5,390 bytes received 54 bytes 10,888.00 bytes/sec
total size is 5,207 speedup is 0.96
Generate API Gateway configuration
The following API Gateway configuration entry includes listener configuration and a reference to the TLS certificate that the gateway exposes.
$ tee ${OUTPUT_FOLDER}config-gateway-api-fs.hcl > /dev/null << EOF
Kind = "api-gateway"
Name = "gateway-api"
// Each listener configures a port which can be used to access the Consul cluster
Listeners = [
{
Port = 8443
Name = "api-gw-listener"
Protocol = "http"
TLS = {
Certificates = [
{
Kind = "file-system-certificate"
Name = "api-gw-certificate"
}
]
}
}
]
EOF
Apply the configuration to Consul datacenter
You can now apply the configuration entries to the Consul datacenter.
$ consul config write ${OUTPUT_FOLDER}config-gateway-api-fs.hcl; \
consul config write ${OUTPUT_FOLDER}config-gateway-api-fs-certificate.hcl
The output is similar to the following:
Config entry written: api-gateway/gateway-api
Config entry written: file-system-certificate/api-gw-certificate
Start Consul API gateway
Now that you configured the API Gateway, start the Envoy process that serves external requests to activate API gateway operations.
Log into gateway-api-0 from the bastion host.
$ ssh -i certs/id_rsa gateway-api-0
##..
admin@gateway-api-0:~
Then, configure the token for the Envoy process.
$ export CONSUL_AGENT_TOKEN=`cat /etc/consul.d/agent-acl-tokens.hcl | grep agent | awk '{print $3}'| sed 's/"//g'`
Finally, start the Envoy sidecar proxy for the API gateway.
$ /usr/bin/consul connect envoy \
-gateway api \
-register \
-service gateway-api \
-token=${CONSUL_AGENT_TOKEN} \
-envoy-binary /usr/bin/envoy > /tmp/api-gw-proxy.log 2>&1 &
Exit the SSH session to return to the bastion host.
$ exit
logout
Connection to gateway-api-0 closed.
admin@bastion:~$
Apply route
At this point, Consul API Gateway is ready to serve requests, but you do not have a route configured to expose services in the mesh to traffic from external sources.
For this tutorial, you will expose the HashiCups application using the hashicups-nginx service as entry point.
Generate API Gateway route
From the bastion host, create a route to redirect ingress traffic to the hashicups-nginx service.
$ tee ${OUTPUT_FOLDER}config-gateway-api-http-route.hcl > /dev/null << EOF
Kind = "http-route"
Name = "hashicups-http-route"
// Rules define how requests will be routed
Rules = [
{
Matches = [
{
Path = {
Match = "prefix"
Value = "/"
}
}
]
Services = [
{
Name = "hashicups-nginx"
}
]
}
]
Parents = [
{
Kind = "api-gateway"
Name = "gateway-api"
SectionName = "api-gw-listener"
}
]
EOF
Then, apply the configuration to Consul.
$ consul config write ${OUTPUT_FOLDER}config-gateway-api-http-route.hcl
Config entry written: http-route/hashicups-http-route
Create a new intention for service access
To allow access to your NGINX service that serves the HashiCups application, create a service intention that allows traffic from gateway-api service to hashicups-nginx service.
First make sure the output folder exists.
$ mkdir -p ${OUTPUT_FOLDER}global
Then create the intention configuration file.
$ tee ${OUTPUT_FOLDER}global/intention-nginx.hcl > /dev/null << EOF
Kind = "service-intentions"
Name = "hashicups-nginx"
Sources = [
{
Name = "gateway-api"
Action = "allow"
}
]
EOF
After you create the configuration file for the intention, apply it.
$ consul config write ${OUTPUT_FOLDER}global/intention-nginx.hcl
Config entry written: service-intentions/hashicups-nginx
Verify API Gateway traffic to HashiCups
After you apply the route, you can access the HashiCups application from the Consul API gateway address.
First, retrieve the API Gateway address.
For this scenario, you can get the API Gateway public IP directly from the bastion host.
$ echo "https://`cat /etc/hosts | grep gateway-api-public | awk '{print $1}'`:8443"
The output will be similar to the following:
https://35.87.44.101:8443
Then, open the address in a browser.
You now have exposed HashiCups using Consul API Gateway and the application is TLS protected. Verify the certificate exposed by the application is the one you created earlier in this tutorial.
$ openssl s_client -connect \
`cat /etc/hosts | grep gateway-api-public | awk '{print $1}'`:8443 </dev/null 2>/dev/null | \
openssl x509 -inform pem -text
The output will be similar to the following:
Certificate:
Data:
Version: 1 (0x0)
Serial Number:
3a:eb:01:5b:72:8c:40:47:e3:8b:c5:49:a4:bb:2f:50:ff:97:3e:c8
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, ST = California, L = San Francisco, O = HashiCorp, CN = hashicups.hashicorp.com
Validity
Not Before: Jun 1 09:36:12 2023 GMT
Not After : May 29 09:36:12 2033 GMT
Subject: C = US, ST = California, L = San Francisco, O = HashiCorp, CN = hashicups.hashicorp.com
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (4096 bit)
Modulus:
## ...
Exponent: 65537 (0x10001)
Signature Algorithm: sha256WithRSAEncryption
## ...
-----BEGIN CERTIFICATE-----
## ...
-----END CERTIFICATE-----
The public access to your HashiCups application instance is now TLS secured.
Remove direct access to hashicups-nginx
At this point, the HashiCups application is still accessible using the insecure NGINX endpoint.
Ensure that the hashicups-nginx service only serves local requests.
$ ssh -i ~/certs/id_rsa hashicups-nginx-0 "bash -c \
'bash ./start_service.sh start --local'"
The output is similar to the following:
# ------------------------------------ #
| HashiCups NGINX service |
# ------------------------------------ #
Stop pre-existing instances.
START - Start services on all interfaces.
START LOCAL - Start services on local interface.
Start service instance.
Service started on local insteface
upstream frontend_upstream {
server localhost:3000;
}
upstream api_upstream {
server localhost:8081;
}
Starting NGINX...attempt 1
The insecure endpoint is not available anymore. The only secure access to your application is through the Consul API Gateway.
Next steps
Access to your service mesh is now secure. The HashiCups application is accessible only using the Consul API Gateway, the internal communication between services that make up the application is fully secured, and you have a custom certificate for the externally exposed services.
If you want to stop at this tutorial, you can destroy the infrastructure now.
From the ./self-managed/infrastruture/aws folder of the repository, use
terraform to destroy the infrastructure.
$ terraform destroy --auto-approve
In the next tutorial, you will learn how to monitor the services in your Consul service mesh using the Grafana suite.
For more information about the topics covered in this tutorial, refer to the following resources: