Secure applications with mutual TLS

23min
|
Vault

Applications that communicate without encryption expose data in transit to anyone on the network. Without authentication, any service that supports the application can potentially connect to and interact with any other service.

Challenge

Applications running in containers or on virtual machines often communicate over internal networks that seem private, but do not actually protect data confidentiality or integrity. Without encryption, anyone with network access can read the data passing between services.

If your services do not use mutual authentication, a compromised service can impersonate another service to exfiltrate data, cause denial of service, or pivot to access other services.

Solution

You can enable mutual TLS authentication (mTLS) and manage certificate lifecycle with Vault through its PKI secrets engine. The PKI secrets engine generates X.509 certificates from a trusted Certificate Authority (CA) that you control.

You can then deploy Vault Agent with each service to automatically manage the service certificate requests and renewals. The services secured with mTLS verify identification, and encrypt all data in transit to preserve confidentiality and integrity.

Securing your services with mTLS provides the following benefits:

Encryption: TLS protects data in transit.
Mutual authentication: Both parties verify identity using certificates.
Automated lifecycle: Vault Agent manages the certificate lifecycle, and renews certificates before they expire.
Least privilege: Each service can request certificates for its own common name. Vault prevents requests for certificates matching any other name outside of the allowed names in PKI secrets engine role configuration.

Prerequisites

You need the following to complete the scenario in this tutorial.

Terminal access and familiarity with the command line.
Docker Desktop installed.
curl (or similar tool) for API requests.
Git to clone the sandbox environment repository.
jq for prettifying API responses.

Scenario introduction

In this scenario, you assume the role of Danielle, a developer at the fictional company HashiCups. Danielle is working on a cup printing platform based on two microservices, called CupFinisher.

The CupFinisher platform is an internal system to handle orders for cups with custom applied finishes. The platform consists of an order service responsible for coordinating customer orders, and an inventory service that exposes an API the order service can use to learn about inventory status for special finishes and patterns.

The diagram in Insecure services shows the initial CupFinisher microservice architecture in its insecure state, followed by details about security issues this configuration exposes.

Click Secure services to show the same architecture using the Vault PKI secrets engine and Vault Agent to secure services with mTLS. You can learn how mTLS addresses the issues detailed in the insecure configuration.

Insecure microservices

In the insecure deployment, services do not use encryption for data in transit, providing a stream of plaintext information that an attacker can intercept. Further, services cannot reliably verify identities, enabling an attacker to masquerade as a legitimate service.
A compromised rogue service can try to intercept the unsecured communications between services and further compromise data confidentiality or integrity. The attacker can use secrets leaked in plaintext to pivot and gain additional unauthorized access.
A compromised rogue service can try to directly interact with other services to gain additional privileges or further exfiltrate data.

Vault and Vault Agent secure microservices

The services do not directly interact with the Vault server. Instead, each service communicates with its own Vault Agent instance. Vault Agent handles authentication and credential management with Vault on behalf of the service.
In this scenario, Vault Agent authenticates with Vault to receive a token. The Vault token enables Vault Agent to manage the lifecycle for credentials and secrets received from Vault.
The Vault Agent for each service authenticates to Vault using the AppRole auth method, and receives a token with capabilities that allow requesting and renewing TLS certificates and private keys from the Vault PKI secrets engine.
Vault issues the certificates and keys for the services to the Vault Agent for each service.
Vault Agent uses templates to write the secrets to the appropriate files for use by the microservices. The services use the certificates and keys to enable mTLS. When Vault Agent automatically completes renewal, it signals services to restart and load the renewed TLS certificate and key.
With mTLS enabled, the rogue service cannot make arbitrary connections to the platform services even with a spoofed TLS certificate. mTLS encrypts the service data in transit, so the rogue service cannot be used to intercept communications between services.

The scenario consists of 4 phases that you work through in order using terminal sessions and command line tools.

You gain deeper understanding of the PKI secrets engine, Vault Agent, and securing microservices as you progress through each phase.

Phase 1: Observe insecure microservices: Deploy the CupFinisher microservices without mTLS. Use the rogue service container to observe how plaintext microservice API traffic is readable to anyone with network access, and how a compromised malicious service can steal data from the other microservices.
Phase 2: Explore configuration and policies: Discover the ACL policies, Vault PKI secrets engine feature configuration, and Vault Agent microservice sidecar configuration examples to understand how Vault helps secure the microservices with mTLS.

Tip

If you are not yet familiar with Vault Agent, check out Vault Agent and Vault Proxy quick start before continuing with this tutorial.

Phase 3: Deploy Vault and microservices secured with mTLS: Use Vault as a root and intermediate Certificate Authority, and use Vault Agent sidecars to manage TLS certificates for each service. After this change, network traffic gets encrypted, and malicious services get blocked due to certificate validation failure during the connection TLS handshake.
Phase 4: Stop services and clean up. Clean up artifacts created by following the tutorial steps. Summarize what you've learned and explore next steps.

Phase 1: Observe insecure microservices

In this phase, you deploy the CupFinisher microservices without TLS protection. The infrastructure includes a "rogue service" that attempts to access the inventory service, and that you can use to inspect network traffic between services.

Clone the tutorial repository and change into the project directory.
```
$ git clone \
    https://github.com/hashicorp-education/learn-vault-app-tls
```
The repository contains the Python microservice source code, Docker Compose files, Vault and Vault Agent configuration, and scripts to run each phase of the tutorial.
Change into the project directory.
```
$ cd learn-vault-app-tls
```

Deploy the insecure infrastructure.

$ docker compose \
    --file docker-compose.insecure.yml \
    up \
    --build \
    --detach

Example output:

...snip...
[+] up 8/8
 ✔ Image learn-vault-app-tls-rogue-service     Built     0.7s
 ✔ Image learn-vault-app-tls-order-service     Built     0.7s
 ✔ Image learn-vault-app-tls-inventory-service Built     0.7s
 ✔ Network cupfinisher                         Created   0.0s
 ✔ Container inventory-service                 Started   0.1s
 ✔ Container order-service                     Started   0.1s
 ✔ Container rogue-service                     Started   0.1s
 ✔ Container rogue-shell                       Started   0.1s

The deployment creates and builds images, then defines a network and starts these containers:

Service	Purpose	Insecure port	Secure port
order-service	Accepts print orders	8080	8443
inventory-service	Tracks cup stock levels	8081	8444
rogue-service	A compromised service attempts to gain additional access or pivot. This service executes Python code, and exits.	N/A	N/A
rogue-shell	Shell on the rogue service container with network packet capture tools	N/A	N/A

Check the health endpoint on the order service to ensure that it is available before continuing with the next step.
```
$ curl --silent http://127.0.0.1:8080/health | jq -r '.status'
ok
```
Before you proceed, make sure the status field value returns ok.
If the service does not return an ok status, check the container status and logs with docker ps and docker logs order-service.

Place a test order

Send an order request to the order service. The order service calls the inventory service to check stock before confirming.

$ curl \
    --silent \
    --request POST http://localhost:8080/order \
    --header 'Content-Type: application/json' \
    --data '{"design": "geometric-wave", "quantity": 2}' \
    | jq

An example response follows.

{
  "design": "geometric-wave",
  "design_name": "Geometric Wave",
  "quantity": 2,
  "remaining_stock": 22,
  "status": "confirmed"
}

The confirmed response status shows that you placed an order without any authentication or authorization to do so.

Check rogue service logs

The rogue service tries to access the inventory service with an automated attack script. Check its logs to learn if it obtained unauthenticated access.

$ docker logs rogue-service

Note

If you notice "Exiting in 30 seconds...", the attack is still in progress. Wait a moment and run the command again.

Example output:

...snip...
=======================================================
  *  Attack summary
=======================================================

  +----------------------------------------------------------+
  |                                                          |
  |  Attack results                                          |
  |                                                          |
  |  The inventory service has no protection:                |
  |                                                          |
  |    - No authentication required                          |
  |    - No encryption for data in transit                   |
  |    - Any service can access sensitive data               |
  |                                                          |
  +----------------------------------------------------------+

The rogue service retrieved inventory data without authentication. You can learn more about inventory levels from the complete output.

The insecure inventory service has no way to distinguish between legitimate requests from the order service and malicious requests from the rogue service.

Tip

The inventory service in this configuration is insecure, but the order service is also insecure in the same ways, and the rogue service could similarly attack the order service as well.

Capture and inspect network traffic

Connect to the rogue-shell container and capture and decode HTTP traffic between the CupFinisher services.

Connect to the rogue-shell container:
```
$ docker exec -it rogue-shell bash
```
Note
Inside the rogue-shell container, the prompt changes to a random hex string combined with the present working directory name and a # character, like efaccc588cc2:~#.

Run tshark to capture HTTP traffic to a JSON file.

The command options output specific HTTP field values like the request method, URI, and response body, and output to JSON in the file /tmp/capture.json.

$ tshark \
    -i eth0 \
    -f "tcp port 8080" \
    -Y http \
    -T json \
    -e http.request.method \
    -e http.request.uri \
    -e http.response.code \
    -e http.file_data \
    > /tmp/capture.json

Expected output from tshark:

Capturing on 'eth0'

Open a second terminal session, and change into the learn-vault-app-tls repository directory.
Use curl to place another order.
```
$ curl -X POST http://localhost:8080/order \
    -H 'Content-Type: application/json' \
    -d '{"design": "midnight-bloom", "quantity": 1}'
```
Notice that in the rogue-shell terminal, just the number of captured TCP/IP packets is printed. The capture file now holds packets you can decode with a shell one-liner.
In the rogue-shell container, press CTRL+C to stop the capture.

In the rogue-shell container, execute this command to extract the useful information from the capture file.

$ jq -r '.[] | ._source.layers | if .["http.request.method"] then "[REQ] " + .["http.request.method"][0] + " " + .["http.request.uri"][0] + (if .["http.file_data"] then "\t" + .["http.file_data"][0] else "" end) elif .["http.response.code"] then "[RES] HTTP " + .["http.response.code"][0] + " " + .["http.request.uri"][0] + (if .["http.file_data"] then "\t" + .["http.file_data"][0] else "" end) else empty end' /tmp/capture.json | while IFS=$'\t' read -r label hex; do [ -n "$hex" ] && echo "$label => $(printf '%s' "$hex" | xxd -r -p)" || echo "$label"; done

The expected output shows the plaintext data involved in the API request and response.

[REQ] GET /stock/midnight-bloom
[RES] HTTP 200 /stock/midnight-bloom => {"available":4,"name":"Midnight Bloom","size":"16oz","style":"Floral Illustration"}
[REQ] POST /stock/midnight-bloom/reserve => {"quantity": 1}
[RES] HTTP 200 /stock/midnight-bloom/reserve => {"design_id":"midnight-bloom","name":"Midnight Bloom","remaining":3,"reserved":1}

In the rogue-shell terminal, type CTRL+D to exit from the container.

The lack of adequate security for the services allows anyone with access to the network segment to capture the plaintext service network traffic without authentication.

Stop phase 1 infrastructure

IMPORTANT

When you finish exploring the phase 1 infrastructure, be sure to stop it before continuing to phase 2.

Stop the phase 1 infrastructure.

$ docker compose -f docker-compose.insecure.yml down

Example output:

[+] down 5/5
 ✔ Container rogue-service                            Removed     0.0s
 ✔ Container order-service                            Removed     1.2s
 ✔ Container rogue-shell                              Removed     1.1s
 ✔ Container inventory-service                        Removed     1.2s
 ✔ Network learn-vault-app-tls_cupfinisher Removed     0.2s

Phase 1 summary

During phase 1, you observed the following security problems with the microservice infrastructure:

Unencrypted traffic: Anyone on the network segment can read plaintext data in transit.
No authentication: The rogue service accessed data from the inventory-service without restriction.
No service identity: Services cannot verify with whom they are communicating.

Phase 2: Explore the configuration

The tutorial scenario uses several Vault components and features working together to implement the mTLS solution. Explore ACL policies and configuration for the Vault PKI secrets engine and Vault Agent to better understand how each component supports the implementation.

The access control list (ACL) policy involved in the scenario focuses on PKI secrets engine certificate requests. After each service's Vault Agent authenticate with Vault, the Vault Agent instance receives a token with the 3 highlighted capabilities shown here.

# Issue certificates from the cupfinisher-service role
path "pki_int/issue/cupfinisher-service" {
  capabilities = ["create", "update"]
}

# Read the intermediate CA cert (needed for CA bundle)
path "pki_int/cert/ca" {
  capabilities = ["read"]
}

# Read the root CA cert (needed for full chain verification)
path "pki/cert/ca" {
  capabilities = ["read"]
}

The ACL policy has 3 capabilities granted to the Vault Agent sidecars so that they can manage the TLS lifecycle for the microservices. The policy is identical for both services, and you can find the actual files in the policies directory.

Lines 2-4 enable issuing new certificates from the cupfinisher service role.
Lines 7-9 enable reading the intermediate CA certificate.
Lines 12-14 enable reading the root CA certifcate.

PKI secrets engine

To get an idea of how Vault acts as an intermediate and root CA, check out the Vault server PKI secrets engine configuration commands and their details here.

Vault's PKI secrets engine acts as a TLS CA. The tutorial configures a two-tier CA hierarchy:

Root CA (pki path): Issues only intermediate CA certificates. In production, you keep the Root CA offline or in a Hardware Security Module (HSM).
Intermediate CA (pki_int path): Issues service certificates. If compromised, you can revoke it without replacing the root.

The following commands from the setup-vault.sh script enabled the root CA:

vault secrets enable -path=pki pki && \
vault secrets tune -max-lease-ttl=87600h pki && \
vault write -field=certificate pki/root/generate/internal \
  common_name="CupFinisher Root CA" \
  issuer_name="cupfinisher-root" \
  ttl=87600h >/tmp/root_ca.crt && \
vault write pki/config/urls \
issuing_certificates="http://vault:8200/v1/pki/ca" \
crl_distribution_points="http://vault:8200/v1/pki/crl"

The following commands from the setup-vault.sh script enabled the intermediate CA:

vault secrets enable -path=pki_int pki && \
vault secrets tune -max-lease-ttl=43800h pki_int && \
vault write -field=csr pki_int/intermediate/generate/internal \
  common_name="CupFinisher Intermediate CA" \
  issuer_name="cupfinisher-intermediate" >/tmp/pki_int.csr && \
vault write -field=certificate pki/root/sign-intermediate \
  issuer_ref="cupfinisher-root" \
  csr=@/tmp/pki_int.csr \
  format=pem_bundle \
  ttl=43800h >/tmp/intermediate.cert.pem && \
vault write pki_int/intermediate/set-signed \
  certificate=@/tmp/intermediate.cert.pem && \
vault write pki_int/config/urls \
  issuing_certificates="http://vault:8200/v1/pki_int/ca" \
  crl_distribution_points="http://vault:8200/v1/pki_int/crl"

The PKI secrets engine cupfinisher-service role defines the following certificate parameters:

Allowed domains: *.cupfinisher.internal
Key type: EC P-256 is smaller and faster than RSA.
Time To Live (TTL): 24 hours; short-lived certificates reduce compromise window.
Flags: Both server_flag and client_flag enabled for mTLS.

The following command from the setup-vault.sh script defines the cupfinisher-service role.

vault write pki_int/roles/cupfinisher-service \
  allowed_domains="cupfinisher.internal" \
  allow_subdomains=true \
  allow_bare_domains=false \
  max_ttl="720h" \
  ttl="24h" \
  key_type="ec" \
  key_bits=256 \
  require_cn=true \
  server_flag=true \
  client_flag=true \
  no_store=true

Tip

You can explore the scripts/setup-vault.sh script to review the entire workflow that defines the root and intermediate CAs.

Inspect issued certificates

Run the certificate inspection script to see details about the certificates Vault issued.

$ bash ./scripts/inspect-certs.sh

The output shows:

Leaf certificates for each service with the common name matching the service hostname (for example, order.cupfinisher.internal).
Issuer chain showing the intermediate CA signed by the root CA.
Certificate attributes including the EC P-256 key type and 24-hour time to live (TTL) values.

Vault Agent

Vault Agent plays a critical role in this solution by managing the TLS certificate and key lifecycle for each service. Vault Agent even writes the information to the files required to secure the service with its template feature, and signals the services to restart when changes to the certificate and key occur.

Here is the configuration used by each Vault Agent instance in the tutorial scenario.

pid_file = "/tmp/pidfile"

vault {
  address = "http://vault:8200"
}

auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/vault-data/order-service/role-id"
      secret_id_file_path = "/vault-data/order-service/secret-id"
      remove_secret_id_file_after_reading = false
    }
  }

  sink "file" {
    config = {
      path = "/tmp/vault-token"
    }
  }
}

template_config {
  static_secret_render_interval = "10s"
  exit_on_retry_failure         = true
}

# Service certificate (cert + issuing CA in chain)
template {
  source      = "/vault/templates/order-cert.tpl"
  destination = "/vault/certs/cert.pem"
  perms       = 0644
  command     = "touch /vault/certs/.reload"
}

# Private key
template {
  source      = "/vault/templates/order-key.tpl"
  destination = "/vault/certs/key.pem"
  perms       = 0600
}

# CA bundle (for verifying peer certificates)
template {
  source      = "/vault/templates/ca-chain.tpl"
  destination = "/vault/certs/ca.pem"
  perms       = 0644
}

Vault Agent runs as a sidecar container beside each service, and automatically performs the following tasks:

Authenticates to Vault using the AppRole auth method and credentials from a shared volume.
Requests a certificate from the PKI secrets engine.
Writes the certificate, key, and CA bundle to a volume mounted by the service
Renews the certificate before it expires.

The service reads TLS material from the mounted volume and does not interact with Vault directly. This separation of concerns keeps Vault complexity out of your application code.

AppRole authentication

Each Vault Agent service instance uses the AppRole auth method to authenticate with Vault.

The vault-init container creates an AppRole for each service with a policy that restricts certificate requests to a specific hostname. The order service can only request certificates for order.cupfinisher.internal, and the inventory service can only request certificates for inventory.cupfinisher.internal.

Tip

By restricting issuance to hostnames, you follow the principle of least privilege, and permit services to get credentials just for their own identity.

Phase 3: Secure services with mTLS

In this phase, you deploy the same microservices from phase 1, but now each service has mTLS protection. Vault acts as the Certificate Authority, and Vault Agent sidecars manage certificate lifecycle for each service.

The Phase 3 infrastructure adds four more containers to the original service infrastructure:

Container	Purpose
vault	Vault server running in dev mode
vault-init	Initializes PKI secrets engine and AppRole auth, then exits
order-vault-agent	Manages certificates for order-service
inventory-vault-agent	Manages certificates for inventory-service

Start phase 3 services

Build and start the updated container infrastructure.

$ docker compose \
    --file docker-compose.secure.yml \
    up \
    --build \
    --detach

Example output:

...snip...
[+] up 15/15
 ✔ Image learn-vault-app-tls-inventory-service Built    9.5s
 ✔ Image learn-vault-app-tls-rogue-service     Built    9.5s
 ✔ Image learn-vault-app-tls-order-service     Built    9.5s
 ✔ Network cupfinisher                         Created  0.0s
 ✔ Volume learn-vault-app-tls_vault-data       Created  0.0s
 ✔ Volume learn-vault-app-tls_inventory-certs  Created  0.0s
 ✔ Volume learn-vault-app-tls_order-certs      Created  0.0s
 ✔ Container vault                             Healthy  3.6s
 ✔ Container rogue-shell                       Started  0.1s
 ✔ Container vault-init                        Exited   4.3s
 ✔ Container inventory-service                 Started  4.3s
 ✔ Container inventory-vault-agent             Started  4.3s
 ✔ Container order-service                     Started  4.3s
 ✔ Container order-vault-agent                 Started  4.3s
 ✔ Container rogue-service                     Started  4.4s

The vault-init container configures Vault with the two-tier CA hierarchy you learned about in phase 2. The vault-init container then creates a PKI role for issuing service certificates, sets up AppRole authentication for each service, and writes credentials to a shared volume.

Vault Agent containers authenticate to Vault with the AppRole auth method, request certificates for their respective services, and write the certificate, key, and CA bundle to mounted volumes.

Use the wait_for_vault.sh script to learn when the infrastructure is up and running. This process can take a few moments seconds while Vault initializes.

$ bash ./scripts/wait_for_vault.sh
Vault PKI initialized.

The expected result is Vault PKI initialized.

Check the rogue service

The rogue service is running and attempting unauthenticated access to the inventory service. Check the logs to learn if the rogue service gained access to the inventory service.

$ docker logs rogue-service

Tip

If you notice "Waiting for target services to be ready...", wait a moment and run the command again.

Example output:

[Rogue] Waiting for target services to be ready...
[Rogue] Generating self-signed certificate...
...snip...
[Rogue] Strategy: Maybe the server doesn't actually check for client certs?
[Rogue] Attempting: GET /stock/geometric-wave (no client cert)...
[Rogue] Request failed: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
[Rogue] Strategy: Present our self-signed cert and hope the server
[Rogue]           doesn't verify the issuer chain...
[Rogue] Attempting: GET /stock/geometric-wave (with self-signed cert)...
[Rogue] REJECTED!
[Rogue] Server TLS error: HTTPSConnectionPool(host='inventory.cupfinisher.internal', port=8444): Max retries exceeded with url: /stock/geometric-wave (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2580)')))
[Rogue] The server verified our certificate's issuer chain
[Rogue] and found it was NOT signed by the Vault CA.
...snip...
[Rogue] Attempting: GET /stock/geometric-wave (with spoofed CN cert)...
[Rogue] REJECTED!
[Rogue] Server TLS error: HTTPSConnectionPool(host='inventory.cupfinisher.internal', port=8444): Max retries exceeded with url: /stock/geometric-wave (Caused by SSLError(SSLError(1, '[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:2580)')))
...snip...
The rogue service will attempt three increasingly clever
attacks against the mTLS-protected Inventory Service.
...snip...
  Attack 1 (no cert): Rejected
  Attack 2 (self-signed): Rejected
  Attack 3 (spoofed CN): Rejected

Now the logs tell a different story. With mTLS enabled, the rogue-service:

Cannot connect without presenting a client certificate.
Cannot use a self-signed certificate because the CA chain is not trusted.
Cannot impersonate a legitimate service because it does not have the required credentials to authenticate with and request certificates from Vault.

Try to capture network traffic

Connect to the rogue-shell container and capture the service traffic.
```
$ docker exec -it rogue-shell bash
```
Inside the rogue-shell container, run tshark to specifically capture traffic on the order service port.
```
$ tshark -i eth0 -f "tcp port 8443"
```
Open a second terminal and change into the learn-vault-app-tls project directory.
Connect to the rogue-shell container:
```
$ docker exec -it rogue-shell bash
```

Try to generate an order.

$ curl -k -X POST https://order-service:8443/order \
    -H 'Content-Type: application/json' \
    -d '{"design": "midnight-bloom", "quantity": 1}'

You cannot create an order due to TLS, and the client gets an error from the SSL library:

curl: (56) OpenSSL SSL_read: OpenSSL/3.5.4: error:0A00045C:SSL routines::tlsv13 alert certificate required, errno 0

On the order service server side, the errors resemble this example:

...snip...
16 380.381005912   172.21.0.5 → 172.21.0.3   TCP 66 8443 → 47102 [ACK] Seq=1 Ack=1565 Win=68352 Len=0 TSval=2558660944 TSecr=663532183
17 380.382411704   172.21.0.5 → 172.21.0.3   TLSv1.3 3344 Server Hello, Change Cipher Spec, Application Data, Application Data, Application Data, Application Data, Application Data
18 380.382460495   172.21.0.3 → 172.21.0.5   TCP 66 47102 → 8443 [ACK] Seq=1565 Ack=3279 Win=70912 Len=0 TSval=663532184 TSecr=2558660945
19 380.384606287   172.21.0.3 → 172.21.0.5   TLSv1.3 176 Change Cipher Spec, Application Data, Application Data
20 380.384807912   172.21.0.3 → 172.21.0.5   TLSv1.3 271 Application Data
21 380.384848162   172.21.0.5 → 172.21.0.3   TLSv1.3 90 Application Data
22 380.384979995   172.21.0.3 → 172.21.0.5   TCP 66 47102 → 8443 [FIN, ACK] Seq=1880 Ack=3303 Win=70912 Len=0 TSval=663532187 TSecr=2558660948
23 380.385087829   172.21.0.5 → 172.21.0.3   TCP 66 8443 → 47102 [RST, ACK] Seq=3303 Ack=1881 Win=71296 Len=0 TSval=2558660948 TSecr=663532187

The rogue-shell terminal displays encrypted captured data (shown as "Application Data"). You cannot read the plaintext data representing the request or response content.

Press CTRL+D in each of the rogue-shell containers to exit.

Phase 3 summary

You resolved the following security problems from Phase 1 with:

Encrypted traffic: TLS protects all service-to-service communication.
Mutual authentication: Services present Vault-issued client certificates.
Trusted identity: Only accept certificates from the Vault CA.

Phase 4: Clean up

Stop all containers and remove project resources created by following the tutorial steps.

Remove all containers.
The rogue-shell container runs in an infinite loop, and this prevents Docker Compose from stopping it during docker compose down.
You must manually stop and remove the rogue-service container.
```
$ docker stop rogue-service && docker rm rogue-service
```

Stop and remove the remaining containers.

$ docker compose -f docker-compose.insecure.yml down && \
    docker compose -f docker-compose.secure.yml down

Remove the container images.

$ docker images --filter "reference=*cupfinisher*" -q \
    | xargs -r docker rmi -f && \
    docker images --filter "reference=zero-to-mtls*" -q \
    | xargs -r docker rmi -f

Prune the build cache.
```
$ docker builder prune -f
```
Knowledge checks
A quiz to test your knowledge.
1. Why does the rogue service fail to connect to the mTLS-protected inventory service in Phase 3?
  The rogue service fails because it cannot present a valid client certificate signed by the Vault CA that the server trusts. When a service connects without a client certificate, or presents its own self-signed certificate, the TLS handshake fails during certificate verification because the issuer is unknown to the server. Only services with certificates issued by Vault can successfully complete the mTLS authentication process.
2. What are the three ACL policy capabilities granted to Vault Agent sidecars in this scenario?
  The three capabilities are:
  - create and update for issuing new certificates from the PKI secrets engine role
  - read for accessing the intermediate CA certificate
  - read for accessing the root CA certificate
  These permissions allow Vault Agent sidecars to manage their TLS certificate lifecycle without requiring direct interaction with Vault or exposing credentials to the application services.
3. How does the two-tier CA hierarchy in this tutorial improve security compared to using a single root CA?
  The two-tier hierarchy separates concerns and limits exposure. The Root CA (pki path) issues only intermediate CA certificates and can be kept offline or in an HSM for maximum protection. The Intermediate CA (pki_int path) issues service certificates. If the intermediate CA is compromised, it can be revoked without needing to replace the root CA, which significantly reduces operational risk and simplifies certificate revocation procedures compared to a single-root design where any compromise requires full replacement of all issued certificates.

Summary

You explored a scenario and learned how unencrypted and unauthenticated service communications create security vulnerabilities, and how Vault-managed mTLS can help solve these problems. You also explored mTLS implemented in services through minimal application code changes with Vault Agent, and discovered details about the configuration and ACL policies to support the implementation.

Next steps

Consider the following resources to extend what you learned here.

Read the PKI secrets engine documentation to dive into certificate revocation, OCSP, and CRL configuration.
Explore Vault Agent configuration for more auto-auth methods and template options.
Try the Build your own certificate authority tutorial for a deeper walk through of PKI configuration.
Use certificate rotation strategies for production deployments.
Authenticate workloads with TLS explains well architected framework principles around mTLS.
Manage TLS with Infrastructure as Code helps you codify TLS requirements for your infrastructure.

Access secrets with Vault Agent

Secure CI/CD pipelines with Vault

Challenge

Solution

Prerequisites

Scenario introduction

Phase 1: Observe insecure microservices

Place a test order

Check rogue service logs

Capture and inspect network traffic

Stop phase 1 infrastructure

Phase 1 summary

Phase 2: Explore the configuration

Phase 3: Secure services with mTLS

Start phase 3 services

Check the rogue service

Try to capture network traffic

Phase 3 summary

Phase 4: Clean up

Knowledge checks

Summary

Next steps