Configure Vault as a Certificate Manager in Kubernetes with Helm
Kubernetes configured to use Vault as a certificate manager enables your services to establish their identity and communicate securely over the network with other services or clients internal or external to the cluster.
Jetstack's cert-manager enables Vault's PKI secrets engine to dynamically generate X.509 certificates within Kubernetes through an Issuer interface.
In this tutorial, you set up Vault with the Vault Helm chart, configure the PKI secrets engine and Kubernetes authentication. Then install Jetstack's cert-manager, configure it to use Vault, and request a certificate.
Prerequisites
This tutorial requires the Kubernetes command-line interface (CLI) and the Helm CLI installed, Minikube, the Vault Helm charts, the sample web application, and additional configuration to bring it all together.
This tutorial was last tested 29 Apr 2020 on a macOS 10.15.4 using this configuration.
Docker version.
Minikube version.
Helm version.
These are recommended software versions and the output displayed may vary depending on your environment and the software versions you use.
First, follow the directions for installing Minikube, including VirtualBox or similar.
Next, install kubectl CLI and helm CLI.
Next, retrieve the web application and additional configuration by cloning the hashicorp/vault-guides repository from GitHub.
This repository contains supporting content for all of the Vault learn guides. The content specific to this tutorial can be found within a sub-directory.
Go into the
vault-guides/operations/provision-vault/kubernetes/minikube/external-vault
directory.
Working directory
This tutorial assumes that the remainder of commands are executed within this directory.
Start Minikube
Minikube is a CLI tool that provisions and manages the lifecycle of single-node Kubernetes clusters. These clusters are run locally inside Virtual Machines (VM).
Start a Kubernetes cluster.
Verify the status of the Minikube cluster.
Additional waiting
Even if the previous step completed successfully, you may have to wait for Minikube to be available. If you see an error, try again after a few minutes.
The host, kubelet, and apiserver report that they are running. The kubectl
, a
command line interface (CLI) for running commands against Kubernetes cluster, is
also configured to communicate with this recently started cluster.
Install the Vault Helm chart
Add the HashiCorp Helm repository.
Update all the repositories to ensure helm
is aware of the latest versions.
Install the latest version of the Vault server running in standalone mode with the Vault Agent Injector service disabled.
The Vault server runs in standalone mode on a single pod. By default the Helm
chart starts a Vault Agent Injector pod but that is disabled
injector.enabled=false
.
Get all the pods within the default namespace.
The vault-0
pod is deployed. The Vault server running in the pod's container
reports that it is running but it is not ready (0/1
). To ready the pod
requires that the Vault server is initialized and unsealed.
Get all the services within the default namespace.
The Vault Helm chart creates a Service that directs requests to the Vault pod.
This enables us to address the Vault server within the cluster with the address
http://vault.default:8200
.
Initialize and unseal Vault
Vault run in standalone mode starts uninitialized and in the sealed state. Prior to initialization the storage backend is not prepared to receive data.
Initialize Vault with one key share and one key threshold.
The operator init
command
generates a root key that it disassembles into key shares -key-shares=1
and
then sets the number of key shares required to unseal Vault -key-threshold=1
.
These key shares are written to the output as unseal keys in JSON format
-format=json
. Here the output is redirected to a local file named
init-keys.json
View the unseal key found in init-keys.json
.
Insecure operation
Do not run an unsealed Vault in production with a single key share and a single key threshold. This approach is only used here to simplify the unsealing process for this demonstration.
Create a variable named VAULT_UNSEAL_KEY
to capture the Vault unseal key.
After initialization, Vault is configured to know where and how to access the storage, but does not know how to decrypt any of it. Unsealing is the process of constructing the root key necessary to read the decryption key to decrypt the data, allowing access to the Vault.
Unseal Vault running on the vault-0
pod with the $VAULT_UNSEAL_KEY
.
The operator unseal
command reports that Vault is initialized and unsealed.
Insecure operation
Providing the unseal key with the command writes the key to your shell's history. This approach is only used here to simplify the unsealing process for this demonstration.
Get all the pods within the default namespace.
The vault-0
pod reports that it is ready 1/1
. Vault is ready for you to
login with the root token generated during the initialization.
View the root token found in init-keys.json
.
Create a variable named VAULT_ROOT_TOKEN
to capture the root token.
Login to Vault running on the vault-0
pod with the $VAULT_ROOT_TOKEN
.
The Vault server is ready to be configured as a certificate store.
Configure PKI secrets engine
First, start an interactive shell session on the vault-0
pod.
Your system prompt is replaced with a new prompt / $
. Commands issued at this
prompt are executed on the vault-0
container.
Enable the PKI secrets engine at its default path.
By default the KPI secrets engine sets the time-to-live (TTL) to 30 days. A certificate can have its lease extended to ensure certificate rotation on a yearly basis (8760h).
Configure the max lease time-to-live (TTL) to 8760h
.
Vault can accept an existing key pair, or it can generate its own self-signed root. In general, we recommend maintaining your root CA outside of Vault and providing Vault a signed intermediate CA.
Generate a self-signed certificate valid for 8760h
.
Example output:
Configure the PKI secrets engine certificate issuing and certificate revocation list (CRL) endpoints to use the Vault service in the default namespace.
Output:
Configure a role named example-dot-com
that enables the creation of
certificates example.com
domain with any subdomains.
Output:
The role, example-dot-com
, is a logical name that maps to a policy used to
generate credentials. This generates a number of endpoints that are used by the
Kubernetes service account to issue and sign these certificates. A policy must
be created that enables these paths.
Create a policy named pki
that enables read access to the PKI secrets engine
paths.
Output:
These paths enable the token to view all the roles created for this PKI secrets
engine and access the sign
and issues
operations for the example-dot-com
role.
Lastly, exit the vault-0
pod.
Configure Kubernetes authentication
Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes Service Account Token.
First, start an interactive shell session on the vault-0
pod.
Your system prompt is replaced with a new prompt / $
. Commands issued at this
prompt are executed on the vault-0
container.
Enable the Kubernetes authentication method.
Configure the Kubernetes authentication method to use location of the Kubernetes API.
For the best compatibility with recent Kubernetes versions, ensure you are using Vault v1.9.3 or greater.
Output:
The environment variable KUBERNETES_PORT_443_TCP_ADDR
references the internal network address of the Kubernetes host.
You can validate the issuer name of your Kubernetes cluster using this method.
Finally, create a Kubernetes authentication role named issuer
that binds
the pki
policy with a Kubernetes service account named issuer
.
Output:
The role connects the Kubernetes service account, issuer
, in the default
namespace with the pki
Vault policy. The tokens returned after authentication
are valid for 20 minutes. This Kubernetes service account name, issuer
, is
created in the Deploy Issuer and Certificate
section.
Lastly, exit the vault-0
pod.
Deploy Cert Manager
Jetstack's cert-manager is a Kubernetes add-on that automates the management and issuance of TLS certificates from various issuing sources. Vault can be configured as one of those sources. The cert-manager requires the creation of a set of Kubernetes resources that provide the interface to the certificate creation.
Install Jetstack's cert-manager's version 0.14.3 resources.
Create a namespace named cert-manager
to host the cert-manager.
Jetstack's cert-manager Helm chart is available in a repository that they maintain. Helm can request and install Helm charts from these custom repositories.
Add the jetstack
chart repository.
Helm maintains a cached list of charts for every repository that it maintains. This list needs to be updated periodically so that Helm knows about all available charts and their releases. A repository recently added needs to be updated before any chart is requested.
Update the local list of Helm charts.
The results show that the jetstack
chart repository has retrieved an update.
Install the cert-manager chart version 0.11 in the cert-manager
namespace.
Output:
The cert-manager chart deploys a number of pods within the cert-manager
namespace.
Get all the pods within the cert-manager
namespace.
Wait until the pods prefixed with cert-manager
are running and ready (1/1
).
Thes pods now require configuration to interface with Vault.
Configure an issuer and generate a certificate
The cert-manager enables you to define Issuers that interface with the Vault certificate generating endpoints. These Issuers are invoked when a Certificate is created.
When you configured Vault's Kubernetes
authentication a Kubernetes service
account, named issuer
, was granted the policy, named pki
, to the certificate
generation endpoints.
Create a service account named issuer
within the default namespace.
The service account generated a secret that is required by the Issuer automatically in Kubernetes 1.23. In Kubernetes 1.24+, you need to create the secret explicitly.
Kubernetes 1.24+ only
Get all the secrets in the default namespace.
The issuer secret is displayed here as the secret prefixed with issuer-token
.
Create a variable named ISSUER_SECRET_REF
to capture the secret name.
Define an Issuer, named vault-issuer
, that sets Vault as a certificate
issuer.
Create the vault-issuer
Issuer.
The specification defines the signing endpoint and the authentication endpoint and credentials.
metadata.name
sets the name of the Issuer tovault-issuer
spec.vault.server
sets the server address to the Kubernetes service created in the default namespacespec.vault.path
is the signing endpoint created by Vault's PKIexample-dot-com
rolespec.vault.auth.kubernetes.mountPath
sets the Vault authentication endpointspec.vault.auth.kubernetes.role
sets the Vault Kubernetes role toissuer
spec.vault.auth.kubernetes/secretRef.name
sets the secret for the Kubernetes service accountspec.vault.auth.kubernetes/secretRef.key
sets the type totoken
.
Define a certificate named example-com
.
The Certificate, named example-com
, requests from Vault the certificate
through the Issuer, named vault-issuer
. The common name and DNS names are
names within the allowed domains for the configured Vault endpoint.
Create the example-com
certificate.
View the details of the example-com
certificate.
The certificate reports that it has been issued successfully.
Next steps
In this tutorial, you installed Vault configured the PKI secrets engine and Kubernetes authentication. Then installed Jetstack's cert-manager, configured it to use Vault, and requested a certificate.
Besides creation, these certificates can be revoked and removed. Learn more about Jetstack's cert-manager used in this tutorial and explore Vault's PKI secrets engine as a certificate authority in the Build Your Own Certificate Authority.