Best practices for Vault Kubernetes Key Management

To use Vault Kubernetes Key Management securely, consider the following best practices.

Use the smallest Kubernetes footprint possible

• Deploy as a Static Pod. Run the vault-kube-kms process as a static Pod on each control plane node.

• Use a minimal container image. Deploy using distroless or UBI-based images that contain only the process binary and necessary certificates.

• Restrict node access. Limit SSH and console access to your control plane nodes to authorized administrators only.

• Enforce network policies. Apply Kubernetes network policies to restrict the vault-kube-kms process so it can only access Vault endpoints.

Use fine-grained controls for Vault access

• Use the minimum required policy. Grant the smallest set of permissions required to the Transit plugin. For example:

  path "<transit_mount_path>/encrypt/<key_name>" {
    capabilities = ["create", "update"]
  }
  path "<transit_mount_path>/decrypt/<key_name>" {
    capabilities = ["create", "update"]
  }
  path "<transit_mount_path>/keys/<key_name>" {
    capabilities = ["read"]
  }
  path "sys/license/status" {
    capabilities = ["read"]
  }
  

• Use AES-256-GCM96 encryption. Configure your Transit key with the aes256-gcm96 type for authenticated encryption.

• Enable key versioning. You must enable key versioning for proper key rotation support.

• Rotate your key encryption keys regularly. Rotate the Transit key at least every 90 days per Kubernetes project recommendations.

• Use Vault namespaces. In multi-tenant environments, we recommend isolating the KMS Transit key in a dedicated Vault namespace.

Protect and rotate keys

Currently, Vault Kubernetes Key Management only supports AppRole authentication.

• Protect your secret ID. Mount the AppRole secret ID as a file with restricted permissions (mode 0400).

• Use short token TTLs. Configure the AppRole plugin with the minimum viable time for token TTLs.

• Rotate your secret IDs. Periodically rotate the AppRole secret IDs and update the mounted secret.

Use TLS and CA certificates

• Always Use TLS. Never start a production Vault server with the skip TLS verification flag (--tls-skip-verify).

• Provide your CA certificates. If you do not provide the CA certificate in the system trust store, you provide the CA certificate to Vault with the --tls-ca-file flag.

• Configure SNI with load balancers. If you deploy Vault behind a load balancer with a different hostname, you must provide the correct hostname at startup with the --tls-sni flag.

Monitor telemetry and audit logs

• Enable Vault audit logging. Always enable audit logging in Vault to record all Transit operations.

• Monitor your metrics. Scrape and review the Prometheus /metrics endpoint for key health metrics:

  • rpc.server.call.duration histograms to ensure latency stays within KMS v2 requirements.
  • Error rates on all status, encrypt, and decrypt operations to ensure proper communication between Vault and the KMS plugin.

• Configure alerts on failures. To catch potential problems early, configure alerts for:

  • All vault-kube-kms process health check failures.
  • Authentication failures to Vault.

• Regularly review process logs. Review the vault-kube-kms process logs regularly for authentication failures or unexpected errors.

Have a disaster recovery plan

• Backup the Transit key. Ensure you have a way to recover the Transit key. If you have appropriate safeguards for the externalized data, we recommend using the built-in key export functionality of the Transit plugin.

• Document your recovery procedures. Always document your recovery processes to ensure people can restore Kubernetes KMS or Vault if they become unavailable.

• Test your recovery process. Periodically test your documented recovery procedures to ensure they remain accurate to your API server and Vault Kubernetes KMS integration over time.

Upgrade carefully

• Test upgrades in non-production. Always test vault-kube-kms image upgrades in a non-production cluster first.

• Monitor logs during upgrades. Watch for authentication failures or latency increases during rolling upgrades.

• Always maintain backward compatibility. To ensure Vault can decrypt data encrypted with older key versions, make sure to configure the min_decryption_version field appropriately after an upgrade.