Vault Enterprise Best Practices Cluster

Vault's enterprise features focus around replication. Typically, one cluster is a leader, and other clusters are followers. There are two main types of following clusters. Disaster replication clusters serve as a warm standby should the leader cluster stop serving traffic. Performance replication clusters increase the number of reads that Vault can service.

AWS is given here in depth as a sample for auto-unseal, but any auto-unseal mechanism may be used.

Walk-Through

1. Create an EKS cluster. This walk-through will assume it's named "vault-cluster".
2. While the cluster is launching, create a role for Vault's node group to use. AWS's docs should be considered the authority on the policies that need to be attached, but at the time of writing, the required policies are the AmazonEKSWorkerNodePolicy, the AmazonEKS_CNI_Policy, and the AmazonEC2ContainerRegistryReadOnly. For the purposes of this walk-through, we will assume you name this new role, "EKSNodeGroupForAutoUnseal".
3. While the cluster is launching, also create a KMS key that will be used for auto-unseal. Under "Define key usage permissions", search for "EKSNodeGroupForAutoUnseal" and add a check next to it.
4. One you've finished creating the key, return to IAM. Create a new policy called "UnsealKeyUsageForVault" and attach it to the "EKSNodeGroupForAutoUnseal" role. The policy contents should be:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt",
                "kms:Encrypt",
                "kms:DescribeKey"
            ],
            "Resource": "arn:aws:kms:us-west-2:012345678901:key/207a68f5-f100-410c-8066-EXAMPLE"
        }
    ]
}

1. Return to EKS and add a node group. While launching it, under "Node IAM Role Name", select "EKSNodeGroupForAutoUnseal".
2. While the nodes are creating, point your local kubectl at your new cluster as shown here.

Generate TLS Certificates

TLS certificate generation is covered here.

Setting Up the Primary Cluster

The below values-custom.yaml can be used to set up a three server Vault cluster using Consul as a highly available storage backend, with AWS KMS for Auto Unseal.

# Create a values-custom.yaml to override default values
cat << EOF > values-custom.yaml
server:
  image:
    repository: "hashicorp/vault-enterprise"
    tag: "1.4.2_ent"

  affinity: |
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchLabels:
              app: {{ template "vault.name" . }}
              release: "{{ .Release.Name }}"
              component: server
          topologyKey: kubernetes.io/hostname

  ha:
    enabled: true
    replicas: 3

    config: |
      ui = true

      service_registration "kubernetes" {
        namespace      = "my-namespace"
        pod_name       = "my-pod-name"
      }

      listener "tcp" {
        tls_cert_file = "/path/to/fullchain.pem"
        tls_key_file  = "/path/to/privkey.pem"
        tls_client_ca_file = "/vault/userconfig/vault-server-tls/vault.ca"
        address = "[::]:8200"
        cluster_address = "[::]:8201"
      }

      storage "consul" {
        path = "vault-us-east"
        address = "HOST_IP:8500"
        token = "45a3bd52-07c7-47a4-52fd-0745e0cfe967"
      }

      seal "awskms" {
        region     = "us-east-2"
        kms_key_id = "19ec80b0-dfdd-4d97-8164-c6examplekey"
      }
EOF

Next, point Helm to the chart release and specify the values-custom.yaml file:

helm install vault-us-east hashicorp/vault -f values-custom.yaml

Note that in the above seal stanza, the AWS key and secret are omitted. That's because Vault picks them up from the node group's Instance Metadata, which in this example has been configured to provide credentials for the "EKSNodeGroupForAutoUnseal" role.

Also note that the namespace and pod_name in the service registration stanza can alternatively be set using the Downward API. Please see Kubernetes Service Registration for elaboration.

Lastly, note that the path for consul is set to vault-us-east. This is because, in a replicated environment, we would expect other clusters to be in other regions such as vault-us-west. A similar distinction was made in the helm install command's naming convention.

After bringing up the first cluster (and each cluster thereafter), Vault enterprise requires a license to be installed within a limited time window, as shown here.

Setting Up a Disaster Replication (DR) Cluster

Setting up a DR cluster follows an identical process and configuration for creating the cluster itself, but presumably in a different region. This would necessitate changing the value under consul and path to one relevant to the region for the DR cluster, such as vault-us-west, as well as the [NAME] used in the helm install command.

Once the cluster is up and the license is added, add it as a follower to the leader cluster. See Disaster Recovery Replication Setup for more information.

Setting Up a Performance Cluster

Setting up a performance cluster also follows an identical process and configuration for creating the cluster itself, with the region differentiations noted above.

Once the cluster is up and the license is added, add it as a follower to the leader cluster. See Setting Up Performance Replication for more on how to configure a cluster to follow the primary once it is up.