Enable DR replication

Enterprise

Appropriate Vault Enterprise license or HCP Vault Dedicated cluster required.

Enable disaster recovery (DR) replication between two Vault Enterprise clusters so the secondary keeps a warm standby copy of the primary. To learn what DR replication does and what to plan first, refer to Disaster recovery replication.

For a guided lab, complete the Enable disaster recovery replication tutorial.

Before you start

Run two Vault Enterprise clusters. Initialize and unseal a primary cluster and a secondary cluster. Both clusters need a storage backend that supports transactional updates, such as integrated storage. The secondary can run the same Vault version as the primary or a newer one.
Back up the secondary cluster. Enabling replication on the secondary deletes all existing data on it, including its root token, auth methods, and secrets.
Open the replication ports. Allow TCP traffic in both directions between the clusters on the cluster port, 8201 by default. The secondary also needs the primary's API port, 8200 by default, unless you activate it with a public key. Refer to Shared design considerations.
Set the cluster addresses. Set api_addr and cluster_addr on every node. Refer to the high availability parameters. If a load balancer fronts the primary's cluster port, it must pass TCP through without terminating TLS and route to the active node.
Choose an activation method. Decide whether the secondary unwraps the activation token from the primary's API or receives a token encrypted with its own public key. Refer to Activation methods.

Check your permissions. Run every command from the root namespace. On the primary, you must have permission to:
- create and update on sys/replication/dr/primary/enable
- create, update, and sudo on sys/replication/dr/primary/secondary-token
On the secondary, you must have permission to:
- create and update on sys/replication/dr/secondary/enable
- create and update on sys/replication/dr/secondary/generate-public-key, if you activate with a public key

Step 1: Enable DR replication on the primary

Run the following command on the primary cluster. The cluster is briefly unavailable while Vault enables replication.

If a load balancer fronts the primary's cluster port, set primary_cluster_addr to the load balancer address so that secondaries receive that address in their activation tokens.

$ vault write -force sys/replication/dr/primary/enable

With a load balancer address:

$ vault write sys/replication/dr/primary/enable \
    primary_cluster_addr="https://<load-balancer-address>:8201"

Parameters:

primary_cluster_addr (string: ""): The cluster address that the primary gives to secondaries. Refer to Enable DR primary replication.

Step 2: Generate a secondary activation token

Generate the token with the method you chose. The id value names the secondary. It must be unique, and you use it later to revoke the secondary's access. The token expires after the ttl, 30 minutes by default, and works once.

Run the following command on the primary cluster:

$ vault write -format=json sys/replication/dr/primary/secondary-token \
    id="<secondary-id>"

The activation token is the wrap_info.token value in the response.

Run the following command on the secondary cluster to generate its public key:
```
$ vault write -force sys/replication/dr/secondary/generate-public-key
```

Run the following command on the primary cluster, with the public key from the secondary:

$ vault write sys/replication/dr/primary/secondary-token \
    id="<secondary-id>" \
    secondary_public_key="<public-key>"

Run the following command on the secondary cluster to generate its public key:

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    $VAULT_ADDR/v1/sys/replication/dr/secondary/generate-public-key

Run the following command on the primary cluster, with the public key from the secondary:

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"id": "<secondary-id>", "secondary_public_key": "<public-key>"}' \
    $VAULT_ADDR/v1/sys/replication/dr/primary/secondary-token

Parameters:

id (string: <required>): An opaque identifier for the secondary.
ttl (string: "30m"): How long the activation token stays valid.
secondary_public_key (string: ""): The secondary's public key. When you set it, the primary encrypts the activation token with the key instead of wrapping it.

Refer to Generate DR secondary token and Generate DR secondary public key.

Step 3: Enable DR replication on the secondary

Warning

This step immediately deletes all data on the secondary cluster.

Run the following command on the secondary cluster with the activation token from step 2:

$ vault write sys/replication/dr/secondary/enable token="<activation-token>"

Parameters:

token (string: <required>): The secondary activation token.
primary_api_addr (string: ""): The primary's API address. Set it when the address in the token is not reachable from the secondary, for example behind a load balancer.
ca_file (string: ""): Path to a PEM-format CA file that the secondary uses to verify the primary's API certificate when it unwraps the token. Applies to the wrapping token method.
ca_path (string: ""): Path to a directory of PEM-format CA files, for the same purpose as ca_file.

If you do not set ca_file or ca_path, the secondary uses the system CA roots. Refer to Enable DR secondary.

After this step, the secondary uses the primary's unseal or recovery keys, and its previous root token stops working. Requests to most paths on the secondary return path disabled in replication DR secondary mode.

Step 4: Verify the replication status

Read the DR replication status on the secondary cluster. The status endpoint does not need a token.

$ vault read -format=json sys/replication/dr/status

Replication works when the secondary shows the following values:

mode is secondary.
connection_state is ready.
state is stream-wals. A new secondary can show merkle-diff or merkle-sync briefly while it copies the initial data.
known_primary_cluster_addrs lists the cluster address of the primary.
cluster_id matches the cluster_id on the primary.

On the primary, the same endpoint shows mode as primary, state as running, and the secondary in secondaries with connection_status as connected. Refer to Monitor replication for every field.

If a step fails, refer to Troubleshoot replication.

Next steps

Prepare for a failover before you need one:

Create the token that authenticates DR operation token generation, and store it where your operators can reach it during an outage. Refer to Prepare before a failover.
Set up monitoring and alerts. Refer to Monitor replication.
Learn how failover works. Refer to DR failover.