Disaster recovery replication

Enterprise

Appropriate Vault Enterprise license or HCP Vault Dedicated cluster required.

Disaster recovery (DR) replication keeps a warm standby copy of a Vault Enterprise cluster. A DR secondary receives every change from its primary, including tokens and leases, but does not serve client requests. When the primary fails, you promote the DR secondary to primary, and clients continue with the tokens and leases they already have.

To compare DR replication with performance replication, refer to Vault Enterprise replication. To set it up, refer to Enable DR replication.

Introduction

A DR secondary mirrors the primary's configuration, policies, secrets, local mounts, tokens, and leases. Because the DR secondary holds the same tokens and leases, applications keep working after a promotion without reauthenticating. Point the clients at the new primary and they continue.

DR replication protects against the loss of a cluster. It does not protect against bad data, and does not replace the need for backups.

A DR secondary replicates deletions, corruption, and accidental changes along with everything else. Keep regular snapshots in addition to DR replication so you can restore data from a point in time.

Use cases

Use DR replication for the following:

Cluster, data center, or region loss: Promote a DR secondary in another location when the primary cluster becomes unavailable.
Planned maintenance and migrations: Move operations to the DR secondary while you work on the primary's infrastructure, then fail back.
Upgrades with a fallback: Upgrade the DR secondary first, then fail over to it. Refer to Upgrade replicated deployments.
Recovery testing: Practice failover and failback to measure your recovery time objective (RTO) and recovery point objective (RPO). Refer to Test disaster recovery.
Protect performance clusters: Give each performance primary and performance secondary its own DR secondary.

How a DR secondary behaves

A DR secondary differs from a normal cluster in the following ways:

No client requests: The secondary rejects most API requests with the error path disabled in replication DR secondary mode until you promote it. It does not forward requests to the primary.
No local tokens: Tokens on the secondary come from the primary. The root token the secondary had before you enabled replication stops working.
Restricted DR operations: Promote, update primary, disable, reindex, and Merkle check on the secondary need a DR operation token.
No automatic promotion: Vault never promotes a DR secondary on its own. You decide when to fail over. Refer to DR failover.

Design considerations

Plan the following before you enable DR replication. The shared design considerations for both replication types also apply.

Location

Place the DR secondary where the events that could take down the primary are unlikely to reach it, such as another region or another cloud provider. Account for latency between the locations, because latency limits how fast changes reach the secondary. Refer to Tune replication.

Capacity

A DR secondary has two workloads: the replication workload before promotion, and the full client workload after promotion. Size for the first and plan for the second.

Storage: Match the primary's storage performance and capacity. The secondary writes every change the primary makes, including tokens and leases, and stores the full data set. A secondary with slower storage falls behind, falls back to Merkle syncs, and increases the data you lose in a failover (recovery point objective, RPO).
Network: Provision enough bandwidth between sites for the primary's peak write rate. Latency between sites limits how fast changes arrive. Refer to Tune replication.
Compute: Before promotion, the secondary serves no client requests, so it can run with less CPU and memory than the primary. Before you promote it, scale it to match the primary so it can serve all client traffic.
Node count: After promotion, the secondary is your only cluster. Run enough nodes to meet the same availability target as the primary, or include adding nodes in the scale-up plan.

Scaling up adds to your recovery time objective (RTO). Include the time to resize or replace nodes in the RTO, confirm that the target instance types are available in the secondary's region during a regional outage, and test the scale-up as part of your failover tests. For sizing guidance, refer to the HashiCorp Validated Design for Vault.

Network access

Plan network access for normal operation and for each failover direction:

Cluster port: Open the cluster port, 8201 by default, in both directions between the clusters. If a load balancer fronts the cluster port, place one in front of each cluster, because the promoted cluster becomes the primary that other secondaries connect to. Set its address with primary_cluster_addr when you promote.
API port between clusters: A secondary calls its primary's API port, 8200 by default, during activation and when you update its primary, to unwrap the activation token. Failback reverses the roles, so open the API port in both directions, or activate with a public key to avoid the API call. Refer to Activation methods.
API port for operators and monitoring: Operators need the secondary's API port to check its status, generate a DR operation token, and promote it. Monitoring systems need it to poll the replication status endpoint.
API port for clients: Clients do not need the DR secondary until you promote it, so you can keep its API port closed to clients. If you do, define who opens it during a failover and how, and include that step in your failover tests.

For bandwidth and latency between sites, refer to Capacity.

Activation method

Choose how the secondary receives the activation token:

Wrapping token: The default. The secondary must reach the primary's API port and trust the primary's TLS certificate.
Public key: The secondary makes no API call to the primary. Use this method when network policy blocks the API port between sites, or when the secondary cannot validate the primary's certificate.

The same choice applies when you later point a secondary at a new primary with update primary. Refer to Activation methods.

Unseal strategy and key holders

After you enable DR replication, the secondary uses the primary's unseal or recovery keys. Decide in advance:

Who holds the key shares, and how quickly a quorum can gather during an outage. Generating a DR operation token needs the same quorum.
Whether both clusters use the same seal type. A secondary that needs a manual Shamir unseal while the primary uses auto-unseal adds work during an outage.
Whether the secondary can reach its auto-unseal provider if the primary's region is down.

DR operation token readiness

A promotion needs a DR operation token, and Vault 2.0.0 and later also need a valid Vault token to generate one. Prepare the token that authenticates generation, such as a batch token with a long TTL, before you need it. Refer to DR operation tokens.

Client redirection

Decide how clients find the new primary after a promotion, for example with a global load balancer, DNS failover records, or service discovery. Test the redirection as part of your failover tests. Refer to the Well-Architected Framework guidance on failover.

Vault versions

After a failover, a promoted secondary on a newer Vault version cannot replicate back to an older original primary. Upgrade the original primary before you reconnect it as a secondary. Plan upgrades so that the DR secondary runs the same version or the newer one.

Performance replication

If the cluster you protect is also part of a performance replication set, set up performance replication first, then add the DR secondary. Enabling performance replication on a secondary deletes all its data. Refer to Combine DR and performance replication.

Operations and testing

Write and test runbooks for failover and failback. Include who decides to fail over, how to confirm the primary is down, and how to keep the original primary from serving clients when it returns. Refer to the HashiCorp Validated Design for DR operations.