Disaster recovery initial setup

Note

This page covers initial disaster recovery replication setup. For recurring failover, promotion, and failback operations, see the Administration Guide: Disaster recovery operations.

Overview

Prior to configuring Vault, it is essential to plan for potential infrastructure failures. HashiCorp advises that Vault Enterprise customers implement, at a minimum, a disaster recovery (DR) solution for their production clusters.

Note

HCP Vault cluster DR posture is managed by the HCP Platform. For more information please review the High Availability and Disaster Recovery documentation.

Prerequisites

You have reviewed and implemented the HashiCorp Vault Installation Guide.
You have deployed two separate Vault clusters, one acting as the DR primary, and the other acting as at the DR secondary.
The DR Vault clusters are initialized and unsealed.
You have a valid root tokens for your clusters.

Types of cluster replication in Vault Enterprise

Vault can be extended across data centers or cloud regions using the replication feature of Vault Enterprise. Vault replication operates on a leader/follower model, wherein a leader cluster (known as a primary) is linked to a series of follower secondary clusters to which the primary cluster replicates data.

Vault DR Architecture

The data replicated between the primary and secondary cluster is determined by the type of replication: disaster recovery (DR) or performance replication (PR).

A DR secondary cluster is a complete mirror of its primary cluster. DR Replication is designed to be a mechanism to protect against catastrophic failure of entire clusters. DR secondaries do not service read or write requests until they are promoted and become a new primary. A DR secondary cluster acts as a warm standby cluster.

A PR secondary cluster replicates everything from the primary except for tokens, leases (dynamic secrets), and any local(opens in new tab) secrets engines, and is covered in this document(opens in new tab).

Setting up DR replication

At this point, you should have two Vault Enterprise clusters: one that will act as the DR primary cluster, and another that will become the DR secondary cluster. We strongly advise carefully reading and implementing the details provided in the official documentation on disaster recovery replication set up(opens in new tab).

Configure DR operation token

A DR operation token is required to promote a DR secondary cluster as the new primary. This token should be created in advance and stored safely outside of Vault in case of a failure condition that would warrant the promotion of the DR secondary cluster.

Because the DR operation token has the ability to fundamentally modify your Vault clusters' topology and operation, it should be securely stored and time limited with an expiration date. While the public document regarding DR operation token strategy suggests that you can create a batch token as an alternative to a DR operation token, we recommend that you use the former, which can be generated from the DR primary cluster without your recovery keys (or unseal key if you are not using auto-unseal). The batch DR operation token has a fixed TTL and the Vault server will automatically delete it after it expires.

In addition to generating the DR operation token, automate its renewal and ensure that you store it outside of Vault’s operational blast radius. This ensures that, in the event Vault becomes unavailable or inaccessible, you can still retrieve the token to perform a DR promotion.

The storage location must be external to Vault and resilient to the same failure scenarios. Suitable options include an object storage service (such as an S3 bucket), a shared password manager, or an on-premises filesystem. Regardless of the mechanism chosen, you must ensure that the token remains both accessible and valid for use during a failure condition.

Note

You must have permission to call the auth/token/create endpoint in the root namespace for this CLI command. In the Configure Admin Policy section, you will configure a policy with super administrators access to call this API in the root namespace.

The recommendation to create a batch DR operation token means the operations team have such but that this is valid for eight hours. Use the token in such a way that when a Vault operator comes on a shift, the operator generates a batch DR operation token with a TTL equal to the duration of the shift.