Perform an unplanned DR failover

An unplanned failover in Vault disaster recovery (DR) replication promotes the DR secondary cluster to primary after the primary cluster fails and you cannot demote it first. In this tutorial, you simulate a failure by deleting the running primary cluster. Then you promote the DR secondary cluster with a batch token and the unseal key, start a new cluster, and connect it as the new DR secondary.

A planned and an unplanned failover end in the same state: one primary cluster and one DR secondary cluster. They differ in the order of operations. In a planned failover, you demote the primary before you promote the secondary. In an unplanned failover, the primary is gone, so you promote the secondary first and replace the failed primary with a new DR secondary.

Prerequisites

This tutorial builds on top of previous tutorials. Complete the Perform a failback tutorial before you begin.

This lab was tested on macOS using Apple silicon-based processors.

To perform the tasks described in this tutorial, you need to have:

• Vault Enterprise 2.x binary installed
• A Vault Enterprise license
• Docker to run Vault Enterprise
• jq installed

Scenario

After the planned failover and failback, HashiCups' clusters are back to their original topology: vault-primary is the primary and vault-dr is the DR secondary.

Now Oliver needs to prepare for a scenario where vault-primary fails and does not come back. Oliver cannot demote a failed cluster. To restore service, Oliver promotes the DR secondary cluster to primary. Once service resumes, Oliver builds a replacement cluster and connects it as the new DR secondary, so HashiCups has a warm standby again.

Set up the lab

This tutorial uses the environment and configuration created during the previous tutorials.

• Complete the Enable disaster recovery replication tutorial
• Complete the Prepare for DR failover tutorial
• Complete the Perform a planned DR failover tutorial
• Complete the Perform a failback tutorial

Design considerations

Unlike a planned DR failover, you cannot verify the current replication status or notify Vault consumers and stakeholders.

Monitoring, logging, and alerting

Plan a workflow to automate updating your monitoring, logging, and alerting tools. Bringing a new cluster up may include new node names, IP addresses, and software versions.

• If you use HashiCorp Terraform, check to see if your services have a supported provider such as Splunk or Datadog.

Manually updating your monitoring and alerting services is time-consuming and error-prone.

Cluster access

Similar to updates to your monitoring, logging, and alerting tools, build a plan to update any cluster access services.

• If you use HashiCorp Boundary, consider using dynamic host catalogs on supported platforms. Bringing new Vault nodes online that match dynamic host catalog settings makes them available to your administrators.
• If you rely on DNS for client access to the Vault cluster, have a runbook to help application teams update the cluster address until you can bring your Vault cluster's topology back to your preferred configuration.

Verify replication before the failure

Confirm the clusters are in sync before you simulate the failure. Data that the primary has not replicated to the secondary when it fails is lost.

In a production environment, configure your monitoring and logging services to continuously verify replication status.

1. Read the replication status on both clusters.

   $ VAULT_ADDR=$PRIMARY_ADDR vault read -format=json sys/replication/dr/status \
       | jq '.data | {mode, state, merkle_root, last_dr_wal}' && \
       VAULT_ADDR=$DR_ADDR vault read -format=json sys/replication/dr/status \
       | jq '.data | {mode, state, merkle_root, last_remote_wal}'
   

   Example output:

   {
     "mode": "primary",
     "state": "running",
     "merkle_root": "93354c80418a9de361e07c4f1b11bf55434b7159",
     "last_dr_wal": 6384
   }
   {
     "mode": "secondary",
     "state": "stream-wals",
     "merkle_root": "93354c80418a9de361e07c4f1b11bf55434b7159",
     "last_remote_wal": 6384
   }
   

   The clusters are in sync when the secondary state is stream-wals, the merkle_root values match, and the secondary last_remote_wal equals the primary last_dr_wal.

2. Create a batch token on the current primary cluster.

   $ export DR_BATCH_TOKEN=$(VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$PRIMARY_ROOT_TOKEN \
       vault token create -role=failover-handler -ttl=8h -field=token) && \
       echo $DR_BATCH_TOKEN
   

   Example output:

   hvb.AAAAAQ...snip...I2QbVl1XH
   

   In a disaster scenario, you cannot perform this step because the cluster is offline. Learn how to automate batch token creation in the Prepare for DR failover tutorial.

Simulate the primary failure

Delete the running primary cluster to simulate a failure that leaves you no chance to demote it.

In a real outage, confirm that the primary is down before you fail over. Vault Enterprise deliberately does not support automatic promotion. Refer to the HashiCorp Validated Design for DR operations for guidance on triage and confirmation.

1. Delete the primary cluster container while it runs.

   $ docker rm -f vault-primary
   
   vault-primary
   

   The -f flag force-removes a running container. Docker sends the main process SIGKILL, so the primary gives Vault no chance to shut down cleanly.

2. Confirm the primary no longer responds.

   $ VAULT_ADDR=$PRIMARY_ADDR vault status
   
   Error checking seal status: Get "http://127.0.0.1:8200/v1/sys/seal-status": dial tcp 127.0.0.1:8200: connect: connection refused
   

3. Read the replication status on the DR secondary cluster.

   $ VAULT_ADDR=$DR_ADDR vault read -format=json sys/replication/dr/status | jq '.data'
   
   {
     "cluster_id": "261b95f0-23fd-3198-8192-e47ce93a7715",
     "connection_state": "transient_failure",
     "corrupted_merkle_tree": false,
     "known_primary_cluster_addrs": [
       "https://vault-primary:8201"
     ],
     "last_corruption_check_epoch": "-62135596800",
     "last_reindex_epoch": "0",
     "last_remote_wal": 575867,
     "last_start": "2026-10-22T14:11:05Z",
     "merkle_root": "ccabd881f9fde89d7cafffaac19cbd70e26bae99",
     "mode": "secondary",
     "primaries": [
       {
         "api_address": "http://vault-primary:8200",
         "clock_skew_ms": "-1",
         "cluster_address": "https://vault-primary:8201",
         "connection_status": "disconnected",
         "last_heartbeat": "2026-10-22T16:24:13Z",
         "last_heartbeat_duration_ms": "1",
         "replication_primary_canary_age_ms": "205"
       }
     ],
     "primary_cluster_addr": "https://vault-primary:8201",
     "secondary_id": "failback-secondary",
     "ssct_generation_counter": 1,
     "state": "stream-wals"
   }
   

   connection_state shows a replication failure, and connection_status shows the secondary disconnected after the primary cluster outage.

   The secondary still reports mode as secondary. It does not serve client requests until you promote it.

Promote the DR secondary cluster

Promoting a DR secondary requires a DR operation token. You generate the token on the DR secondary cluster. The batch token you created before the failure authenticates each call, and the unseal key authorizes the token generation.

1. Start the DR operation token generation and authenticate with the batch token.

   $ INIT=$(VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
       vault operator generate-root -dr-token -init -format=json)
   

2. Verify the nonce value.

   $ echo "$INIT" | jq -er .nonce
   

3. Verify the otp value.

   $ echo "$INIT" | jq -er .otp
   

4. Submit the unseal key to the DR secondary cluster.

   $ ENCODED=$(printf '%s' "$PRIMARY_UNSEAL_KEY" | VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault operator generate-root -dr-token -nonce=$(echo "$INIT" | jq -er .nonce) -format=json - \
      | jq -er '.encoded_token | select(length > 0)')
   

   The primary's unseal key works on the DR secondary because enabling replication replaced the secondary's unseal key with the primary's.

   The unseal threshold in this lab is 1, so a single submission completes the attempt. Do not submit the key twice.

   In production environments:

   • If the cluster uses auto-unseal, submit a recovery key instead of the unseal key.
   • If the threshold is higher than 1, each key holder repeats the command until progress reaches the threshold.

5. Decode the DR operation token.

   $ DR_OP_TOKEN=$(VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault operator generate-root -dr-token -decode=$ENCODED -otp=$(echo "$INIT" | jq -er .otp)) && \
      echo $DR_OP_TOKEN
   

   Example output:

   hvs.uIvW...snip...L6Glj
   

6. Promote the DR secondary cluster.

   $ VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault write sys/replication/dr/secondary/promote dr_operation_token=$DR_OP_TOKEN
   

   Example output:

   WARNING! The following warnings were returned from Vault:
   
     * This cluster is being promoted to a replication primary. Vault will be
     unavailable for a brief period and will resume service shortly.
   

   The promote endpoint has an optional force parameter that skips safety checks. Forcing a promotion can lose data that the primary did not replicate before it failed. This tutorial does not use force.

7. Verify the DR secondary cluster is now the primary cluster.

   $ VAULT_ADDR=$DR_ADDR vault read -format=json sys/replication/dr/status \
       | jq '.data | {mode, state, known_secondaries, secondaries}'
   

   Example output:

   {
     "mode": "primary",
     "state": "running",
     "known_secondaries": [],
     "secondaries": []
   }
   

   The mode value is primary, and state is running. The secondaries list is empty until you connect the replacement cluster.

8. Log in to the new primary cluster as oliver. When prompted, enter primary-pw at the Password (will be hidden): prompt.

   $ OLIVER_TOKEN=$(VAULT_ADDR=$DR_ADDR vault login -field=token -method=userpass username=oliver)
   

   The promoted cluster now serves requests.

Start the replacement cluster

HashiCups' primary cluster failed and is no longer available. Oliver opts to build a new cluster in the primary site. Once the new cluster is online, Oliver adds the new cluster as a DR secondary and configures replication. Once replication completes between the two clusters, Oliver can fail back operations to the primary site.

1. Start the replacement cluster.

   $ docker run -d --name new-cluster --hostname new-cluster \
       --network vault-primary-net \
       -p 8200:8200 \
       -p 8201:8201 \
       -e VAULT_LICENSE \
       -e VAULT_DISABLE_MLOCK=true \
       -e VAULT_API_ADDR=http://new-cluster:8200 \
       -e VAULT_CLUSTER_ADDR=https://new-cluster:8201 \
       -v $(pwd)/vault-config.hcl:/vault/config/vault-config.hcl \
       hashicorp/vault-enterprise:2.1.1-ent \
       server
   

   The replacement cluster uses the host ports that the deleted primary released. It joins vault-primary-net, where the promoted cluster vault-dr is also attached, so Docker resolves the name vault-dr from the new container.

2. Set an environment variable for the replacement cluster address.

   $ export NEW_ADDR=http://127.0.0.1:8200
   

3. Initialize the replacement cluster.

   $ NEW_INIT=$(VAULT_ADDR=$NEW_ADDR vault operator init -key-shares=1 -key-threshold=1 -format=json)
   

   Enabling a DR secondary requires an initialized and unsealed cluster. The HashiCorp Validated Design for DR initial setup lists both as prerequisites.

4. Export the replacement cluster's root token and unseal key as environment variables.

   $ NEW_UNSEAL_KEY=$(echo "$NEW_INIT" | jq -er '.unseal_keys_b64[0]') \
       NEW_ROOT_TOKEN=$(echo "$NEW_INIT" | jq -er '.root_token')
   

5. Unseal the replacement cluster.

   $ VAULT_ADDR=$NEW_ADDR vault operator unseal $NEW_UNSEAL_KEY
   

   Vault's unseal output is similar to the output from the earlier tutorials.

Connect the replacement cluster as the DR secondary

1. Generate a secondary activation token on the promoted cluster. This endpoint requires the sudo capability, so use the oliver token.

   $ SECONDARY_TOKEN=$(VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$OLIVER_TOKEN \
      vault write -format=json sys/replication/dr/primary/secondary-token id=new-secondary \
      | jq -er .wrap_info.token)
   

   The activation token expires after 30 minutes by default.

2. Enable DR replication on the replacement cluster.

   Warning

   This immediately clears all data in the replacement cluster.

   $ VAULT_ADDR=$NEW_ADDR VAULT_TOKEN=$NEW_ROOT_TOKEN \
      vault write sys/replication/dr/secondary/enable token=$SECONDARY_TOKEN
   

   Example output:

   WARNING! The following warnings were returned from Vault:
   
     * Vault has successfully found secondary information; it may take a while to
     perform setup tasks. Vault will be unavailable until these tasks and initial
     sync complete.
   

   The activation token carries the promoted cluster's API address. If the replacement cluster cannot resolve that address, add primary_api_addr=<promoted-cluster-api-address> to the command.

   Enabling replication replaced the replacement cluster's unseal key with the promoted cluster's unseal key. From now on, use $PRIMARY_UNSEAL_KEY to unseal the replacement cluster. The root token you created during initialization no longer works.

3. Verify the replacement cluster is a secondary and replicating.

   $ VAULT_ADDR=$NEW_ADDR vault read -format=json sys/replication/dr/status \
       | jq '.data | {mode, state, connection_state, merkle_root, last_remote_wal, primary_cluster_addr, secondary_id}'
   

   Example output:

   {
     "mode": "secondary",
     "state": "stream-wals",
     "connection_state": "ready",
     "merkle_root": "93354c80418a9de361e07c4f1b11bf55434b7159",
     "last_remote_wal": 0,
     "primary_cluster_addr": "https://vault-dr:8201",
     "secondary_id": "new-secondary"
   }
   

   The mode value is secondary, and state is stream-wals.

4. Verify the promoted cluster sees the replacement cluster.

   $ VAULT_ADDR=$DR_ADDR vault read -format=json sys/replication/dr/status | jq '.data'
   

   The secondaries list includes the replacement cluster with a connection_status of connected. The merkle_root matches the replacement cluster, and last_dr_wal equals the replacement cluster's last_remote_wal.

   Oliver can now follow HashiCups' failback procedure to bring the new cluster and Vault cluster topology back to the desired state.

   Tutorial state

   For the purposes of this tutorial series, do not perform a fail back.

Clean up

If you do not plan to continue with the rest of the tutorials in this series, clean up the resources from the Enable disaster recovery replication tutorial.

Run the following steps from the ~/tmp/dr-tutorial working directory to remove the containers, networks, files, and environment variables from this tutorial.

1. Stop and remove the containers, including their anonymous data volumes.

   $ docker rm -f -v new-cluster vault-dr
   

   The -f flag removes running containers. The -v flag removes the anonymous volumes that Docker creates for the /vault/file and /vault/logs paths.

2. Remove the Docker networks.

   $ docker network rm vault-primary-net vault-dr-net
   

3. Remove the configuration file and the working directory.

   $ rm vault-config.hcl && cd ~/tmp && rm -rf dr-tutorial
   

4. Unset the environment variables.

   $ unset VAULT_LICENSE PRIMARY_ADDR DR_ADDR PRI_INIT PRIMARY_UNSEAL_KEY \
       PRIMARY_ROOT_TOKEN DR_INIT DR_UNSEAL_KEY DR_ROOT_TOKEN SECONDARY_TOKEN \
       NEW_ADDR NEW_INIT NEW_ROOT_TOKEN NEW_UNSEAL_KEY
   

5. (Optional) Remove the Docker image.

   $ docker image rm hashicorp/vault-enterprise:2.1.1-ent
   

Summary

You have completed an unplanned failover. You simulated a disaster by deleting the running primary cluster, then promoted the DR secondary cluster to primary with a batch token and the unseal key. After the promotion, you connected a new cluster as the DR secondary. After the failover, you reviewed the replication status on both clusters to confirm the promoted cluster is the primary and the replacement cluster is replicating from it.

The batch token you used in this tutorial was created before the failure. In the next tutorial, you handle the scenario where no batch token is available — using the break glass workflow to promote the secondary with only the unseal key.

Help and reference

Learn more about disaster recovery concepts in the HashiCorp Well-Architected Framework.

• Design resilient systems - Disaster recovery
• Design resilient systems - Plan for resilience

For more information on Vault disaster recovery, refer to the Vault documentation.

• Replication support in Vault
• Seal/Unseal
• Tokens
• Policies
• Upgrading Vault - Replication installations
• Configuration - enable_unauthenticated_access
• operator generate-root command
• operator init command
• operator unseal command

For the API endpoints used in this tutorial, refer to the Vault API documentation.

• Vault DR replication API
• Vault replication API
• Token auth method API

For operational guidance, refer to the HashiCorp Validated Designs.

• HashiCorp Validated Design for DR operations
• HashiCorp Validated Design for DR initial setup
• HashiCorp Validated Design for backup and restore operations

To manage the policy and the batch token role with Terraform, refer to the Vault provider documentation.

• vault_policy
• vault_token_auth_backend_role