Perform a planned DR failover

A planned failover in Vault disaster recovery (DR) replication moves operations from your primary Vault cluster to your secondary cluster on your schedule, before a failure occurs. In this tutorial, you use a batch token to perform a planned failover between two Vault clusters.

So far in this tutorial series, you deployed two Vault clusters, enabled DR replication between them, and created the batch token that authenticates against the DR endpoints.

You can never fully plan for a disaster, but a planned failover lets you:

• Practice your business continuity and disaster recovery runbooks before a disaster occurs.
• Test and manage Vault cluster upgrades with DR replication.
• Fail over ahead of predictable events such as hurricanes or blizzards.

In the Prepare for DR failover tutorial, you learned how to create a batch token to assist with the failover. Most organizations create the batch token ahead of time, store it securely, and keep it available to on-call operators.

In this tutorial, you use that token to generate a DR operation token and promote the DR secondary cluster.

Prerequisites

This tutorial builds on top of previous tutorials. Complete the Prepare for DR failover tutorial before you begin.

This lab was tested on macOS using Apple silicon-based processors.

To perform the tasks described in this tutorial, you need to have:

• Vault Enterprise 2.x binary installed
• A Vault Enterprise license
• Docker to run Vault Enterprise
• jq installed

Scenario

So far, Oliver has deployed two Vault clusters, configured DR replication, and learned how to create a batch token to start the failover process.

Oliver performs a planned failover from the primary cluster to the secondary cluster while both clusters are operational.

Design considerations

A planned failover moves production traffic to the secondary site on purpose. Plan for the systems around Vault as well as the Vault commands such as monitoring, logging, operator access, client access, and client routing all change when the secondary cluster becomes the primary.

Before you promote

• Verify that replication is healthy and the clusters are in sync.
• Take a snapshot of the primary cluster's data before the failover. If you use integrated storage (Raft), use vault operator raft snapshot save to write a backup to a location outside the cluster's failure domain.
• Notify Vault consumers and stakeholders of the planned outage.
• Demote the original primary before you promote the secondary.
• Update your load balancer, DNS TTL, DNS, or service discovery so clients find the correct cluster.

Monitoring and alerting

When the secondary cluster becomes the primary, you need to ensure your metric reporting reflects the new primary cluster.

• Confirm that dashboards and alerts for the secondary cluster apply active-primary thresholds, such as request rate and lease count. Standby thresholds are often looser.
• Pause or snooze alerts for the original primary during the failover window so the demotion does not page on-call operators. Leave the alerts active if you want to test them as part of the exercise.
• Confirm that alerts from the promoted cluster route to the right on-call rotation.
• Check that your load balancer and monitoring health checks treat a DR secondary as healthy and an active primary as healthy.

Similarly, ensure you have a plan to monitor the appropriate logs before and after a failover.

• Confirm that the log hosts in the secondary site are reachable and ingesting logs before you start.
• Confirm that the secondary site's logging pipeline can handle the volume of an active primary.
• After promotion, verify that audit and operational logs from the new primary arrive in your log platform.

Cluster access

Once your secondary cluster becomes the primary cluster, you have to ensure operators, administrators, and clients are able to access the cluster.

• Test access to the jump hosts or bastions in the secondary site before the failover window. Operators need a working network path to the secondary cluster during the promotion.
• If you use HashiCorp Boundary for session access, confirm that your targets in the secondary site are defined and that operators can open sessions to them.
• Decide how clients find the active cluster, for example through DNS or a load balancer.
• If you use DNS, lower the TTL ahead of the failover so clients pick up the change quickly.
• Test that applications, agents, and CI/CD systems can reach the secondary cluster's address.

Success criteria and abort point

Agree on what success looks like, and when you stop, before the failover window opens.

• The promoted cluster reports that it is a primary and is running.
• A smoke test passes against the new primary. For example, an operator authenticates, reads a known secret, and renews a lease.
• Applications and agents connect to the promoted cluster, and their error rates return to baseline.
• Monitoring and logs show the promoted cluster as active.
• If your plan keeps the original primary online, it rejoins as a secondary and replication resumes.

Your success criteria should have specific points for when to abort the planned failover and return to regular operation.

• Set a time limit for the whole window and for the promotion step.
• Name one person who calls the abort.
• List the abort triggers, such as replication out of sync, a failed smoke test, or error rates that do not recover within the time limit.
• Know where the clean abort ends. Before you demote the original primary, you can stop without changing replication roles. After the demotion, reversing the failover means running the failback procedure, so decide in advance whether you fail back or fix forward.

Set up the lab

This tutorial uses the environment and configuration created during the previous tutorials.

• Complete the Enable disaster recovery replication tutorial
• Complete the Prepare for DR failover tutorial

Demote the primary cluster

When performing a planned failover, the primary cluster is still operational and serving requests. The first step is to demote the primary cluster, causing a brief outage until you promote the secondary cluster.

1. Verify replication between the clusters is up to date.

   $ VAULT_ADDR=$DR_ADDR vault read -format=json sys/replication/dr/status | jq '.data'
   

   Example output:

   {
     "cluster_id": "261b95f0-23fd-3198-8192-e47ce93a7715",
     "connection_state": "ready",
     "corrupted_merkle_tree": false,
     "known_primary_cluster_addrs": [
       "https://vault-primary:8201"
     ],
     "last_corruption_check_epoch": "-62135596800",
     "last_reindex_epoch": "1791565606",
     "last_remote_wal": 6384,
     "last_start": "2026-10-09T17:06:46Z",
     "merkle_root": "93354c80418a9de361e07c4f1b11bf55434b7159",
     "mode": "secondary",
     "primaries": [
       {
         "api_address": "http://vault-primary:8200",
         "clock_skew_ms": "-1",
         "cluster_address": "https://vault-primary:8201",
         "connection_status": "connected",
         "last_heartbeat": "2026-10-09T18:33:46Z",
         "last_heartbeat_duration_ms": "2",
         "replication_primary_canary_age_ms": "593"
       }
     ],
     "primary_cluster_addr": "https://vault-primary:8201",
     "secondary_id": "dr-secondary",
     "ssct_generation_counter": 0,
     "state": "stream-wals"
   }
   

   The state value shows the current replication state. stream-wals means the secondary cluster is up to date with the primary cluster.

2. Create a batch token on the primary cluster.

   $ export DR_BATCH_TOKEN=$(VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$PRIMARY_ROOT_TOKEN \
       vault token create -role=failover-handler -ttl=8h -field=token) && \
       echo $DR_BATCH_TOKEN
   

   Example output:

   hvb.AAAAAQJP3Up...snip...MHlZpUZoymdnxgVqaPg
   

   At this stage, you would stop traffic to your primary cluster by redirecting your load balancer or updating DNS. If you rely on DNS for your clients to connect to Vault, make sure to lower the TTL for the primary cluster's DNS record before demoting your primary cluster.

3. Log in to the primary cluster as oliver. When prompted, enter primary-pw at the Password (will be hidden): prompt.

   $ OLIVER_TOKEN=$(VAULT_ADDR=$PRIMARY_ADDR vault login -field=token -method=userpass username=oliver) && \
       echo $OLIVER_TOKEN
   

   Example output:

   hvs.CAESIIRaEF...snip...ATEQhbIC
   

   In a production environment, you should revoke your root token once the cluster has enough initial configuration for your Vault administrators to log in.

4. Demote the primary cluster.

   $ VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$OLIVER_TOKEN \
      vault write -f sys/replication/dr/primary/demote
   

   Example output:

   WARNING! The following warnings were returned from Vault:
   
   * This cluster is being demoted to a replication secondary. Vault will be
   unavailable for a brief period and will resume service shortly.
   

   Your Vault clusters are now unavailable to client requests.

Promote the secondary cluster

Now that you have demoted the primary cluster, you need to promote the secondary cluster so it can serve client requests.

1. Authenticate to the DR secondary cluster using the batch token and start the workflow to create a DR operation token.

   $ INIT=$(VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
       vault operator generate-root -dr-token -init -format=json)
   

2. Verify the nonce value.

   $ echo "$INIT" | jq -er .nonce
   13a357a2...snip...8442493a9bf7
   

3. Verify the otp value.

   $ echo "$INIT" | jq -er .otp
   otIcVLgb...snip...xnOLmlxwt
   

4. Submit the primary cluster's unseal key to the DR secondary cluster.

   $ ENCODED=$(printf '%s' "$PRIMARY_UNSEAL_KEY" | VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault operator generate-root -dr-token -nonce=$(echo "$INIT" | jq -er .nonce) -format=json - \
      | jq -er '.encoded_token | select(length > 0)')
   

   This tutorial uses unseal keys for simplicity in setting up the lab.

   In production environments:

   • If the primary cluster uses auto-unseal, submit a recovery key instead of the unseal key.
   • If the unseal threshold is higher than 1, each key holder repeats the command until progress reaches the defined threshold.

5. Decode the DR operation token.

   $ DR_OP_TOKEN=$(VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault operator generate-root -dr-token -decode=$ENCODED -otp=$(echo "$INIT" | jq -er .otp)) && \
      echo $DR_OP_TOKEN
   

   Example output:

   hvs.uIvWV...snip...lDL6Glj
   

6. Promote the DR secondary cluster.

   $ VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault write sys/replication/dr/secondary/promote dr_operation_token=$DR_OP_TOKEN
   

   Example output:

   WARNING! The following warnings were returned from Vault:
   
     * This cluster is being promoted to a replication primary. Vault will be
     unavailable for a brief period and will resume service shortly.
   

7. Verify the DR secondary cluster is now the primary cluster.

   $ VAULT_ADDR=$DR_ADDR vault read -format=json sys/replication/dr/status | jq '.data'
   

   Example output:

   {
     "cluster_id": "261b95f0-23fd-3198-8192-e47ce93a7715",
     "corrupted_merkle_tree": false,
     "known_secondaries": [],
     "last_corruption_check_epoch": "-62135596800",
     "last_dr_wal": 6384,
     "last_reindex_epoch": "0",
     "last_wal": 6384,
     "merkle_root": "93354c80418a9de361e07c4f1b11bf55434b7159",
     "mode": "primary",
     "primary_cluster_addr": "",
     "secondaries": [],
     "ssct_generation_counter": 0,
     "state": "running"
   }
   

   The mode value is primary, and state is running. The secondaries list is empty until you connect the demoted cluster back as a secondary.

Reconnect the old primary as a secondary

After promoting the DR secondary cluster, reconnect the old primary cluster as a DR secondary of the new primary cluster.

1. Generate a secondary activation token on the new primary cluster.

   $ SECONDARY_TOKEN=$(VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$OLIVER_TOKEN \
      vault write -format=json sys/replication/dr/primary/secondary-token id=old-primary \
      | jq -er .wrap_info.token)
   

   The activation token expires after 30 minutes by default.

2. Authenticate to the old primary cluster using the batch token to start the workflow to create a DR operation token. The old primary is now a DR secondary, so it needs its own DR operation token.

   $ INIT=$(VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
       vault operator generate-root -dr-token -init -format=json)
   

3. Verify the nonce value.

   $ echo "$INIT" | jq -er .nonce
   

4. Verify the otp value.

   $ echo "$INIT" | jq -er .otp
   

5. Submit the old primary cluster's unseal key and capture the encoded DR operation token.

   $ ENCODED=$(printf '%s' "$PRIMARY_UNSEAL_KEY" | VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault operator generate-root -dr-token -nonce=$(echo "$INIT" | jq -er .nonce) -format=json - \
      | jq -er '.encoded_token | select(length > 0)')
   

6. Decode the DR operation token.

   $ DR_OP_TOKEN=$(VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault operator generate-root -dr-token -decode=$ENCODED -otp=$(echo "$INIT" | jq -er .otp))
   

7. Point the old primary cluster at the new primary cluster.

   $ VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault write sys/replication/dr/secondary/update-primary \
      dr_operation_token=$DR_OP_TOKEN token=$SECONDARY_TOKEN
   

   Example output:

   WARNING! The following warnings were returned from Vault:
   
     * Vault has successfully found secondary information; it may take a while to
     perform setup tasks. Vault will be unavailable until these tasks and initial
     sync complete.
   

8. Verify the old primary cluster is a secondary and replicating data.

   $ VAULT_ADDR=$PRIMARY_ADDR vault read -format=json sys/replication/dr/status | jq '.data'
   

   Example output:

   {
     "cluster_id": "261b95f0-23fd-3198-8192-e47ce93a7715",
     "connection_state": "ready",
     "corrupted_merkle_tree": false,
     "known_primary_cluster_addrs": [
       "https://vault-dr:8201"
     ],
     "last_corruption_check_epoch": "-62135596800",
     "last_reindex_epoch": "0",
     "last_remote_wal": 504115,
     "last_start": "2026-10-20T20:44:47Z",
     "merkle_root": "4cb50cc5581a67268a94fd28aceefd1043ca6020",
     "mode": "secondary",
     "primaries": [
       {
         "api_address": "http://vault-dr:8200",
         "clock_skew_ms": "-1",
         "cluster_address": "https://vault-dr:8201",
         "connection_status": "connected",
         "last_heartbeat": "2026-10-20T20:45:42Z",
         "last_heartbeat_duration_ms": "1",
         "replication_primary_canary_age_ms": "727"
       }
     ],
     "primary_cluster_addr": "https://vault-dr:8201",
     "secondary_id": "old-primary",
     "ssct_generation_counter": 0,
     "state": "stream-wals"
   }
   

   The mode value is secondary, and state is stream-wals.

9. Verify the new primary cluster sees the old primary cluster.

   $ VAULT_ADDR=$DR_ADDR vault read -format=json sys/replication/dr/status | jq '.data'
   

   The secondaries list includes the old primary cluster with a connection_status of connected.

10. Delete the DR operation token from the old primary cluster.

    $ VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
       vault write sys/replication/dr/secondary/operation-token/delete \
       dr_operation_token=$DR_OP_TOKEN
    

    Example output:

    Success! Data written to: sys/replication/dr/secondary/operation-token/delete
    

11. (Optional) Update the cluster address in the Gitea action to connect to the new primary cluster.

    $ sed -i .bak 's/PRIMARY_ADDR/DR_ADDR/g' dr-automation/.gitea/workflows/rotate-dr-token.yml && \
        grep -n 'DR_ADDR' dr-automation/.gitea/workflows/rotate-dr-token.yml
    

    Example output:

    25:          VAULT_ADDR: ${{ secrets.DR_ADDR }}
    

    This step is optional for the tutorial, but in a production scenario you need to update clients and other services, such as the batch token automation workflow, to connect to the new primary cluster.

Clean up

1. Remove the cached token that vault login saved on your machine.

   $ rm ~/.vault-token
   

If you do not plan to continue with the rest of the tutorials in this series, clean up the resources from the Enable disaster recovery replication tutorial.

Run the following steps from the ~/tmp/dr-tutorial working directory to remove the containers, networks, files, and environment variables from that tutorial.

1. Stop and remove the containers, including their anonymous data volumes.

   $ docker rm -f -v vault-primary vault-dr
   

   The -f flag removes running containers. The -v flag removes the anonymous volumes that Docker creates for the /vault/file and /vault/logs paths.

2. Remove the Docker networks.

   $ docker network rm vault-primary-net vault-dr-net
   

3. Remove the configuration file and the working directory.

   $ rm vault-config.hcl && cd ~/tmp && rm -rf dr-tutorial
   

4. Unset the environment variables.

   $ unset VAULT_LICENSE PRIMARY_ADDR DR_ADDR PRI_INIT PRIMARY_UNSEAL_KEY \
       PRIMARY_ROOT_TOKEN DR_INIT DR_UNSEAL_KEY DR_ROOT_TOKEN SECONDARY_TOKEN
   

5. (Optional) Remove the Docker image.

   $ docker image rm hashicorp/vault-enterprise:2.1.1-ent
   

Summary

You have completed a planned failover from the primary cluster to the secondary cluster. After the failover, you reviewed the replication status to confirm the original secondary cluster is now the primary cluster, and the original primary cluster is now a secondary.

In a production environment, you would complete this workflow when a predictable event puts operations at your primary site at risk, or when you need to perform maintenance on your primary cluster, such as a version upgrade.

Help and reference

Learn more about disaster recovery concepts in the HashiCorp Well-Architected Framework.

• Design resilient systems - Disaster recovery
• Design resilient systems - Plan for resilience

For more information on Vault disaster recovery, refer to the Vault documentation.

• Replication support in Vault
• Tokens
• Policies
• Upgrading Vault - Replication installations
• Configuration - enable_unauthenticated_access
• operator generate-root command

For the API endpoints used in this tutorial, refer to the Vault API documentation.

• Vault DR replication API
• Vault replication API
• Token auth method API

For operational guidance, refer to the HashiCorp Validated Designs.

• HashiCorp Validated Design for DR operations
• HashiCorp Validated Design for DR initial setup
• HashiCorp Validated Design for backup and restore operations

To manage the policy and the batch token role with Terraform, refer to the Vault provider documentation.

• vault_policy
• vault_token_auth_backend_role