Perform a failback

After a failover, your original secondary cluster serves client requests. The original primary cluster serves as the DR secondary cluster.

In the Perform a planned DR failover tutorial, you learned how to fail over from a running primary cluster to the secondary cluster.

In this tutorial, you return to the original cluster topology by performing a planned failover in the opposite direction:

• Demote vault-dr
• Promote vault-primary
• Reconnect vault-dr as the DR secondary

Prerequisites

This tutorial builds on top of previous tutorials. Complete the Perform a planned DR failover tutorial before you begin.

This lab was tested on macOS using Apple silicon-based processors.

To perform the tasks described in this tutorial, you need to have:

• Vault Enterprise 2.x binary installed
• A Vault Enterprise license
• Docker to run Vault Enterprise
• jq installed

Scenario

Oliver performed a planned failover of HashiCups' Vault clusters. The original primary cluster now serves as a secondary cluster, and the original secondary cluster now serves client requests as the primary cluster.

Oliver now performs a planned failback so the original primary cluster is once again the primary.

Set up the lab

This tutorial uses the environment and configuration created during the previous tutorials.

• Complete the Enable disaster recovery replication tutorial
• Complete the Prepare for DR failover tutorial
• Complete the Perform a planned DR failover tutorial

Design considerations

The failback process is similar to the planned failover process.

Refer to the Design considerations section in the Perform a planned DR failover tutorial for more information.

Demote the original secondary cluster

Demote vault-dr, the current primary cluster, which previously served as the DR secondary cluster.

1. Verify the replication status on both clusters.

   $ VAULT_ADDR=$DR_ADDR vault read -format=json sys/replication/dr/status \
       | jq '.data | {mode, state, merkle_root, last_dr_wal}' && \
       VAULT_ADDR=$PRIMARY_ADDR vault read -format=json sys/replication/dr/status \
       | jq '.data | {mode, state, merkle_root, last_remote_wal}'
   

   Example output:

   {
     "mode": "primary",
     "state": "running",
     "merkle_root": "ccdf4fbe2cebdf40aed3d16a04329928de934405",
     "last_dr_wal": 573153
   }
   {
     "mode": "secondary",
     "state": "stream-wals",
     "merkle_root": "ccdf4fbe2cebdf40aed3d16a04329928de934405",
     "last_remote_wal": 573153
   }
   

   The state value on the secondary is stream-wals, the merkle_root values match, and last_dr_wal on vault-dr equals last_remote_wal on vault-primary.

   Checking the status of each cluster at different times shows different merkle_root values.

2. Create a batch token on the current primary cluster.

   $ export DR_BATCH_TOKEN=$(VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$PRIMARY_ROOT_TOKEN \
       vault token create -role=failover-handler -ttl=8h -field=token) && \
       echo $DR_BATCH_TOKEN
   

   Example output:

   hvb.AAAAAQIN05hEdz...snip...AAAAAQIN05hEdz
   

3. Log in to the current primary cluster as oliver. When prompted, enter primary-pw at the Password (will be hidden): prompt.

   $ OLIVER_TOKEN=$(VAULT_ADDR=$DR_ADDR vault login -field=token -method=userpass username=oliver) && \
      echo $OLIVER_TOKEN
   

   Example output:

   hvs.CAESIBn...snip...hkTkoQlYEjGAE
   

4. Demote vault-dr.

   $ VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$OLIVER_TOKEN \
      vault write -f sys/replication/dr/primary/demote
   

   Example output:

   WARNING! The following warnings were returned from Vault:
   
     * This cluster is being demoted to a replication secondary. Vault will be
     unavailable for a brief period and will resume service shortly.
   

5. Start the DR operation token generation on vault-primary and authenticate with the batch token.

   $ INIT=$(VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
       vault operator generate-root -dr-token -init -format=json) && \
       echo $INIT
   

   Example output:

   {
     "nonce": "1953d409-87b3-ba26-b9b2-c8cc4f178bfa",
     "started": true,
     "progress": 0,
     "required": 1,
     "complete": false,
     "encoded_token": "",
     "encoded_root_token": "",
     "pgp_fingerprint": "",
     "otp": "hkDd9...snip...uShqtUm",
     "otp_length": 28
   }
   

6. Submit the unseal key to vault-primary.

   $ ENCODED=$(printf '%s' "$PRIMARY_UNSEAL_KEY" | VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault operator generate-root -dr-token -nonce=$(echo "$INIT" | jq -er .nonce) -format=json - \
      | jq -er '.encoded_token | select(length > 0)') && echo $ENCODED
   

   Example output:

   AB03SnNg...snip...BfISYaOhEfFA
   

7. Decode the DR operation token.

   $ DR_OP_TOKEN=$(VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault operator generate-root -dr-token -decode=$ENCODED -otp=$(echo "$INIT" | jq -er .otp)) && \
      echo $DR_OP_TOKEN
   

   Example output:

   hvs.JVl...snip...rKeJy
   

Promote the original primary cluster

Promote vault-primary, the current secondary cluster, which previously served as the primary cluster.

1. Promote the current secondary cluster.

   $ VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault write sys/replication/dr/secondary/promote dr_operation_token=$DR_OP_TOKEN
   

   Example output:

   WARNING! The following warnings were returned from Vault:
   
     * This cluster is being promoted to a replication primary. Vault will be
     unavailable for a brief period and will resume service shortly.
   

2. Verify vault-primary is the primary cluster. The mode value is primary, and state is running.

   $ VAULT_ADDR=$PRIMARY_ADDR vault read -format=json sys/replication/dr/status \
       | jq '.data | {mode, state, secondaries}'
   

   Example output:

   {
     "mode": "primary",
     "state": "running",
     "secondaries": []
   }
   

Re-enable replication

The original primary cluster, vault-primary, is back as the primary Vault cluster. Re-enable replication to the secondary cluster.

1. Create a secondary activation token on the primary cluster.

   $ SECONDARY_TOKEN=$(VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$OLIVER_TOKEN \
      vault write -format=json sys/replication/dr/primary/secondary-token id=failback-secondary \
      | jq -er .wrap_info.token)
   

   The activation token expires after 30 minutes by default.

2. Start the DR operation token generation on vault-dr. vault-dr is a secondary again, so it needs its own DR operation token.

   $ INIT=$(VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
       vault operator generate-root -dr-token -init -format=json)
   

3. Submit the unseal key to vault-dr.

   $ ENCODED=$(printf '%s' "$PRIMARY_UNSEAL_KEY" | VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault operator generate-root -dr-token -nonce=$(echo "$INIT" | jq -er .nonce) -format=json - \
      | jq -er '.encoded_token | select(length > 0)')
   

4. Decode the DR operation token.

   $ DR_OP_TOKEN=$(VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault operator generate-root -dr-token -decode=$ENCODED -otp=$(echo "$INIT" | jq -er .otp))
   

5. Point vault-dr at vault-primary.

   $ VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault write sys/replication/dr/secondary/update-primary \
      dr_operation_token=$DR_OP_TOKEN token=$SECONDARY_TOKEN
   

   Example output:

   WARNING! The following warnings were returned from Vault:
   
     * Vault has successfully found secondary information; it may take a while to
     perform setup tasks. Vault will be unavailable until these tasks and initial
     sync complete.
   

6. Verify vault-dr is a secondary cluster that is replicating. The mode value is secondary, and state is stream-wals.

   $ VAULT_ADDR=$DR_ADDR vault read -format=json sys/replication/dr/status \
       | jq '.data | {mode, state, connection_state, merkle_root, last_remote_wal, primary_cluster_addr}'
   

   Example output:

   {
     "mode": "secondary",
     "state": "stream-wals",
     "connection_state": "ready",
     "merkle_root": "bdeb98f990db981c5c70bd601491040453727e44",
     "last_remote_wal": 568072,
     "primary_cluster_addr": "https://vault-primary:8201"
   }
   

7. Verify vault-primary lists vault-dr as a connected secondary. The connection_status value is connected.

   $ VAULT_ADDR=$PRIMARY_ADDR vault read -format=json sys/replication/dr/status | jq '.data'
   

   Example output:

   {
     "cluster_id": "261b95f0-23fd-3198-8192-e47ce93a7715",
     "corrupted_merkle_tree": false,
     "known_secondaries": [
       "failback-secondary"
     ],
     "last_corruption_check_epoch": "-62135596800",
     "last_dr_wal": 568128,
     "last_reindex_epoch": "0",
     "last_wal": 568128,
     "merkle_root": "d9ba2c6e2e46da8494245a114ef10bc509f5017f",
     "mode": "primary",
     "primary_cluster_addr": "",
     "secondaries": [
       {
         "api_address": "http://vault-dr:8200",
         "clock_skew_ms": "-1",
         "cluster_address": "https://vault-dr:8201",
         "connection_status": "connected",
         "last_heartbeat": "2026-10-22T14:12:30Z",
         "last_heartbeat_duration_ms": "1",
         "node_id": "failback-secondary",
         "replication_primary_canary_age_ms": "656"
       }
     ],
     "ssct_generation_counter": 2,
     "state": "running"
   }
   

8. Delete the DR operation token from the secondary cluster.

   $ VAULT_ADDR=$DR_ADDR VAULT_TOKEN=$DR_BATCH_TOKEN \
      vault write sys/replication/dr/secondary/operation-token/delete \
      dr_operation_token=$DR_OP_TOKEN
   

   Example output:

   Success! Data written to: sys/replication/dr/secondary/operation-token/delete
   

Clean up

If you do not plan to continue with the rest of the tutorials in this series, clean up the resources from the Enable disaster recovery replication tutorial.

Run the following steps from the ~/tmp/dr-tutorial working directory to remove the containers, networks, files, and environment variables from that tutorial.

1. Stop and remove the containers, including their anonymous data volumes.

   $ docker rm -f -v vault-primary vault-dr
   

   The -f flag removes running containers. The -v flag removes the anonymous volumes that Docker creates for the /vault/file and /vault/logs paths.

2. Remove the Docker networks.

   $ docker network rm vault-primary-net vault-dr-net
   

3. Remove the configuration file and the working directory.

   $ rm vault-config.hcl && cd ~/tmp && rmdir dr-tutorial
   

   The rmdir command fails if the directory still holds files, such as files from other tutorials in this series.

4. Unset the environment variables.

   $ unset VAULT_LICENSE PRIMARY_ADDR DR_ADDR PRI_INIT PRIMARY_UNSEAL_KEY \
       PRIMARY_ROOT_TOKEN DR_INIT DR_UNSEAL_KEY DR_ROOT_TOKEN SECONDARY_TOKEN
   

5. (Optional) Remove the Docker image.

   $ docker image rm hashicorp/vault-enterprise:2.1.1-ent
   

Summary

You have now failed back to your original Vault topology. The original primary cluster is once again the primary. In a production environment, you would complete this workflow when operations at your primary site become available again after a disaster.

Help and reference

Learn more about disaster recovery concepts in the HashiCorp Well-Architected Framework.

• Design resilient systems - Disaster recovery
• Design resilient systems - Plan for resilience

For more information on Vault disaster recovery, refer to the Vault documentation.

• Replication support in Vault
• Seal/Unseal
• Tokens
• Policies
• Upgrade replicated Vault deployments
• Configuration - enable_unauthenticated_access
• operator generate-root command
• login command
• read command
• token create command
• token revoke command
• write command

For the API endpoints used in this tutorial, refer to the Vault API documentation.

• Vault DR replication API
• Vault replication API
• Token auth method API

For operational guidance, refer to the HashiCorp Validated Designs.

• HashiCorp Validated Design for DR operations
• HashiCorp Validated Design for DR initial setup
• HashiCorp Validated Design for backup and restore operations

To manage the policy and the batch token role with Terraform, refer to the Vault provider documentation.

• vault_policy
• vault_token_auth_backend_role