HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream
Set up DR replication

Prepare for DR failover

  • 14min
  • |
  • Enterprise
  • Vault

Enterprise only

Disaster recovery replication requires a Vault Enterprise license.

Once you set up your Vault primary and secondary clusters for disaster recovery replication, you need to plan for how to access your secondary cluster in the event of a primary cluster failure.

To promote your secondary cluster to a primary cluster and start servicing requests, you need a DR operation token to authorize the promotion. Generating that token takes two things:

  • A quorum of key holders, who enter their unseal keys (manual unsealing) or recovery keys (auto-unseal).
  • A batch token, created on the primary cluster before the failover event, to authenticate the token generation requests to the secondary cluster.

Without both, you cannot promote the secondary cluster.

Most organizations create the batch token ahead of a disaster, store it securely, and keep it available to on-call operators.

Prerequisites

This tutorial builds on top of previous tutorials. Complete the Enable disaster recovery replication tutorial before you begin.

This lab was tested on macOS using Apple silicon-based processors.

To perform the tasks described in this tutorial, you need to have:

Scenario

Oliver completed the initial setup and configuration of the HashiCups Vault Enterprise clusters.

Diagram showing the current two Vault cluster Vault topology

Once Olivery completed the cluster configuration, they verified replication on the secondary cluster using the sys/replication/dr/status endpoint.

{
  "cluster_id": "261b95f0-23fd-3198-8192-e47ce93a7715",
  "connection_state": "ready",
  "corrupted_merkle_tree": false,
  "known_primary_cluster_addrs": [
    "https://vault-primary:8201"
  ],
  "last_corruption_check_epoch": "-62135596800",
  "last_reindex_epoch": "1791565606",
  "last_remote_wal": 6384,
  "last_start": "2026-10-09T17:06:46Z",
  "merkle_root": "93354c80418a9de361e07c4f1b11bf55434b7159",
  "mode": "secondary",
  "primaries": [
    {
      "api_address": "http://vault-primary:8200",
      "clock_skew_ms": "-1",
      "cluster_address": "https://vault-primary:8201",
      "connection_status": "connected",
      "last_heartbeat": "2026-10-09T18:33:46Z",
      "last_heartbeat_duration_ms": "2",
      "replication_primary_canary_age_ms": "593"
    }
  ],
  "primary_cluster_addr": "https://vault-primary:8201",
  "secondary_id": "dr-secondary",
  "ssct_generation_counter": 0,
  "state": "stream-wals"
}

Oliver now needs to create a standard operating procedure to allow the HashiCups operations team to test failover operations and respond to unplanned outages.

Design considerations

There are several questions to consider when designing and planning for Vault disaster recovery failover tokens.

Cluster unseal strategy

You can configure your Vault clusters with either a manual unseal or auto unseal process.

The unseal strategy for your secondary cluster impacts both replication and the unseal process.

Impact on replication

If your secondary cluster does not remain in an unsealed state, replication will fail.

  • Auto unseal with an HSM or KMS local to the secondary cluster: The secondary cluster's unseal process is not tied to the primary cluster's site. Replication resumes after a restart, without waiting on key holders.
  • Auto unseal with a shared HSM or KMS region used by both clusters: If the secondary cluster relies on services running in the primary cluster's site, you will not be able to unseal the cluster during a disaster event without using seal HA.
  • Manual unseal: A restart of the cluster leaves the secondary cluster sealed and not replicating until key holders manually unseal the cluster.

Impact on promoting the secondary cluster

When you promote the secondary cluster to the primary cluster, you need to enter a quorum of unseal or recovery keys to generate a DR operation token.

  • Auto unseal with an HSM or KMS local to the secondary cluster: Use the recovery keys generated when you initialized the cluster.
  • Auto unseal with a shared HSM or KMS region used by both clusters: If your primary site's HSM or KMS is down, you will not be able to unseal the secondary cluster. You can use seal HA, but the backup KMS or HSM must be available during the primary HSM or KMS outage.
  • Manual unseal: Use the operator unseal keys generated when you initialized the secondary Vault cluster.

In all three scenarios, you need a token to authenticate to the necessary endpoints to begin the promotion process, in addition to the unseal or recovery keys.

Token TTL

When you create a token to authenticate to the secondary cluster to promote it, you need to decide on the TTL for the token.

Current security practices promote shorter TTLs to limit exposure if your token becomes compromised. A longer TTL eases the operational burden of rotating and storing the token, at the expense of increased exposure.

The batch token in this example expires after 8 hours. In production, the token has likely expired by the time a disaster occurs. Establish one of these approaches:

  • Manual rotation: An operator mints a new batch token at shift handoff, stores the token securely outside the primary cluster's failure domain.
  • Automated rotation: Mint a token with an automated process, such as a GitHub Actions or other CI/CD workflow, and store the token securely outside the primary cluster's failure domain.

Set up the lab

This tutorial uses the environment and configuration created during the previous tutorials.

You will also deploy Gitea to simulate automated token rotation. Gitea Actions function similarly to other Git-based automation workflows.

  1. Start Gitea on the primary cluster network.

    $ docker run -d --name gitea --network vault-primary-net \
 -p 3001:3000 -e GITEA__security__INSTALL_LOCK=true \
 gitea/gitea:latest

  2. Create an admin user.

    $ docker exec --user git gitea gitea admin user create \
    --username admin \
    --password password \
    --email admin@example.com \
    --admin

    Example output:

    New user 'admin' has been successfully created!

    This example uses a plaintext password to keep the lab simple. Do not use or store plaintext passwords in production.

  3. Create a token to connect a runner.

    $ RUNNER_TOKEN=$(docker exec --user git gitea gitea actions generate-runner-token)

  4. Create a runner configuration that starts job containers on the primary Vault cluster's network.

    $ tee runner-config.yaml << 'EOF'
container:
  network: "vault-primary-net"
EOF

    Defining the network for the runner ensures Gitea starts job containers on a network that can reach the primary Vault cluster.

  5. Start the runner.

    $ docker run -d --name gitea-runner --network vault-primary-net \
   -v /var/run/docker.sock:/var/run/docker.sock \
   -v gitea-runner-data:/data \
   -v "$(pwd)/runner-config.yaml:/config.yaml" \
   -e CONFIG_FILE=/config.yaml \
   -e GITEA_INSTANCE_URL=http://gitea:3000 \
   -e GITEA_RUNNER_REGISTRATION_TOKEN=$RUNNER_TOKEN \
   gitea/act_runner:latest

  6. Create a repository.

    $ curl -u admin:password -X POST http://127.0.0.1:3001/api/v1/user/repos \
   -H "Content-Type: application/json" \
   -d '{"name":"dr-automation","auto_init":true}'

  7. Open a new web browser (or browser tab) and navigate to https://webhook.site/.

  8. Copy the Your unique URL value, for example https://webhook.site/4774ed18-94f1-4a75-bf13-8dff46b11d20.

    Leave this site open for the remainder of the tutorial.

  9. Return to the terminal where you started Gitea.

  10. Export an environment variable for the URL.

    Example:

    $ export WEBHOOK_URL="https://webhook.site/4774ed18-94f1-4a75-bf13-8dff46b11d20"

Create a token for failover

Oliver needs to create a token that allows the operations team to generate a DR operation token to enable the secondary cluster to service client requests.

Perform these steps manually to generate the batch token and understand the entire workflow.

  1. Create a policy named dr-secondary-promotion that allows the promote, update-primary, and generate-operation-token paths. These paths cover DR failover, failback, and DR operation token generation.

    $ VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$PRIMARY_ROOT_TOKEN \
    vault policy write dr-secondary-promotion - <<'EOF'
# To promote the secondary cluster to be primary
path "sys/replication/dr/secondary/promote" {
  capabilities = ["update"]
}

# To update the assigned primary cluster
path "sys/replication/dr/secondary/update-primary" {
  capabilities = ["update"]
}

# Vault 2.0.0+ requires a token to initiate DR operation token generation
path "sys/replication/dr/secondary/generate-operation-token/attempt" {
  capabilities = ["create", "read", "update", "delete"]
}

# Vault 2.0.0+ requires a token to supply key shares
path "sys/replication/dr/secondary/generate-operation-token/update" {
  capabilities = ["create", "update"]
}

# Only if using integrated storage (raft) as the storage backend
path "sys/storage/raft/autopilot/state" {
  capabilities = ["update", "read"]
}
EOF

    Example output:

    Success! Uploaded policy: dr-secondary-promotion

  2. Create a token role named failover-handler that attaches the dr-secondary-promotion policy and creates non-renewable, orphan batch tokens.

    $ VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$PRIMARY_ROOT_TOKEN \
    vault write auth/token/roles/failover-handler \
    allowed_policies=dr-secondary-promotion \
    orphan=true \
    renewable=false \
    token_type=batch

    Example output:

    Success! Data written to: auth/token/roles/failover-handler

  3. Create a batch token for the failover-handler role with a TTL of 8 hours and store it in DR_BATCH_TOKEN.

    $ export DR_BATCH_TOKEN=$(VAULT_ADDR=$PRIMARY_ADDR VAULT_TOKEN=$PRIMARY_ROOT_TOKEN \
    vault token create -format=json -role=failover-handler -ttl=8h -field=token)

    You can access the token value in the DR_BATCH_TOKEN environment variable.

  4. Perform a token lookup to validate the token.

    $ VAULT_TOKEN=$PRIMARY_ROOT_TOKEN VAULT_ADDR=$PRIMARY_ADDR \
    vault token lookup $(echo $DR_BATCH_TOKEN | jq -r)

    Example output:

    Key                 Value
---                 -----
accessor            n/a
creation_time       1792009434
creation_ttl        8h
display_name        token
entity_id           n/a
expire_time         2026-10-15T04:23:54Z
explicit_max_ttl    0s
id                  hvb.AAAAAQJnaove...snip...jJpn0f-nIRtPJiHFGu3k7bLY_7dw
issue_time          2026-10-14T20:23:54Z
meta                <nil>
num_uses            0
orphan              true
path                auth/token/create/failover-handler
policies            [default dr-secondary-promotion]
renewable           false
role                failover-handler
ttl                 7h34m57s
type                batch

    The token values match the role settings you create earlier.

  5. Validate the token's capabilities against the sys/replication/dr/secondary/promote endpoint.

    $ VAULT_TOKEN=$PRIMARY_ROOT_TOKEN VAULT_ADDR=$PRIMARY_ADDR \
    vault token capabilities $(echo $DR_BATCH_TOKEN | jq -r) \
    sys/replication/dr/secondary/promote

    The token capabilities command returns update, which matches the dr-secondary-promotion policy.

    Now that Oliver understands how to create the token, they will set up an automation workflow to rotate the token.

Automate token rotation

Manually rotating a token to support DR failover adds operational overhead to the team and is error-prone. Consider how to automate token rotation and securely store the token.

This example uses Gitea to generate a token and write it to an external endpoint. This workflow is conceptually similar to using GitHub Actions or GitLab CI/CD with a local runner that has access to the primary Vault cluster.

  1. Clone the repository created in the Set up the lab section.

    $ git clone http://admin:password@127.0.0.1:3001/admin/dr-automation.git

    Example output:

    Cloning into 'dr-automation'...
remote: Enumerating objects: 3, done.
remote: Counting objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
Receiving objects: 100% (3/3), done.

    Beyond storing the automated workflow, you can store other artifacts to assist in the DR failover process such as documentation or scripts specific to your environment.

  2. Make a directory for the automated workflow.

    $ mkdir -p dr-automation/.gitea/workflows

  3. Write the Webhook.site URL to the dr-automation repository's secrets.

    $ curl -u admin:password -X PUT \
   http://127.0.0.1:3001/api/v1/repos/admin/dr-automation/actions/secrets/WEBHOOK_URL \
   -H "Content-Type: application/json" \
   -d '{"data":"'"$WEBHOOK_URL"'"}' \
   -w '\nHTTP %{http_code}\n'

    The curl command returns HTTP 201 validating the request completed.

  4. Write the address of the primary Vault cluster to the repository's secrets.

    $ curl -u admin:password -X PUT \
   http://127.0.0.1:3001/api/v1/repos/admin/dr-automation/actions/secrets/PRIMARY_ADDR \
   -H "Content-Type: application/json" \
   -d '{"data":"http://vault-primary:8200"}'

    Storing this as a secret or variable in your automation tool makes it easier to update if you change your Vault address later.

  5. Write the primary cluster's root token to the repository's secrets.

    $ curl -u admin:password -X PUT \
   http://127.0.0.1:3001/api/v1/repos/admin/dr-automation/actions/secrets/PRIMARY_ROOT_TOKEN \
   -H "Content-Type: application/json" \
   -d '{"data":"'"$PRIMARY_ROOT_TOKEN"'"}'

    This tutorial uses the token auth method for simplicity to replicate how an automated process might work.

    In a production environment, revoke the root token once you complete the configuration of your primary cluster and use a Vault auth method that supports your automation workflows. For example, if you run your automation workflows in AWS or Azure, use the corresponding auth method with a policy that allows creating the token.

  6. Create the action.

    Tutorial use only

    This action writes the Vault batch token to a publicly accessible site. This tutorial uses Webhook.site to show the output of an automated workflow that generates the token needed to start the Vault failover process.

    In a production environment, use a secrets manager that keeps the token secure. Examples include a separate, operations-only Vault cluster or a third-party secrets manager such as AWS Secrets Manager, Azure Key Vault, Kubernetes Secret, or Nomad variable.

    $ tee dr-automation/.gitea/workflows/rotate-dr-token.yml << 'EOF'
name: rotate-dr-batch-token
on:
  schedule:
    - cron: "*/5 * * * *"
  workflow_dispatch: {}

jobs:
  rotate:
    runs-on: ubuntu-latest
    steps:
      - name: Install vault CLI
        run: |
          ARCH=$(uname -m)
          case "$ARCH" in
            x86_64) VAULT_ARCH=amd64 ;;
            aarch64|arm64) VAULT_ARCH=arm64 ;;
            *) echo "unsupported arch: $ARCH"; exit 1 ;;
          esac
          curl -fsSL "https://releases.hashicorp.com/vault/2.1.1/vault_2.1.1_linux_${VAULT_ARCH}.zip" -o vault.zip
          unzip vault.zip && sudo mv vault /usr/local/bin/

      - name: Create batch DR token
        id: create_token
        env:
          VAULT_ADDR: ${{ secrets.PRIMARY_ADDR }}
          VAULT_TOKEN: ${{ secrets.PRIMARY_ROOT_TOKEN }}
        run: |
          TOKEN=$(vault token create -role=failover-handler -ttl=8h -field=token)
          echo "::add-mask::$TOKEN"
          echo "token=$TOKEN" >> "$GITHUB_OUTPUT"

      - name: Send token to external endpoint
        env:
          WEBHOOK_URL: ${{ secrets.WEBHOOK_URL }}
          DR_TOKEN: ${{ steps.create_token.outputs.token }}
        run: |
          curl -sS --fail -w '\nHTTP %{http_code}\n' -X POST \
            -H "Content-Type: text/plain" \
            --data-raw "$DR_TOKEN" \
            "$WEBHOOK_URL"
EOF

  7. Commit the action to the repository.

    $ git -C dr-automation add . && \
    git -C dr-automation commit -m "add DR token rotation workflow" && \
    git -C dr-automation push

    Example output:

    [main 60e631b] add DR token rotation workflow
 1 file changed, 48 insertions(+)
 create mode 100644 .gitea/workflows/rotate-dr-token.yml
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 12 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (5/5), 1.13 KiB | 1.13 MiB/s, done.
Total 5 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)
To http://127.0.0.1:3001/admin/dr-automation.git
   68fd5c6..60e631b  main -> main

  8. Run the action.

    $ curl -u admin:password -X POST \
   http://127.0.0.1:3001/api/v1/repos/admin/dr-automation/actions/workflows/rotate-dr-token.yml/dispatches \
   -H "Content-Type: application/json" \
   -d '{"ref":"main"}'

    The action pulls the job image and the Vault binary, uses the primary cluster's root token and address to generate a new batch token in Vault, and writes the batch token to the external endpoint.

  9. Return to the web browser session where Webhook.site is running. Webhook.site displays the token value generated by the action.

    Webhook.site displaying the batch token value generated from Vault by running the example action

Clean up

Remove the resources that the remaining tutorials do not need.

  1. Delete the Webhook.site URL.

    $ curl -sS -X DELETE "https://webhook.site/token/${WEBHOOK_URL##*/}" -w '\nHTTP %{http_code}\n'

    Example output:

    HTTP 204

  2. Delete the Gitea container, the runner container, and the runner volume.

    $ docker rm -f -v gitea gitea-runner && docker volume rm gitea-runner-data

  3. Delete the runner configuration file.

    $ rm runner-config.yaml

  4. Unset the Webhook.site and runner registration environment variables.

    $ unset WEBHOOK_URL RUNNER_TOKEN

If you do not plan to continue with the rest of the tutorials in this series, clean up the resources from the Enable disaster recovery replication tutorial.

Run the following steps from the ~/tmp/dr-tutorial working directory to remove the containers, networks, files, and environment variables from that tutorial.

  1. Stop and remove the containers, including their anonymous data volumes.

    $ docker rm -f -v vault-primary vault-dr

    The -f flag removes running containers. The -v flag removes the anonymous volumes that Docker creates for the /vault/file and /vault/logs paths.

  2. Remove the Docker networks.

    $ docker network rm vault-primary-net vault-dr-net

  3. Remove the configuration file and the working directory.

    $ rm vault-config.hcl && cd ~/tmp && rm -rf dr-tutorial

  4. Unset the environment variables.

    $ unset VAULT_LICENSE DR_BATCH_TOKEN PRIMARY_ADDR DR_ADDR PRI_INIT PRIMARY_UNSEAL_KEY \
    PRIMARY_ROOT_TOKEN DR_INIT DR_UNSEAL_KEY DR_ROOT_TOKEN SECONDARY_TOKEN

  5. (Optional) Remove the Docker image.

    $ docker image rm hashicorp/vault-enterprise:2.1.1-ent

Summary

You created the dr-secondary-promotion policy and the failover-handler batch token role on the primary cluster, generated a batch token with an 8-hour TTL, and verified the token's capabilities.

You then built a Gitea Actions workflow that creates a new batch token every five minutes and sends it to an external endpoint outside the primary cluster's failure domain.

Diagram showing the current two Vault cluster topology with batch token automation

In the next tutorial, you will use the batch token and a quorum of unseal or recovery keys to generate a DR operation token and promote the secondary cluster.

Help and reference

Learn more about disaster recovery concepts in the HashiCorp Well-Architected Framework.

For more information on Vault disaster recovery, refer to the Vault documentation.

For the API endpoints used in this tutorial, refer to the Vault API documentation.

For operational guidance, refer to the HashiCorp Validated Designs.

To manage the policy and the batch token role with Terraform, refer to the Vault provider documentation.

Previous

Enable disaster recovery replication

 Next

Planned DR failover