HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream
Set up DR replication

Perform a break glass DR promotion

  • 12min
  • |
  • Enterprise
  • Vault

Enterprise only

Disaster recovery replication requires a Vault Enterprise license.

In Vault 2.x and later, the generate-operation-token endpoints require a Vault token. If you never created a batch token, it expired, or you lost it, you cannot generate a DR operation token the standard way.

A break glass promotion in Vault disaster recovery (DR) replication promotes a DR secondary cluster to primary when the primary has failed and you have no token to authenticate the DR operation token requests.

The break glass path uses the cluster's unseal key and a temporary enable_unauthenticated_access setting in the DR secondary's server configuration.

In this tutorial, you delete the promoted cluster, turn on the setting, generate a DR operation token with no Vault token, promote the surviving DR secondary, and turn the setting off.

Break glass is a last resort. A prepared batch token is the supported approach because it keeps the DR secondary's API closed to unauthenticated requests. Use this tutorial to prepare for scenarios when that approach is not available.

Prerequisites

This tutorial builds on top of previous tutorials. Complete the Perform an unplanned DR failover tutorial before you begin.

This lab was tested on macOS using Apple silicon-based processors.

To perform the tasks described in this tutorial, you need to have:

Scenario

Oliver runs HashiCups' Vault clusters. After the last failover, vault-dr is the primary and new-cluster is the DR secondary. The original primary cluster is no longer available, replaced by a new cluster.

Diagram showing the current two Vault cluster Vault topology with batch token automation

Now Oliver wants to learn how to promote a cluster if the batch token automation workflow fails, or the token expires before the HashiCups team promptes the secondary cluster.

Oliver will replicate a failure of vault-dr and it does not come back. Oliver has no batch token, so they cannot authenticate the request that generates a DR operation token. Oliver does have the unseal key. Oliver plans to use the break glass workflow on new-cluster to generate the DR operation token, promote new-cluster, and close break glass access.

Set up the lab

This tutorial uses the environment and configuration created during the previous tutorials.

Design considerations

Break glass versus a prepared batch token

Both paths end with a DR operation token and a promote call. They differ in what protects the token generation.

  • With a prepared batch token, the secondary's API stays closed to unauthenticated requests, and the token plus the unseal key authorize the generation.
  • With break glass, you open the API for the generation endpoints and rely on the unseal key alone.

Learn how to prepare the batch token ahead of time in the Prepare for DR failover tutorial.

Key quorum dependency

Break glass removes the token requirement. It does not remove the key requirement. Vault 2.0.0 and later requires the key fragments in addition to a token, as the 2.0.0 release announcement describes. Without enough key holders, you cannot finish the generation.

If the cluster uses auto-unseal, submit recovery keys instead of unseal keys.

Unauthenticated access exposure

While enable_unauthenticated_access includes generate-operation-token, any caller who can reach the secondary's API can start a DR operation token generation attempt. Only one attempt can run at a time, so an unwanted attempt blocks yours until someone cancels it. CVE-2026-5807, fixed in Vault 2.0.0, describes the same denial of service. An attacker still needs the key fragments to finish an attempt.

Keep the window short. Turn the setting on, generate the token, and turn the setting off before you do anything else.

If someone else's attempt blocks you, cancel it with vault operator generate-root -dr-token -cancel and start again.

Apply the setting on a running server

The configuration documentation states that Vault applies enable_unauthenticated_access when you send the server SIGHUP. The sys/config/reload endpoint does not apply it, because that endpoint only reloads the license.

In this lab, you send SIGHUP with Docker. On a Linux host with systemd, the Vault SIGHUP behavior support article says to run systemctl reload vault.service.

Simulate the primary failure

Delete the running promoted cluster to simulate a failure that leaves you no chance to demote it.

In a real outage, confirm that the primary is down before you fail over. Vault Enterprise deliberately does not support automatic promotion. Refer to the HashiCorp Validated Design for DR operations for guidance on triage and confirmation.

  1. Read the replication status on both clusters.

    $ VAULT_ADDR=$DR_ADDR vault read -format=json sys/replication/dr/status \
    | jq '.data | {mode, state, merkle_root, last_dr_wal}' && \
    VAULT_ADDR=$NEW_ADDR vault read -format=json sys/replication/dr/status \
    | jq '.data | {mode, state, merkle_root, last_remote_wal}'

    Example output:

    {
  "mode": "primary",
  "state": "running",
  "merkle_root": "d47091057d8a01f26f5f367256c0b5e7f8001621",
  "last_dr_wal": 593631
}
{
  "mode": "secondary",
  "state": "stream-wals",
  "merkle_root": "d47091057d8a01f26f5f367256c0b5e7f8001621",
  "last_remote_wal": 593631
}

    The clusters are in sync when the secondary state is stream-wals, the merkle_root values match, and the secondary last_remote_wal equals the primary last_dr_wal.

  2. Delete the vault-dr container.

    $ docker rm -f vault-dr

vault-dr

    The -f flag force-removes a running container. Docker sends the main process SIGKILL, so Vault has no chance to shut down cleanly.

  3. Confirm the primary no longer responds.

    $ VAULT_ADDR=$DR_ADDR vault status

Error checking seal status: Get "http://127.0.0.1:8210/v1/sys/seal-status": dial tcp 127.0.0.1:8210: connect: connection refused

  4. Read the replication status on the DR secondary cluster.

    $ VAULT_ADDR=$NEW_ADDR vault read -format=json sys/replication/dr/status \
    | jq '.data | {mode, state, connection_state, known_primary_cluster_addrs}'

    Example output:

    {
  "mode": "secondary",
  "state": "stream-wals",
  "connection_state": "transient_failure",
  "known_primary_cluster_addrs": [
    "https://vault-dr:8201"
  ]
}

    connection_state shows a replication failure after the primary outage. The secondary still reports mode as secondary. It does not serve client requests until you promote it.

Start break glass access

Add enable_unauthenticated_access = ["generate-operation-token"] to the configuration of new-cluster only.

The container reads every .hcl file in /vault/config. The main file vault-config.hcl is a bind mount from your host. Instead of editing that file, you write a second file inside the container. Deleting the second file later turns the setting off, and your host file never changes.

  1. Write the break glass configuration file in the new-cluster container.

    $ printf '%s\n' 'enable_unauthenticated_access = ["generate-operation-token"]' \
    | docker exec -i new-cluster sh -c 'cat > /vault/config/break-glass.hcl'

  2. Confirm the file content.

    $ docker exec new-cluster cat /vault/config/break-glass.hcl

enable_unauthenticated_access = ["generate-operation-token"]

  3. Send SIGHUP to the Vault server in the new-cluster container.

    $ docker kill --signal=HUP new-cluster

new-cluster

    Despite its name, docker kill --signal=HUP does not stop the container. The command sends the signal you name to the main process. Refer to the docker container kill documentation for details.

  4. Confirm Vault reloaded its configuration.

    $ docker logs --tail 10 new-cluster

    Example output:

    ==> Vault reload triggered
2026-10-22T20:43:58.580Z [INFO]  storage.raft: reloaded raft config: settings="{10240 2m0s 8192 5s 5s}"
2026-10-22T20:43:58.580Z [WARN]  replication.wal.aop: write controller disabled
2026-10-22T20:43:58.580Z [ERROR] core: license reload triggered but license path is empty
license reload triggered but license path is empty

    Look for ==> Vault reload triggered. If the line is missing, Vault did not receive the signal.

  5. Confirm the DR secondary now answers generation requests with no token.

    $ VAULT_ADDR=$NEW_ADDR vault operator generate-root -dr-token -status

    Example output:

    Nonce         n/a
Started       false
Progress      0/1
Complete      false
OTP Length    28

    The command reads the generation status and changes nothing.

Generate the DR operation token

The next steps use the same generation flow as the unplanned DR failover tutorial, without the batch token.

  1. Start the DR operation token generation.

    $ INIT=$(VAULT_ADDR=$NEW_ADDR \
    vault operator generate-root -dr-token -init -format=json)

  2. Verify the nonce value.

    $ echo "$INIT" | jq -er .nonce

  3. Verify the otp value.

    $ echo "$INIT" | jq -er .otp

  4. Submit the unseal key to the DR secondary cluster.

    $ ENCODED=$(printf '%s' "$PRIMARY_UNSEAL_KEY" | VAULT_ADDR=$NEW_ADDR \
    vault operator generate-root -dr-token -nonce=$(echo "$INIT" | jq -er .nonce) -format=json - \
    | jq -er '.encoded_token | select(length > 0)')

    The unseal threshold in this lab is 1, so a single submission completes the attempt. Do not submit the key twice.

    In production environments:

    • If the cluster uses auto-unseal, submit a recovery key instead of the unseal key.
    • If the threshold is higher than 1, each key holder repeats the command until progress reaches the threshold.

  5. Decode the DR operation token. Leave break glass access on until this step finishes, because the decode command contacts the secondary to read the OTP length.

    $ DR_OP_TOKEN=$(VAULT_ADDR=$NEW_ADDR \
    vault operator generate-root -dr-token -decode=$ENCODED -otp=$(echo "$INIT" | jq -er .otp)) && \
    echo $DR_OP_TOKEN

    Example output:

    hvs.uIvW...snip...L6Glj

Promote the DR secondary cluster

Promote new-cluster with the DR operation token.

  1. Promote the DR secondary cluster.

    $ VAULT_ADDR=$NEW_ADDR \
    vault write sys/replication/dr/secondary/promote dr_operation_token=$DR_OP_TOKEN

    Example output:

    WARNING! The following warnings were returned from Vault:

  * This cluster is being promoted to a replication primary. Vault will be
  unavailable for a brief period and will resume service shortly.

    The promote endpoint has an optional force parameter that skips safety checks. Forcing a promotion can lose data that the primary did not replicate before it failed. This tutorial does not use force.

  2. Verify the DR secondary cluster is now the primary cluster.

    $ VAULT_ADDR=$NEW_ADDR vault read -format=json sys/replication/dr/status \
    | jq '.data | {mode, state, known_secondaries, secondaries}'

    Example output:

    {
  "mode": "primary",
  "state": "running",
  "known_secondaries": [],
  "secondaries": []
}

    The mode value is primary, and state is running. The secondaries list is empty until you connect a new secondary cluster.

  3. Log in to the new primary cluster as oliver. When prompted, enter primary-pw at the Password (will be hidden): prompt.

    $ VAULT_ADDR=$NEW_ADDR vault login -method=userpass username=oliver

Key                    Value
---                    -----
token                  hvs.CAESIOBjDquJ0CScDnqjGOEqFb4FUUqU6ylwwiK88CpK2_yHGiMKHGh2cy5obmh4ZU16dW8wVFB3VlRzSHZhTEZjVU8QzVIYBA
token_accessor         7UaijxSMrUpSBOMNyFgN8C5M
token_duration         768h
token_renewable        true
token_policies         ["default" "dr-admin"]
identity_policies      []
policies               ["default" "dr-admin"]
token_meta_username    oliver

    Oliver can log in because the DR secondary replicated the userpass auth method and its users from the failed primary. The promoted cluster now serves requests.

Turn off break glass access

You have the DR operation token, so close the window now. The setting only covers the generation endpoints. You do not need it to promote the cluster.

  1. Remove Oliver's cached token.

    $ rm ~/.vault-token

    The policy attached to the token allows interacting with the Vault DR endpoints.

  2. Delete the break glass configuration file.

    $ docker exec new-cluster rm /vault/config/break-glass.hcl

  3. Send SIGHUP to the Vault server again.

    $ docker kill --signal=HUP new-cluster

new-cluster

  4. Confirm Vault reloaded its configuration.

    $ docker logs --tail 10 new-cluster

==> Vault reload triggered
2026-10-22T20:53:57.532Z [INFO]  storage.raft: reloaded raft config: settings="{10240 2m0s 8192 5s 5s}"
2026-10-22T20:53:57.533Z [INFO]  replication.wal.aop: starting write controller: interval=10ms history_weight=0.99 target_queue_len_batches=5 Kp, Ki, Kd="0.010000,1.000000,0.000000"
2026-10-22T20:53:57.533Z [ERROR] core: license reload triggered but license path is empty
license reload triggered but license path is empty

  5. Confirm the DR secondary refuses generation requests with no token again.

    $ VAULT_ADDR=$NEW_ADDR vault operator generate-root -dr-token -status

Error getting root generation status: Error making API request.

URL: GET http://127.0.0.1:8200/v1/sys/replication/dr/secondary/generate-operation-token/attempt
Code: 403. Errors:

* permission denied

    The 403 error proves the API is closed. If the command prints a status table, the setting is still on, there is a cached token, or a VAULT_TOKEN environment variable is authenticating your session to Vault.

Clean up

Run the following steps from the ~/tmp/dr-tutorial working directory to remove the containers, networks, files, and environment variables from this tutorial.

  1. Revoke the oliver token.

    $ VAULT_ADDR=$NEW_ADDR VAULT_TOKEN=$OLIVER_TOKEN vault token revoke -self

  2. Remove the cached token file.

    $ rm ~/.vault-token

  3. Stop and remove the new-cluster container, including its anonymous data volumes.

    $ docker rm -f -v new-cluster

    The -f flag removes a running container. The -v flag removes the anonymous volumes that Docker creates for the /vault/file and /vault/logs paths. The vault-dr container is already gone.

  4. Remove the Docker networks.

    $ docker network rm vault-primary-net vault-dr-net

  5. Remove the configuration file and the working directory.

    $ rm vault-config.hcl && cd ~/tmp && rm -rf dr-tutorial

  6. Unset the environment variables.

    $ unset VAULT_LICENSE PRIMARY_ADDR DR_ADDR PRI_INIT PRIMARY_UNSEAL_KEY \
    PRIMARY_ROOT_TOKEN DR_INIT DR_UNSEAL_KEY DR_ROOT_TOKEN SECONDARY_TOKEN \
    NEW_ADDR NEW_INIT NEW_ROOT_TOKEN NEW_UNSEAL_KEY OLIVER_TOKEN INIT \
    ENCODED DR_OP_TOKEN

  7. (Optional) Remove the Docker image.

    $ docker image rm hashicorp/vault-enterprise:2.1.1-ent

Summary

You have completed a break glass promotion. You deleted the promoted cluster, turned on enable_unauthenticated_access for generate-operation-token on the DR secondary, generated a DR operation token with the unseal key, promoted the DR secondary to primary, and turned the setting off. The key quorum, not a token, authorized the token generation, so keep the window short and prepare a batch token before the next failure.

Similar to the Perform an unplanned DR failover tutorial, you would now bring a new cluster online and restore your original Vault topology.

Help and reference

Learn more about disaster recovery concepts in the HashiCorp Well-Architected Framework.

For more information on Vault disaster recovery, refer to the Vault documentation.

For the API endpoints used in this tutorial, refer to the Vault API documentation.

For operational guidance, refer to the HashiCorp Validated Designs.

Previous

Unplanned DR failover

 Next Collection

App integration