Troubleshoot replication
Enterprise
Appropriate Vault Enterprise license or HCP Vault Dedicated cluster required.
Most replication problems fall into one of four groups: the clusters cannot
reach each other, bootstrapping or a token fails, a secondary cannot catch up,
or replication is slower than the workload needs. Start with the replication
status on the secondary. The
connection_state and state fields narrow down the cause.
Find the cause
Match the symptom you see to a troubleshooting page:
| Symptom | Refer to |
|---|---|
connection_state is not ready, or the log shows failed to connect to primary | A secondary cannot connect to the primary |
| Connection refused errors after the primary changes leader | A secondary loses the primary after a leader change |
state is idle and the log shows a local (non-replicated) mount already exists | A shared mount conflicts with a local mount |
Bootstrap fails with tls: internal error | Bootstrap fails with tls internal error |
Bootstrap fails with tls: handshake failure | Bootstrap fails with tls handshake failure |
error unwrapping secondary token | Error unwrapping secondary token |
x509: certificate signed by unknown authority | Certificate signed by unknown authority |
Bad Request, or an invalid or expired activation token | The secondary activation token is rejected |
| A DR operation fails with a root token | A DR secondary operation rejects the token |
state stays merkle-sync, or cycles between merkle-diff, merkle-sync, and stream-wals | Merkle sync loops |
| The log suggests a reindex | Reindex replication |
corrupted_merkle_tree is true, or Merkle syncs never resolve | Merkle tree corruption |
| Secondaries fall behind under load | Tune replication |
Collect information
Before you change anything, collect the following from both clusters:
- The output of
vault read -format=json sys/replication/status. - The Vault operational logs from the active node of each cluster. Set
log_leveltodebugon the secondary for more replication detail. - The replication metrics described in Monitor replication.
If you open a support case, include this information.