Monitor replication

Monitor Vault Enterprise replication to confirm that each secondary is connected to its primary and receives changes in near real time. The replication status shows whether a cluster is a primary or a secondary, whether a secondary streams write-ahead log (WAL) entries or catches up with a Merkle sync, and whether the connection is healthy. Telemetry metrics show the same signals over time, so you can alert on them.

Check both clusters in each replication pair. For how WAL streaming and Merkle syncs work, refer to Replication internals.

Check the replication status

Vault provides one status endpoint for each replication type and one that reports both:

The status endpoints do not need a token, so a monitoring system can poll them without credentials. The combined endpoint returns a dr object and a performance object. The mode of a type that the cluster does not use is disabled.

The following examples show how to request replication status from the combined status endpoint with the Vault CLI and curl:

$ vault read -format=json sys/replication/status

$ curl $VAULT_ADDR/v1/sys/replication/status

The web UI shows the same information under Replication, with separate views for DR and performance replication.

Primary status fields

The following fields appear in the status of a primary cluster:

Secondary status fields

The following fields appear in the status of a secondary cluster:

The primaries and secondaries entries also include clock_skew_ms, last_heartbeat_duration_ms, and replication_primary_canary_age_ms, which estimate clock differences and replication lag between the clusters. Refer to Clock skew and replication lag.

Secondary states

The state field on a secondary has one of the following values:

When you troubleshoot replication, check state first. It is the quickest clue to whether the primary and secondary are in sync.

Connection states

The connection_state field on a secondary shows the state of its connection to the primary:

Use this field to detect connection failures between a secondary and its primary.

Check whether clusters are in sync

The checks differ by replication type, because DR replication copies all data and performance replication copies a subset.

DR clusters

DR secondaries replicate all data. When the clusters are in sync:

• state on the secondary is stream-wals.
• merkle_root on the primary and the secondary match.
• last_remote_wal on the secondary is at or close to last_dr_wal on the primary.

Performance clusters

Performance replication copies only replicated data. The primary also writes WAL entries for data it does not ship, such as tokens, so last_wal and last_remote_wal are not always equal. When the clusters are in sync:

• state on the secondary is stream-wals.
• merkle_root on the primary and the secondary match. The value changes on every update, so compare it at a quiet moment.
• last_remote_wal on the secondary is at or close to last_performance_wal on the primary.

To test a performance pair, write a piece of replicated data on the primary. For a short time afterward, last_remote_wal on the secondary matches last_wal on the primary.

New secondaries

last_remote_wal on a new secondary is 0 until the primary ships a new replicated write. The initial data copy happens through a Merkle sync, not through WAL streaming. After the secondary enters stream-wals, the next replicated write moves last_remote_wal off 0.

Persistent differences

A WAL difference between a primary and a secondary that does not close has several possible causes:

• Network problems between the clusters.
• Resource limits on the primary or the secondary.
• Problems with specific keys. The Vault operational and storage logs name the keys.

Refer to Troubleshoot replication to find the cause.

Status examples

The following examples show only the fields to read. The API documentation has full sample responses.

DR secondary in sync

connection_state is ready, state is stream-wals, and known_primary_cluster_addrs lists the cluster address of the primary.

{
  "dr": {
    "cluster_id": "d62cd840-8429-308a-6ff2-34c57f0c99bf",
    "connection_state": "ready",
    "known_primary_cluster_addrs": ["https://10.1.1.150:8201"],
    "last_remote_wal": 511,
    "merkle_root": "080c01d50875a1d1943451d8a24ef570b2ff41e1",
    "mode": "secondary",
    "primary_cluster_addr": "https://10.1.1.150:8201",
    "secondary_id": "dr-secondary",
    "state": "stream-wals"
  },
  "performance": {
    "mode": "disabled"
  }
}

DR secondary catching up

state is merkle-sync. sync_progress and sync_total_keys show that the secondary resolved 512 of 740 conflicting keys. Its merkle_root does not match the primary until the sync finishes. When all keys resolve, the secondary returns to stream-wals.

{
  "dr": {
    "last_remote_wal": 180,
    "mode": "secondary",
    "state": "merkle-sync",
    "sync_progress": 512,
    "sync_total_keys": 740
  }
}

Monitor with telemetry

Enable the telemetry stanza on every node to collect replication metrics. The following metrics cover the main replication signals:

The key metrics page has the recommended alert thresholds. The metrics reference lists every metric.

Alert conditions

Alert on the following conditions:

• state on a primary or a secondary is idle.
• connection_state on a secondary is not ready for longer than a short grace period.
• A secondary stays in merkle-diff or merkle-sync, or cycles between those states and stream-wals.
• The gap between the primary's last WAL index and the secondary's last_remote_wal keeps growing.
• corrupted_merkle_tree is true.
• WAL flush or persist times exceed the thresholds on the key metrics page.

For a monitoring architecture, refer to the HashiCorp Validated Design for monitoring and observability.

Next steps

• Troubleshoot replication
• Tune replication
• Replication API