Troubleshoot replication connections

A secondary stays connected to the active node of its primary over the cluster port. Check connection_state and state in the replication status on the secondary, and the connection_status of each secondary on the primary.

A secondary cannot connect to the primary

Problem

One or more of the following symptoms appear:

• connection_state on the secondary is not ready.
• The operational log shows failed to connect to primary errors.
• The gap between the primary's last WAL index and the secondary's last_remote_wal keeps growing.

Cause

Replication needs TCP traffic in both directions between the primary and secondary clusters on the cluster port, 8201 by default. A firewall, security group, network policy, or load balancer blocks or breaks that traffic.

Solution

1. Confirm that firewalls allow TCP traffic in both directions on the cluster port, and on the API port during activation. Refer to the ports reference.
2. If a load balancer sits on the cluster port path, confirm that it meets the requirements in Load balancers.
3. Confirm that the addresses in known_primary_cluster_addrs on the secondary are reachable from the secondary.

A secondary loses the primary after a leader change

Problem

After the primary cluster changes its active node, the secondary reports connection refused errors and does not reconnect.

Cause

The secondary finds the primary's active node from the known_primary_cluster_addrs list. The primary's active node builds that list from heartbeats it receives from its standby nodes. If network problems stop the active node from collecting those addresses, the list is incomplete. When the primary then elects an active node that is not on the list, the secondary cannot find it.

The resolver_discover_servers parameter, true by default, lets secondary nodes ask the primary cluster for the active node. If you turned it off, the secondary depends on the list alone.

Solution

1. Run update primary on the secondary to tell it where the primary is:
   • On a DR secondary, call update DR secondary's primary with a DR operation token. Use the update_primary_addrs parameter with the primary's cluster addresses, or the token parameter with a new activation token.
   • On a performance secondary, call update performance secondary's primary with the update_primary_addrs parameter.
2. Fix the network problem between the primary's active node and its standby nodes, so the primary can give secondaries a complete list.
3. If resolver_discover_servers is false, set it to true.

After a DR failover, a newly demoted DR secondary does not accept update_primary_addrs. Use the token parameter instead. Refer to DR failover.

A shared mount conflicts with a local mount

Problem

A performance secondary shows state as idle and disconnects from the primary. The secondary's log shows that replication stopped on core/mounts with an error that includes the following text:

a local (non-replicated) mount already exists on this Vault cluster

Cause

The primary has a shared mount at the same path as a local mount on the secondary. Replication cannot apply the shared mount, so it stops.

Solution

1. On the primary, remove the shared mount that conflicts with the secondary's local mount.
2. On the secondary, call the attempt recovery endpoint to restart replication.
3. Check the replication status on both clusters until the secondary returns to stream-wals.

To prevent conflicts, give local mounts paths that no shared mount uses. Refer to Local mounts.

Related

• Troubleshoot bootstrap and token errors
• Monitor replication
• replication configuration stanza