Monitoring Vault Replication

Disaster Recovery Replication Setup, and Performance Replication Setup guides walked through the steps to configure Vault replication.

This tutorial focuses on monitoring and troubleshooting the replication setup.

The following topics are addressed in this tutorial:

• Replication Status Check
• Port Traffic Consideration with Load Balancer
• WAL Replays
• Key Monitoring Metrics
• Vault Configuration Consideration
• Update Replication Primary
• Issues Encountered During Bootstrapping

Replication Status Check

The status endpoint provides information on WAL streaming and the Merkle Tree which are basic elements of Vault's replication system. In short, WAL streaming is used for normal ongoing replication, whereas the merkle tree and syncs are used for recovery when replication is too far out of sync to use WALs. For more detail on how these work, please review Replication Implementation Details.

Let's assume that you have a cluster 1 which is set to be the performance replication primary, and it has a disaster recovery (DR) site which is cluster 3. The cluster 2 is setup to be the performance replication secondary; it also has two DR secondaries, cluster 4 and cluster 5.

$ vault read -format=json sys/replication/status

$ vault read -format=json sys/replication/performance/status

$ vault read -format=json sys/replication/dr/status

$ curl -s $VAULT_ADDR/v1/sys/replication/status | jq

$ curl -s $VAULT_ADDR/v1/sys/replication/performance/status | jq

$ curl -s $VAULT_ADDR/v1/sys/replication/dr/status | jq

Parameters to check

On the Primary:

• cluster_id: Unique ID for this set of replicas. This value should always match on the Primary and Secondary.
• known_secondaries: List of the IDs of all non-revoked secondary activation tokens created by this Primary. The ID will be listed regardless of whether or not the token was used to activate an actual secondary cluster.
• last_dr_wal: (Vault version 1.4.0+) The last WAL known to primary that was successfully shipped to secondary. (Please review Are my DR Clusters in Sync? for more details.)
• last_performance_wal: (Vault version 1.4.0+) The last WAL known to primary that was successfully shipped to secondary. (Please review Are my Performance Clusters in Sync? for more details.)
• last_wal: Index of the last Write-Ahead Logs (WAL) entry written on the local storage. Note that WALs are created for every write to storage.
• merkle_root: A snapshot in time of the merkle tree's root hash. The merkle_root changes on every update to storage.
• mode: This should be "primary".
• primary_cluster_addr: If you set a primary_cluster_addr when enabling replication, it will appear here. If you did not explicitly set this, this field will be blank on the primary. As such, a blank field here can be completely normal.
• state: This value should be running on the primary. If the value is idle, it indicates an issue and needs to be investigated.

On the Secondary:

• cluster_id: Unique ID for this set of replicas. This value should always match on the Primary and Secondary.
• connection_state: The value can be idle, connecting, ready, transientfailure, or shutdown with ready being the healthy state. This parameter can be used to detect connection failures between the secondary and primary cluster.
• known_primary_cluster_addrs: List of cluster_addr values from each of the nodes in the Primary's cluster. This list is updated approximately every 5 seconds and is used by the Secondary to know how to communicate with the Primary in the event of a Primary node's active leader changing.
• last_remote_wal: The last WAL index that the secondary received from the primary via WAL streaming.
• merkle_root: A snapshot in time of the merkle tree's root hash. The merkle_root changes on every update to storage.
• mode: This should be "secondary".
• primary_cluster_addr: This records the very first address that the secondary uses to communicate with the Primary after replication is enabled. It may not reflect the current address being used (please review known_primary_cluster_addrs).
• secondary_id: The ID of the secondary activation token used to enable replication on this secondary cluster.
• state:
  • stream-wals: Indicates normal streaming. This is the value you want to observe.
  • merkle-diff: Indicates that the cluster is determining the sync status to determine if a merkle sync is required in order for the secondary to catch up to the primary.
  • merkle-sync: Indicates that the cluster is syncing. This happens when the secondary is too far behind the primary to use the normal stream-wals state for catching up. This state is blocking.
  • idle: Indicates an issue. You need to investigate.
• sync_progress: When the state value is merkle-sync, this value represents the count of conflicting keys which have been resolved.
• sync_total_keys: When the state value is merkle-sync, this value represents the total number conflicting keys which require resolution before WAL streaming will resume.

Are my DR Clusters in Sync?

It is relatively straightforward to tell if DR clusters are in sync since all data is replicated.

When the clusters are fully in sync, you can expect to observe:

• state of secondary will be stream-wals
• Generally, the merkle_root on the primary and secondary will match
• Vault versions 1.4.0 and higher: The last_dr_wal field in the replication status output on the primary represents the last known WAL shipped to the secondary. This can be compared with last_remote_wal on a secondary to learn how far behind it is from the primary
• Vault versions below 1.4.0: last_remote_wal on the secondary should match (or be very close to) the last_wal on the primary

Are my Performance Clusters in Sync?

Performance replication only replicates a subset of data, so checking to make sure the clusters are in sync is a bit more involved.

Most importantly, the last_wal and last_remote_wal values are NOT always the same for performance replication. This is because the last_wal on the primary tracks all of its WALs, not only the data that is being replicated. (Remember not everything gets replicated to the secondary. For example, tokens.) However, they should become the same momentarily after you write a replicated piece of data to the cluster since the last_wal written to the primary should be the piece of data and the last_remote_wal that the secondary learns via WAL streaming would be the same WAL.

When the clusters are fully in sync, you can expect to observe:

• state of secondary will be stream-wals
• Right after writing a piece of replicated data, the last_remote_wal on the secondary should match the last_wal on the primary for a short period of time.
• Generally, the merkle_root on the primary and secondary will match. Keep in mind that the merkle_root changes at every update.

Important Notes About Sync

• When you first set up replication, the last_remote_wal on the secondary will be 0 until a new piece of replicated information is written and replicated. The reason is that the initial sync of data when you first enable replication on a secondary is done via Merkle sync rather than via WAL streaming. Therefore, it does not have any remote WALs until after it bootstraps, enters stream-wals mode, and then receives a new piece of replicated data from the primary via WAL streaming.
• Be aware that a persistent difference in WAL values between the primary and a secondary can have a range of possible causes, including but not limited to:
  • Network issues between primary and secondary
  • Resource constrained systems in the primary or secondary
  • Problems with specific keys - in this case, the Vault operational logs and storage logs will provide details about the problematic key(s)
• In cases where the difference between WALs is extreme or when issues with the underlying storage have been identified it could become necessary to perform a reindex operation on the primary. In such cases, the operational logs will contain errors with suggestions to reindex like this example:

[ERROR] replication: encountered error, applying backoff: backoff=2s error="state is still irreconcilable after reindex, try reindexing primary cluster"

Again, here is our scenario:

Example - Cluster 1

$ vault read -format=json sys/replication/status | jq -r ".data"
{
  "dr": {
    "cluster_id": "d62cd840-8429-308a-6ff2-34c57f0c99bf",
    "known_secondaries": [
      "dr-secondary"
    ],
    "last_dr_wal": 119,
    "last_reindex_epoch": "0",
    "last_wal": 119,
    "merkle_root": "ab4cac6991f3f7a4217819ed9153c663610ec694",
    "mode": "primary",
    "primary_cluster_addr": "",
    "secondaries": [
      {
        "api_address": "http://dr-secondary-of-perf-primary.example.com:8200",
        "cluster_address": "https://10.1.1.13:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:18:39Z",
        "node_id": "dr-secondary"
      }
    ],
    "state": "running"
  },
  "performance": {
    "cluster_id": "4bff765c-9786-7e37-b196-d3a79f11ea7d",
    "known_secondaries": [
      "perf-secondary"
    ],
    "last_performance_wal": 119,
    "last_reindex_epoch": "0",
    "last_wal": 119,
    "merkle_root": "e8e60e66c09f87da30b8e026775519723f5ec332",
    "mode": "primary",
    "primary_cluster_addr": "",
    "secondaries": [
      {
        "api_address": "http://perf-secondary.example.com:8200",
        "cluster_address": "https://10.1.2.219:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:18:36Z",
        "node_id": "perf-secondary"
      }
    ],
    "state": "running"
  }
}

Example - Cluster 2

$ vault read -format=json sys/replication/status | jq -r ".data"
{
  "dr": {
    "cluster_id": "5a0ddeb8-3285-81d4-e5a4-a63f02086130",
    "known_secondaries": [
      "dr-secondary-1",
      "dr-secondary-2"
    ],
    "last_dr_wal": 678,
    "last_reindex_epoch": "0",
    "last_wal": 678,
    "merkle_root": "408fbbc895fb63ae7e0d392d5d34e5b5b5a9f6be",
    "mode": "primary",
    "primary_cluster_addr": "",
    "secondaries": [
      {
        "api_address": "http://dr-secondary-1.example.com:8200",
        "cluster_address": "https://10.1.2.131:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:40:02Z",
        "node_id": "dr-secondary-1"
      },
      {
        "api_address": "http://dr-secondary-2.example.com:8200",
        "cluster_address": "https://10.1.1.182:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:40:02Z",
        "node_id": "dr-secondary-2"
      }
    ],
    "state": "running"
  },
  "performance": {
    "cluster_id": "4bff765c-9786-7e37-b196-d3a79f11ea7d",
    "connection_state": "ready",
    "known_primary_cluster_addrs": [
      "https://10.1.1.150:8201"
    ],
    "last_reindex_epoch": "1594876385",
    "last_remote_wal": 375,
    "merkle_root": "5f132ea8db357817f89d7c7a326b1e0d728fcc56",
    "mode": "secondary",
    "primaries": [
      {
        "api_address": "http://perf-primary.example.com:8200",
        "cluster_address": "https://10.1.1.150:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:40:01Z"
      }
    ],
    "primary_cluster_addr": "https://10.1.1.150:8201",
    "secondary_id": "perf-secondary",
    "state": "stream-wals"
  }
}

Example - Cluster 3

$ vault read -format=json sys/replication/status | jq -r ".data"
{
  "dr": {
    "cluster_id": "d62cd840-8429-308a-6ff2-34c57f0c99bf",
    "connection_state": "ready",
    "known_primary_cluster_addrs": [
      "https://10.1.1.150:8201"
    ],
    "last_reindex_epoch": "1594876549",
    "last_remote_wal": 511,
    "merkle_root": "080c01d50875a1d1943451d8a24ef570b2ff41e1",
    "mode": "secondary",
    "primaries": [
      {
        "api_address": "http://perf-primary.example.com:8200",
        "cluster_address": "https://10.1.1.150:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:51:19Z"
      }
    ],
    "primary_cluster_addr": "https://10.1.1.150:8201",
    "secondary_id": "dr-secondary",
    "state": "stream-wals"
  },
  "performance": {
    "mode": "disabled"
  }
}

Example - Cluster 4

$ vault read -format=json sys/replication/status | jq -r ".data"
{
  "dr": {
    "cluster_id": "5a0ddeb8-3285-81d4-e5a4-a63f02086130",
    "connection_state": "ready",
    "known_primary_cluster_addrs": [
      "https://10.1.2.219:8201"
    ],
    "last_reindex_epoch": "1594876602",
    "last_remote_wal": 993,
    "merkle_root": "3bd45885345947c8f2a12d5b1309e8a62160aa18",
    "mode": "secondary",
    "primaries": [
      {
        "api_address": "http://perf-secondary.example.com:8200",
        "cluster_address": "https://10.1.2.219:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:53:02Z"
      }
    ],
    "primary_cluster_addr": "https://10.1.2.219:8201",
    "secondary_id": "dr-secondary-1",
    "state": "stream-wals"
  },
  "performance": {
    "mode": "disabled"
  }
}

Example - Cluster 5

$ vault read -format=json sys/replication/status | jq -r ".data"
{
  "dr": {
    "cluster_id": "559369e0-4897-013a-ed7c-2b817969c643",
    "known_primary_cluster_addrs": [
      "https://perf-secondary.example.com:8201"
    ],
    "last_remote_wal": 180,
    "merkle_root": "8617b6c31d806c7d64079678d5b76e4f763f7512",
    "mode": "secondary",
    "primaries": [
      {
        "api_address": "http://perf-secondary.example.com:8200",
        "cluster_address": "https://10.1.2.219:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:53:02Z"
      }
    ],
    "primary_cluster_addr": "https://10.1.2.219:8201",
    "secondary_id": "dr-secondary-2",
    "state": "merkle-sync",
    "sync_progress": 512,
    "sync_total_keys": 740,
  },
  "performance": {
    "mode": "disabled"
  }
}

Port Traffic Consideration with Load Balancer

Vault generates its own certificates for cluster members. All replication traffic uses the cluster port using these Vault-generated certificates after initial bootstrapping. Because of this, the cluster traffic can NOT be terminated at the cluster port on the load balancer level.

If your platform of choice is AWS, use the Classic Load Balancer (ELB) or Network Load Balancer (NLB) rather than Application Load Balancer (ALB). ALB requires the connection to be terminated and does not support TCP pass through. Therefore, if you are trying to use an ALB for replication traffic, you will run into issues.

If needed, set the primary_cluster_addr to override the cluster address value for replication when you enable the primary.

WAL Replays

Write-Ahead Logs (WALs) are replayed at startup as well as during a reindex. At startup, the WAL replay is completely blocking of incoming requests (no writes or reads).

During a reindex, the behavior depends on the Vault version:

• Version 0.11.2 and later: When the reindex is triggered manually via /sys/replication/reindex endpoint, the very last WAL replay of the reindex will block writes, but reads will be allowed. The rest of the reindex is non-blocking. If a reindex occurs at startup (e.g. if index records were lost or manually deleted from the underlying storage), the WAL replay blocks the incoming requests (no writes or reads) until the reindex completes.
• Version 0.11.1 and earlier: The entire reindex is fully blocking of all operations (no reads or writes).

Reindex

To keep the replicated data consistent across the primary and secondary, Vault maintains Merkle trees. If replication is in a bad state or data has been removed from the storage backend without Vault's knowledge, you can trigger reindexing of the Merkle tree via the /sys/replication/reindex endpoint. Reindex replication process can restore the potentially out-of-sync Merkle tree from the underlying storage so that the tree reflects the correct state of all encrypted secrets.

This is a powerful tool in a situation where unintentional loss of index records occurred.

NOTE: The time it takes to reindex depends on the number and size of objects in the data store.

Key Monitoring Metrics

This section explains some of the key metrics to look for that are specific to Vault Replication.

Your Vault configuration must define the telemetry stanza to collect the telemetry. For example:

...
telemetry {
  dogstatsd_addr = "localhost:8125"
  disable_hostname = true
}

It is important to collect the storage backend (Raft) telemetry to monitor the overall health of your Vault cluster. Refer to the Telemetry Metrics reference tutorial for more key monitoring metrics to monitor.

Key Metrics

The WAL is purged every few seconds by a garbage collector, but if Vault is under heavy load, the WAL may start to grow, putting a lot of pressure on the storage backend (Consul). To detect back pressure from a slow storage backend, monitor the vault.wal_flushready and vault.wal_persistwals metrics.

Vault Configuration Consideration

It is recommended to explicitly set Vault's HA parameters in your Vault configuration file. Often, it is not necessary to configure api_addr and cluster_addr when using Consul as Vault's storage backend as Consul will attempt to automatically discover and advertise the address of the active Vault nodes. However, when you have a load balancer in front of Vault, it is necessary to explicitly set these parameter values (and you typically have a load balancer in larger deployment).

Example:

listener "tcp" {
  address          = "0.0.0.0:8200"
  cluster_address  = "10.1.42.201:8201"
  tls_cert_file = "/path/to/fullchain.pem"
  tls_key_file  = "/path/to/privkey.pem"
}

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}

telemetry {
  dogstatsd_addr = "localhost:8125"
  disable_hostname = true
}

api_addr = "http://10.1.42.201:8200"
cluster_addr = "https://10.1.42.201:8201"

• api_addr: Specifies the full URL to advertise to other Vault servers in the cluster for client redirection.
• cluster_addr: Specifies the full URL to advertise to other Vault servers in the cluster for request forwarding.

Issues Encountered During Bootstrapping

It is possible that you can encounter problematic edge cases while setting up Performance or DR replication clusters. This section introduces common errors seen in various failures that you could encounter during the secondary bootstrap step, the root cause of these errors, and how to resolve them.

Error message: "transport: Error while dialing remote error: tls: internal error"

* error fetching secondary bootstrap package:
error authing for bootstrap:
rpc error: code = Unavailable desc = all SubConns are in TransientFailure,
latest connection error: connection error:
desc = "transport: Error while dialing remote error: tls: internal error"

Likely root cause:

Typically, this error is observed when the primary_cluster_addr value is set to a load balancer configured to route requests to any member of the primary Vault cluster. This is a common configuration for API communications, as Vault standby nodes can service read requests (and forward write requests to the active node), but the bootstrap process uses RPC to communicate (over a different port) and a standby will not forward bootstrap requests to the active node in the primary cluster. This causes the bootstrap to fail with the ambiguous tls error message shown in the example.

Resolution:

To maintain the benefits of the HA cluster's ability to service read requests from all nodes, a separate load balancer can be set up specifically for the cross-cluster RPC communications that only forwards requests to the active node in the cluster. Typically this is done by configuring the load balancer's health check to access the Vault API /sys/health endpoint, and only considering a 200 response code as "healthy". This is detailed in the API documentation.

Once you establish a separate load balancer for the cluster communications, you can specify it for use with the primary_cluster_addr parameter when enabling replication on the primary with the /sys/replication/dr/primary/enable endpoint.

Error message: "transport: Error while dialing remote error: tls: handshake failure"

* error fetching secondary bootstrap package:
error authing for bootstrap:
rpc error: code = Unavailable desc = connection error:
desc = "transport: Error while dialing remote error: tls: handshake failure"

Likely root cause:

This error message indicates that the RPC communication between this secondary cluster and the primary cluster is failing when initiating an mTLS connection. The server logs should have more details, usually this error is the result of one of the following conditions:

• Improper forwarding by a load balancer of the cluster port (8201 by default) to the API port (8200 by default), resulting in a protocol mismatch (HTTPS vs RPC) and subsequent TLS failure.

• An application-level load balancer (as opposed to a network-level load balancer) that attempts to use HTTPS instead of the RPC protocol, resulting in a protocol mismatch and subsequent TLS failure.

• The load balancer attempting to offer its own certificate (as with TLS termination) on the RPC port, which will conflict with the internal self-signed certificates generated by Vault for cluster communications.

Resolution:

Ensure that the load balancers used for cross-cluster communication are:

• Forwarding the RPC connections to the correct ports to the Vault server listening for RPC communications.

• Operating at the network level (and not application level) to prevent protocol mismatch.

• Not offering up their own certificates for RPC communications between the clusters.

Update Replication Primary

Consider a scenario where the performance primary (cluster 1) has two performance secondaries (clusters 2 and 5).

When an unexpected event causes cluster 1 to become inoperative, you would need its DR secondary (cluster 3) to be promoted to be DR primary.

After cluster 3 was promoted, the performance secondaries are pointed to cluster 3 as its performance primary. You can verify the replication status.

Hints & Tips

When you update the replication primary:

• Use the primary_api_addr parameter to specify the primary's API address if you received error unwrapping secondary token error from the update-primary operation.

• Ensure that your token(activation token) hasn't been expired. If the token has been expired, you would receive Bad Request error.

Some other scenarios where you may need to use the update-primary endpoint are:

• If you have network issues between the Primary's active and standby nodes, the secondary may have an incomplete list of known_primary_cluster_addrs. If this list of known addresses is incomplete at the time of a Primary leadership change and it does not include the new Primary's active node address, your secondary will start encountering errors indicating that the connection is refused. In this scenario, you will need to use update-primary to inform the secondary of its new primary. If you run into this situation, it's important to also investigate the network problems on the Primary in order to make sure that it's able to collect all of the standby nodes' cluster information to give to the secondary.

Help and Reference

• Replication API
• DR Replication API
• Performance Replication API
• Disaster Recovery Replication Setup
• Performance Replication with Mount Filters
• Telemetry Metrics Reference
• Write-ahead Logging
• Merkle tree