• HashiCorp Developer

  • HashiCorp Cloud Platform
  • Terraform
  • Packer
  • Consul
  • Vault
  • Boundary
  • Nomad
  • Waypoint
  • Vagrant
Vault
  • Install
  • Tutorials
  • Documentation
  • API
  • Try Cloud(opens in new tab)
  • Sign up
Monitor & Troubleshoot

Skip to main content
19 tutorials
  • Troubleshooting Vault
  • Troubleshooting Vault on Kubernetes
  • Diagnose Server Issues
  • Use hcdiag with Vault
  • Monitoring Vault Replication
  • Vault Usage Metrics
  • Monitor Telemetry & Audit Device Log Data
  • Monitor Telemetry with Prometheus & Grafana
  • Inspect Data in BoltDB
  • Inspecting Data in Consul Storage
  • Inspecting Data in Integrated Storage
  • Blocked Audit Devices
  • Querying Audit Device Logs
  • Troubleshoot Irrevocable Leases
  • Vault Cluster Lost Quorum Recovery
  • Operate Vault in Recovery Mode
  • Telemetry Metrics Reference
  • Monitoring Vault with Datadog
  • Audit Device Logs and Incident Response with Elasticsearch

  • Resources

  • Tutorial Library
  • Certifications
  • Community Forum
    (opens in new tab)
  • Support
    (opens in new tab)
  • GitHub
    (opens in new tab)
  1. Developer
  2. Vault
  3. Tutorials
  4. Monitor & Troubleshoot
  5. Monitoring Vault Replication

Monitoring Vault Replication

  • 23min

  • EnterpriseEnterprise
  • VaultVault

Enterprise Only: Vault replication feature is a part of Vault Enterprise. Disaster Recovery requires Vault Enterprise Standard license. Performance Replication requires Vault Enterprise Premium license.

Disaster Recovery Replication Setup, and Performance Replication Setup guides walked through the steps to configure Vault replication.

Deployment Topology

This tutorial focuses on monitoring and troubleshooting the replication setup.

The following topics are addressed in this tutorial:

  • Replication Status Check
  • Port Traffic Consideration with Load Balancer
  • WAL Replays
  • Key Monitoring Metrics
  • Vault Configuration Consideration
  • Update Replication Primary
  • Issues Encountered During Bootstrapping

Replication Status Check

The status endpoint provides information on WAL streaming and the Merkle Tree which are basic elements of Vault's replication system. In short, WAL streaming is used for normal ongoing replication, whereas the merkle tree and syncs are used for recovery when replication is too far out of sync to use WALs. For more detail on how these work, please review Replication Implementation Details.

Let's assume that you have a cluster 1 which is set to be the performance replication primary, and it has a disaster recovery (DR) site which is cluster 3. The cluster 2 is setup to be the performance replication secondary; it also has two DR secondaries, cluster 4 and cluster 5.

Cluster Relationship

To verify the replication status of those clusters, use the sys/replication/status endpoint.

$ vault read -format=json sys/replication/status

This will output the status for both types of replication, with performance replication status under "performance" and disaster recovery replication status under "dr".

Note that you can also check the status of a specific replication type by specifying a type in the endpoint path.

These examples check only the status of performance replication by using the /v1/sys/replication/performance/status endpoint.

$ vault read -format=json sys/replication/performance/status

Likewise, you can retrieve the status of disaster replication with by using the /v1/sys/replication/dr/status endpoint.

$ vault read -format=json sys/replication/dr/status

To verify the replication status of those clusters, use the sys/replication/status endpoint.

$ curl -s $VAULT_ADDR/v1/sys/replication/status | jq

This will output the status for both types of replication, with performance replication status under "performance" and disaster recovery replication status under "dr".

Note that you can also check the status of a specific replication type by specifying a type in the endpoint path.

These examples check only the status of performance replication by using the /v1/sys/replication/performance/status endpoint.

$ curl -s $VAULT_ADDR/v1/sys/replication/performance/status | jq

Likewise, you can retrieve the status of disaster replication with by using the /v1/sys/replication/dr/status endpoint.

$ curl -s $VAULT_ADDR/v1/sys/replication/dr/status | jq

To check replication status in the Web UI, navigate to the cluster's dashboard from the Status drop-down menu.

Replication Enabled

From this page, you can gauge the status of your cluster, primary or secondary, using the parameters outlined in more detail below. The most immediately important are state as well as the Write-Ahead-Logs (WAL) data.

Parameters to check

On the Primary:

  • cluster_id: Unique ID for this set of replicas. This value should always match on the Primary and Secondary.
  • known_secondaries: List of the IDs of all non-revoked secondary activation tokens created by this Primary. The ID will be listed regardless of whether or not the token was used to activate an actual secondary cluster.
  • last_dr_wal: (Vault version 1.4.0+) The last WAL known to primary that was successfully shipped to secondary. (Please review Are my DR Clusters in Sync? for more details.)
  • last_performance_wal: (Vault version 1.4.0+) The last WAL known to primary that was successfully shipped to secondary. (Please review Are my Performance Clusters in Sync? for more details.)
  • last_wal: Index of the last Write-Ahead Logs (WAL) entry written on the local storage. Note that WALs are created for every write to storage.
  • merkle_root: A snapshot in time of the merkle tree's root hash. The merkle_root changes on every update to storage.
  • mode: This should be "primary".
  • primary_cluster_addr: If you set a primary_cluster_addr when enabling replication, it will appear here. If you did not explicitly set this, this field will be blank on the primary. As such, a blank field here can be completely normal.
  • state: This value should be running on the primary. If the value is idle, it indicates an issue and needs to be investigated.

On the Secondary:

  • cluster_id: Unique ID for this set of replicas. This value should always match on the Primary and Secondary.
  • connection_state: The value can be idle, connecting, ready, transientfailure, or shutdown with ready being the healthy state. This parameter can be used to detect connection failures between the secondary and primary cluster.
  • known_primary_cluster_addrs: List of cluster_addr values from each of the nodes in the Primary's cluster. This list is updated approximately every 5 seconds and is used by the Secondary to know how to communicate with the Primary in the event of a Primary node's active leader changing.
  • last_remote_wal: The last WAL index that the secondary received from the primary via WAL streaming.
  • merkle_root: A snapshot in time of the merkle tree's root hash. The merkle_root changes on every update to storage.
  • mode: This should be "secondary".
  • primary_cluster_addr: This records the very first address that the secondary uses to communicate with the Primary after replication is enabled. It may not reflect the current address being used (please review known_primary_cluster_addrs).
  • secondary_id: The ID of the secondary activation token used to enable replication on this secondary cluster.
  • state:
    • stream-wals: Indicates normal streaming. This is the value you want to observe.
    • merkle-diff: Indicates that the cluster is determining the sync status to determine if a merkle sync is required in order for the secondary to catch up to the primary.
    • merkle-sync: Indicates that the cluster is syncing. This happens when the secondary is too far behind the primary to use the normal stream-wals state for catching up. This state is blocking.
    • idle: Indicates an issue. You need to investigate.
  • sync_progress: When the state value is merkle-sync, this value represents the count of conflicting keys which have been resolved.
  • sync_total_keys: When the state value is merkle-sync, this value represents the total number conflicting keys which require resolution before WAL streaming will resume.

NOTE: When you are troubleshooting Vault replication, checking the state value can give you the first clue of whether or not the primary and secondary are in sync.

Are my DR Clusters in Sync?

It is relatively straightforward to tell if DR clusters are in sync since all data is replicated.

When the clusters are fully in sync, you can expect to observe:

  • state of secondary will be stream-wals
  • Generally, the merkle_root on the primary and secondary will match
  • Vault versions 1.4.0 and higher: The last_dr_wal field in the replication status output on the primary represents the last known WAL shipped to the secondary. This can be compared with last_remote_wal on a secondary to learn how far behind it is from the primary
  • Vault versions below 1.4.0: last_remote_wal on the secondary should match (or be very close to) the last_wal on the primary

Are my Performance Clusters in Sync?

Performance replication only replicates a subset of data, so checking to make sure the clusters are in sync is a bit more involved.

Most importantly, the last_wal and last_remote_wal values are NOT always the same for performance replication. This is because the last_wal on the primary tracks all of its WALs, not only the data that is being replicated. (Remember not everything gets replicated to the secondary. For example, tokens.) However, they should become the same momentarily after you write a replicated piece of data to the cluster since the last_wal written to the primary should be the piece of data and the last_remote_wal that the secondary learns via WAL streaming would be the same WAL.

With Vault version 1.4.0 and higher you can observe the last WAL shipped to the secondary by the primary with the last_performance_wal field as part of replication status output on the primary. You can compare this value to the last_remote_wal value on a secondary to determine how far behind it is from the primary.

When the clusters are fully in sync, you can expect to observe:

  • state of secondary will be stream-wals
  • Right after writing a piece of replicated data, the last_remote_wal on the secondary should match the last_wal on the primary for a short period of time.
  • Generally, the merkle_root on the primary and secondary will match. Keep in mind that the merkle_root changes at every update.

Important Notes About Sync

  • When you first set up replication, the last_remote_wal on the secondary will be 0 until a new piece of replicated information is written and replicated. The reason is that the initial sync of data when you first enable replication on a secondary is done via Merkle sync rather than via WAL streaming. Therefore, it does not have any remote WALs until after it bootstraps, enters stream-wals mode, and then receives a new piece of replicated data from the primary via WAL streaming.
  • Be aware that a persistent difference in WAL values between the primary and a secondary can have a range of possible causes, including but not limited to:
    • Network issues between primary and secondary
    • Resource constrained systems in the primary or secondary
    • Problems with specific keys - in this case, the Vault operational logs and storage logs will provide details about the problematic key(s)
  • In cases where the difference between WALs is extreme or when issues with the underlying storage have been identified it could become necessary to perform a reindex operation on the primary. In such cases, the operational logs will contain errors with suggestions to reindex like this example:
[ERROR] replication: encountered error, applying backoff: backoff=2s error="state is still irreconcilable after reindex, try reindexing primary cluster"

Again, here is our scenario:

Cluster Relationship

Example - Cluster 1

Cluster 1 is both a Performance and DR primary; therefore, the mode for dr is set to "primary". The same for performance. The state is set to "running" for both dr and performance replication. The Performance and DR outputs for this one node have separate cluster_ids and merkle_roots since each of these forms of replication works independently of the other.

$ vault read -format=json sys/replication/status | jq -r ".data"
{
  "dr": {
    "cluster_id": "d62cd840-8429-308a-6ff2-34c57f0c99bf",
    "known_secondaries": [
      "dr-secondary"
    ],
    "last_dr_wal": 119,
    "last_reindex_epoch": "0",
    "last_wal": 119,
    "merkle_root": "ab4cac6991f3f7a4217819ed9153c663610ec694",
    "mode": "primary",
    "primary_cluster_addr": "",
    "secondaries": [
      {
        "api_address": "http://dr-secondary-of-perf-primary.example.com:8200",
        "cluster_address": "https://10.1.1.13:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:18:39Z",
        "node_id": "dr-secondary"
      }
    ],
    "state": "running"
  },
  "performance": {
    "cluster_id": "4bff765c-9786-7e37-b196-d3a79f11ea7d",
    "known_secondaries": [
      "perf-secondary"
    ],
    "last_performance_wal": 119,
    "last_reindex_epoch": "0",
    "last_wal": 119,
    "merkle_root": "e8e60e66c09f87da30b8e026775519723f5ec332",
    "mode": "primary",
    "primary_cluster_addr": "",
    "secondaries": [
      {
        "api_address": "http://perf-secondary.example.com:8200",
        "cluster_address": "https://10.1.2.219:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:18:36Z",
        "node_id": "perf-secondary"
      }
    ],
    "state": "running"
  }
}

Enter <Cluster_2_API_Address>/ui/vault/replication in a web browser to see the replication summary page. Cluster 1 is both a Performance and DR primary.

In the Web UI, the dashboard is updated every ten seconds.

Summary Dashboard

The state displays at the top of the dashboard. For both Disaster Recovery and Performance replications, the last WAL entries will be shown, as well as their merkle_root values and number of secondaries respectively.

In the summary, there's a Details button which will take you to a more comprehensive view for both the Performance and Disaster Recovery primary.

Example - Cluster 2

Cluster 2 is a DR primary cluster and a Performance secondary cluster; therefore, its mode for dr is set to "primary"; however, for performance replication, it's set to "secondary". Notice that the secondary displays last_remote_wal rather than last_wal and its state is stream-wals.

The secondary's merkle_root should generally match the merkle_root of its primary. If that is off, you can write a test piece of replicated data to the performance primary and then check to make sure the last_wal on the primary matches (or almost matches) the last_remote_wal on the secondary right afterwards.

$ vault read -format=json sys/replication/status | jq -r ".data"
{
  "dr": {
    "cluster_id": "5a0ddeb8-3285-81d4-e5a4-a63f02086130",
    "known_secondaries": [
      "dr-secondary-1",
      "dr-secondary-2"
    ],
    "last_dr_wal": 678,
    "last_reindex_epoch": "0",
    "last_wal": 678,
    "merkle_root": "408fbbc895fb63ae7e0d392d5d34e5b5b5a9f6be",
    "mode": "primary",
    "primary_cluster_addr": "",
    "secondaries": [
      {
        "api_address": "http://dr-secondary-1.example.com:8200",
        "cluster_address": "https://10.1.2.131:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:40:02Z",
        "node_id": "dr-secondary-1"
      },
      {
        "api_address": "http://dr-secondary-2.example.com:8200",
        "cluster_address": "https://10.1.1.182:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:40:02Z",
        "node_id": "dr-secondary-2"
      }
    ],
    "state": "running"
  },
  "performance": {
    "cluster_id": "4bff765c-9786-7e37-b196-d3a79f11ea7d",
    "connection_state": "ready",
    "known_primary_cluster_addrs": [
      "https://10.1.1.150:8201"
    ],
    "last_reindex_epoch": "1594876385",
    "last_remote_wal": 375,
    "merkle_root": "5f132ea8db357817f89d7c7a326b1e0d728fcc56",
    "mode": "secondary",
    "primaries": [
      {
        "api_address": "http://perf-primary.example.com:8200",
        "cluster_address": "https://10.1.1.150:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:40:01Z"
      }
    ],
    "primary_cluster_addr": "https://10.1.1.150:8201",
    "secondary_id": "perf-secondary",
    "state": "stream-wals"
  }
}

Cluster 2 is a DR primary cluster and a Performance secondary cluster. In the Web UI, this means two dashboards, one on the primary cluster and one on the secondary. Each dashboard is updated every ten seconds.

On the primary, you'll notice the last_wal value that you can use to assess the sync status of the clusters.

Replication primary with last_wal

On the secondary, you'll be presented with data that will help you determine the state of the secondary, as well as the health of its connection with the primary. Remember that the last_wal and last_remote_wal values are NOT always the same for performance replication.

Replication with stream_wals

Example - Cluster 3

Cluster 3 is a DR secondary of cluster 1; therefore, its list of known_primary_cluster_addrs should include the cluster address of each node in cluster 1 (or a load balancer that will direct it there).

The DR Secondary must be kept synchronized with its DR primary. The last_wal of primary and last_remote_wal of secondary should be the same (or nearly the same with a little latency introduced by the network).

$ vault read -format=json sys/replication/status | jq -r ".data"
{
  "dr": {
    "cluster_id": "d62cd840-8429-308a-6ff2-34c57f0c99bf",
    "connection_state": "ready",
    "known_primary_cluster_addrs": [
      "https://10.1.1.150:8201"
    ],
    "last_reindex_epoch": "1594876549",
    "last_remote_wal": 511,
    "merkle_root": "080c01d50875a1d1943451d8a24ef570b2ff41e1",
    "mode": "secondary",
    "primaries": [
      {
        "api_address": "http://perf-primary.example.com:8200",
        "cluster_address": "https://10.1.1.150:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:51:19Z"
      }
    ],
    "primary_cluster_addr": "https://10.1.1.150:8201",
    "secondary_id": "dr-secondary",
    "state": "stream-wals"
  },
  "performance": {
    "mode": "disabled"
  }
}

Cluster 3 is a DR secondary of cluster 1.

In the Web UI, the DR secondary contains the last_remote_wal that can be checked against the primary, as well as the list of known_primary_cluster_addrs. This list of known_primary_cluster_addrs should include the cluster address of each node in cluster 1 (or a load balancer that will direct it there). It also includes a link to view the primary cluster's UI.

Disaster Recovery Secondary with known_primary_cluster_addrs

Example - Cluster 4

Cluster 4 is a DR secondary of cluster 2; as in the example for cluster 3, the last_wal of primary and last_remote_wal of secondary should be the same (or nearly the same with a little latency introduced by the network).

$ vault read -format=json sys/replication/status | jq -r ".data"
{
  "dr": {
    "cluster_id": "5a0ddeb8-3285-81d4-e5a4-a63f02086130",
    "connection_state": "ready",
    "known_primary_cluster_addrs": [
      "https://10.1.2.219:8201"
    ],
    "last_reindex_epoch": "1594876602",
    "last_remote_wal": 993,
    "merkle_root": "3bd45885345947c8f2a12d5b1309e8a62160aa18",
    "mode": "secondary",
    "primaries": [
      {
        "api_address": "http://perf-secondary.example.com:8200",
        "cluster_address": "https://10.1.2.219:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:53:02Z"
      }
    ],
    "primary_cluster_addr": "https://10.1.2.219:8201",
    "secondary_id": "dr-secondary-1",
    "state": "stream-wals"
  },
  "performance": {
    "mode": "disabled"
  }
}

Cluster 4 is a DR secondary of cluster 2.

In the Web UI, the DR secondary contains the last_remote_wal, shown below.

Replication secondary with last_remote_wal

This number can be checked against the primary's last_wal. These two should be the same (or close to the same).

Example - Cluster 5

Cluster 5 is another DR secondary of cluster 2; in this contrived example, there was a problem with the cluster 5 host and it is now behind on replication with its primary (cluster 2).

Its merkle-root value does not match the primary and you will notice that it has values for both sync_progress and sync_total_keys indicating that at the time of the status query, 512 of 740 conflicting keys had been resolved.

Provided that all keys are successfully resolved, this secondary should return to the stream-wals state.

$ vault read -format=json sys/replication/status | jq -r ".data"
{
  "dr": {
    "cluster_id": "559369e0-4897-013a-ed7c-2b817969c643",
    "known_primary_cluster_addrs": [
      "https://perf-secondary.example.com:8201"
    ],
    "last_remote_wal": 180,
    "merkle_root": "8617b6c31d806c7d64079678d5b76e4f763f7512",
    "mode": "secondary",
    "primaries": [
      {
        "api_address": "http://perf-secondary.example.com:8200",
        "cluster_address": "https://10.1.2.219:8201",
        "connection_status": "connected",
        "last_heartbeat": "2020-07-16T05:53:02Z"
      }
    ],
    "primary_cluster_addr": "https://10.1.2.219:8201",
    "secondary_id": "dr-secondary-2",
    "state": "merkle-sync",
    "sync_progress": 512,
    "sync_total_keys": 740,
  },
  "performance": {
    "mode": "disabled"
  }
}

In the Web UI, if a secondary is behind, the status will change to merkle-sync and you will see a progress indicator for that process. Provided this is successfully resolved, you'll see the cluster return to stream_wals.

Disaster Recovery Secondary in merkle-sync

Port Traffic Consideration with Load Balancer

Vault generates its own certificates for cluster members. All replication traffic uses the cluster port using these Vault-generated certificates after initial bootstrapping. Because of this, the cluster traffic can NOT be terminated at the cluster port on the load balancer level.

If your platform of choice is AWS, use the Classic Load Balancer (ELB) or Network Load Balancer (NLB) rather than Application Load Balancer (ALB). ALB requires the connection to be terminated and does not support TCP pass through. Therefore, if you are trying to use an ALB for replication traffic, you will run into issues.

If needed, set the primary_cluster_addr to override the cluster address value for replication when you enable the primary.

WAL Replays

Write-Ahead Logs (WALs) are replayed at startup as well as during a reindex. At startup, the WAL replay is completely blocking of incoming requests (no writes or reads).

During a reindex, the behavior depends on the Vault version:

  • Version 0.11.2 and later: When the reindex is triggered manually via /sys/replication/reindex endpoint, the very last WAL replay of the reindex will block writes, but reads will be allowed. The rest of the reindex is non-blocking. If a reindex occurs at startup (e.g. if index records were lost or manually deleted from the underlying storage), the WAL replay blocks the incoming requests (no writes or reads) until the reindex completes.
  • Version 0.11.1 and earlier: The entire reindex is fully blocking of all operations (no reads or writes).

Reindex

To keep the replicated data consistent across the primary and secondary, Vault maintains Merkle trees. If replication is in a bad state or data has been removed from the storage backend without Vault's knowledge, you can trigger reindexing of the Merkle tree via the /sys/replication/reindex endpoint. Reindex replication process can restore the potentially out-of-sync Merkle tree from the underlying storage so that the tree reflects the correct state of all encrypted secrets.

This is a powerful tool in a situation where unintentional loss of index records occurred.

NOTE: The time it takes to reindex depends on the number and size of objects in the data store.

Key Monitoring Metrics

This section explains some of the key metrics to look for that are specific to Vault Replication.

For a full list of Vault telemetry, refer to the Vault documentation.

Your Vault configuration must define the telemetry stanza to collect the telemetry. For example:

...
telemetry {
  dogstatsd_addr = "localhost:8125"
  disable_hostname = true
}

It is important to collect the storage backend (Raft) telemetry to monitor the overall health of your Vault cluster. Refer to the Telemetry Metrics reference tutorial for more key monitoring metrics to monitor.

Key Metrics

Metric NameDescription
logshipper.streamWALs.missing_guardNumber of incidences where the starting Merkle Tree index used to begin streaming WAL entries is not matched/found
logshipper.streamWALs.guard_foundNumber of incidences where the starting Merkle Tree index used to begin streaming WAL entries is matched/found
replication.fetchRemoteKeysTime taken to fetch keys from a remote cluster participating in replication prior to Merkle Tree based delta generation
replication.merkleDiffTime taken to perform a Merkle Tree based delta generation between the clusters participating in replication
replication.merkleSyncTime taken to perform a Merkle Tree based synchronization using the last delta generated between the clusters participating in replication
replication.wal.gcTime taken to complete one run of the WAL garbage collection process
vault.wal_persistwalsTime taken to persist a WAL to storage
vault.wal_flushreadyTime taken to flush a ready WAL to storage
wal.gc.totalTotal number of WAL on disk
wal.gc.deletedNumber of WAL deleted during each garbage collection run

The WAL is purged every few seconds by a garbage collector, but if Vault is under heavy load, the WAL may start to grow, putting a lot of pressure on the storage backend (Consul). To detect back pressure from a slow storage backend, monitor the vault.wal_flushready and vault.wal_persistwals metrics.

Vault Configuration Consideration

It is recommended to explicitly set Vault's HA parameters in your Vault configuration file. Often, it is not necessary to configure api_addr and cluster_addr when using Consul as Vault's storage backend as Consul will attempt to automatically discover and advertise the address of the active Vault nodes. However, when you have a load balancer in front of Vault, it is necessary to explicitly set these parameter values (and you typically have a load balancer in larger deployment).

Example:

listener "tcp" {
  address          = "0.0.0.0:8200"
  cluster_address  = "10.1.42.201:8201"
  tls_cert_file = "/path/to/fullchain.pem"
  tls_key_file  = "/path/to/privkey.pem"
}

storage "consul" {
  address = "127.0.0.1:8500"
  path    = "vault/"
}

telemetry {
  dogstatsd_addr = "localhost:8125"
  disable_hostname = true
}

api_addr = "http://10.1.42.201:8200"
cluster_addr = "https://10.1.42.201:8201"
  • api_addr: Specifies the full URL to advertise to other Vault servers in the cluster for client redirection.
  • cluster_addr: Specifies the full URL to advertise to other Vault servers in the cluster for request forwarding.

This is particularly important when you have multiple listeners and plan to setup Vault replications.

Issues Encountered During Bootstrapping

It is possible that you can encounter problematic edge cases while setting up Performance or DR replication clusters. This section introduces common errors seen in various failures that you could encounter during the secondary bootstrap step, the root cause of these errors, and how to resolve them.

Error message: "transport: Error while dialing remote error: tls: internal error"

* error fetching secondary bootstrap package:
error authing for bootstrap:
rpc error: code = Unavailable desc = all SubConns are in TransientFailure,
latest connection error: connection error:
desc = "transport: Error while dialing remote error: tls: internal error"

Likely root cause:

Typically, this error is observed when the primary_cluster_addr value is set to a load balancer configured to route requests to any member of the primary Vault cluster. This is a common configuration for API communications, as Vault standby nodes can service read requests (and forward write requests to the active node), but the bootstrap process uses RPC to communicate (over a different port) and a standby will not forward bootstrap requests to the active node in the primary cluster. This causes the bootstrap to fail with the ambiguous tls error message shown in the example.

Resolution:

To maintain the benefits of the HA cluster's ability to service read requests from all nodes, a separate load balancer can be set up specifically for the cross-cluster RPC communications that only forwards requests to the active node in the cluster. Typically this is done by configuring the load balancer's health check to access the Vault API /sys/health endpoint, and only considering a 200 response code as "healthy". This is detailed in the API documentation.

Once you establish a separate load balancer for the cluster communications, you can specify it for use with the primary_cluster_addr parameter when enabling replication on the primary with the /sys/replication/dr/primary/enable endpoint.

NOTE: Establishing a separate load balancer could require that you disable the previously enabled replication and generate a new token for the secondary.

Error message: "transport: Error while dialing remote error: tls: handshake failure"

* error fetching secondary bootstrap package:
error authing for bootstrap:
rpc error: code = Unavailable desc = connection error:
desc = "transport: Error while dialing remote error: tls: handshake failure"

Likely root cause:

This error message indicates that the RPC communication between this secondary cluster and the primary cluster is failing when initiating an mTLS connection. The server logs should have more details, usually this error is the result of one of the following conditions:

  • Improper forwarding by a load balancer of the cluster port (8201 by default) to the API port (8200 by default), resulting in a protocol mismatch (HTTPS vs RPC) and subsequent TLS failure.

  • An application-level load balancer (as opposed to a network-level load balancer) that attempts to use HTTPS instead of the RPC protocol, resulting in a protocol mismatch and subsequent TLS failure.

  • The load balancer attempting to offer its own certificate (as with TLS termination) on the RPC port, which will conflict with the internal self-signed certificates generated by Vault for cluster communications.

Resolution:

Ensure that the load balancers used for cross-cluster communication are:

  • Forwarding the RPC connections to the correct ports to the Vault server listening for RPC communications.

  • Operating at the network level (and not application level) to prevent protocol mismatch.

  • Not offering up their own certificates for RPC communications between the clusters.

Update Replication Primary

Consider a scenario where the performance primary (cluster 1) has two performance secondaries (clusters 2 and 5).

When an unexpected event causes cluster 1 to become inoperative, you would need its DR secondary (cluster 3) to be promoted to be DR primary.

Replication Groups

After cluster 3 was promoted, the performance secondaries are pointed to cluster 3 as its performance primary. You can verify the replication status.

Hints & Tips

When you update the replication primary:

  • Use the primary_api_addr parameter to specify the primary's API address if you received error unwrapping secondary token error from the update-primary operation.

  • Ensure that your token(activation token) hasn't been expired. If the token has been expired, you would receive Bad Request error.

Some other scenarios where you may need to use the update-primary endpoint are:

  • If you have network issues between the Primary's active and standby nodes, the secondary may have an incomplete list of known_primary_cluster_addrs. If this list of known addresses is incomplete at the time of a Primary leadership change and it does not include the new Primary's active node address, your secondary will start encountering errors indicating that the connection is refused. In this scenario, you will need to use update-primary to inform the secondary of its new primary. If you run into this situation, it's important to also investigate the network problems on the Primary in order to make sure that it's able to collect all of the standby nodes' cluster information to give to the secondary.

Help and Reference

  • Replication API
  • DR Replication API
  • Performance Replication API
  • Disaster Recovery Replication Setup
  • Performance Replication with Mount Filters
  • Telemetry Metrics Reference
  • Write-ahead Logging
  • Merkle tree
 Previous
 Next

This tutorial also appears in:

  •  
    23 tutorials
    Vault Enterprise
    Learn features that are only available to Vault Enterprise.
    • Vault

On this page

  1. Monitoring Vault Replication
  2. Replication Status Check
  3. Port Traffic Consideration with Load Balancer
  4. WAL Replays
  5. Key Monitoring Metrics
  6. Vault Configuration Consideration
  7. Issues Encountered During Bootstrapping
  8. Update Replication Primary
  9. Help and Reference
Give Feedback(opens in new tab)
  • Certifications
  • System Status
  • Terms of Use
  • Security
  • Privacy
  • Trademark Policy
  • Trade Controls
  • Give Feedback(opens in new tab)