Enable performance replication

Enterprise

Appropriate Vault Enterprise license or HCP Vault Dedicated cluster required.

Enable performance replication between two Vault Enterprise clusters so the secondary serves client requests with data from the primary. To learn what performance replication does and what to plan first, refer to Performance replication.

For a guided lab, complete the Enable performance replication tutorial.

Before you start

Run two Vault Enterprise clusters. Initialize and unseal a primary cluster and a secondary cluster. Both clusters need a storage backend that supports transactional updates, such as integrated storage. The secondary can run the same Vault version as the primary or a newer one.
Back up the secondary cluster. Enabling replication on the secondary deletes all existing data on it, including its root token, auth methods, and secrets.
Open the replication ports. Allow TCP traffic in both directions between the clusters on the cluster port, 8201 by default. The secondary also needs the primary's API port, 8200 by default, unless you activate it with a public key. Refer to Shared design considerations.
Set the cluster addresses. Set api_addr and cluster_addr on every node. Refer to the high availability parameters. If a load balancer fronts the primary's cluster port, it must pass TCP through without terminating TLS and route to the active node.
Choose an activation method. Decide whether the secondary unwraps the activation token from the primary's API or receives a token encrypted with its own public key. Refer to Activation methods.

Create a way to log in to the secondary. Enabling replication replaces the secondary's auth methods and policies with the primary's, and tokens do not replicate. Enable an auth method on the primary, such as userpass, with a user whose policy grants the operations you run on the secondary.
Check your permissions. Run every command from the root namespace. On the primary, you must have permission to:
- create and update on sys/replication/performance/primary/enable
- create, update, and sudo on sys/replication/performance/primary/secondary-token
- create and update on sys/replication/performance/primary/paths-filter/*, if you set a path filter
On the secondary, you must have permission to:
- create and update on sys/replication/performance/secondary/enable
- create and update on sys/replication/performance/secondary/generate-public-key, if you activate with a public key

Step 1: Enable performance replication on the primary

Run the following command on the primary cluster.

If a load balancer fronts the primary's cluster port, set primary_cluster_addr to the load balancer address so that secondaries receive that address in their activation tokens. If you do not set it, the primary uses its configured cluster_addr.

$ vault write -force sys/replication/performance/primary/enable

With a load balancer address:

$ vault write sys/replication/performance/primary/enable \
    primary_cluster_addr="https://<load-balancer-address>:8201"

Parameters:

primary_cluster_addr (string: ""): The cluster address that the primary gives to secondaries. Refer to Enable performance primary replication.

Step 2: Generate a secondary activation token

Generate the token with the method you chose. The id value names the secondary. It must be unique. Use the same value later to set path filters and to revoke the secondary's access. The token expires after the ttl, 30 minutes by default, and works once.

Run the following command on the primary cluster:

$ vault write -format=json sys/replication/performance/primary/secondary-token \
    id="<secondary-id>"

The activation token is the wrap_info.token value in the response.

Run the following command on the secondary cluster to generate its public key:

$ vault write -force sys/replication/performance/secondary/generate-public-key

Run the following command on the primary cluster, with the public key from the secondary:

$ vault write sys/replication/performance/primary/secondary-token \
    id="<secondary-id>" \
    secondary_public_key="<public-key>"

Run the following command on the secondary cluster to generate its public key:

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    $VAULT_ADDR/v1/sys/replication/performance/secondary/generate-public-key

Run the following command on the primary cluster, with the public key from the secondary:

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"id": "<secondary-id>", "secondary_public_key": "<public-key>"}' \
    $VAULT_ADDR/v1/sys/replication/performance/primary/secondary-token

Parameters:

id (string: <required>): An opaque identifier for the secondary.
ttl (string: "30m"): How long the activation token stays valid.
secondary_public_key (string: ""): The secondary's public key. When you set it, the primary encrypts the activation token with the key instead of wrapping it.

Refer to Generate performance secondary token and Fetch performance secondary public key.

Step 3: Enable performance replication on the secondary

Warning

This step immediately deletes all data on the secondary cluster.

Run the following command on the secondary cluster with the activation token from step 2:

$ vault write sys/replication/performance/secondary/enable \
    token="<activation-token>"

Parameters:

token (string: <required>): The secondary activation token.
primary_api_addr (string: ""): The primary's API address. Set it when the address in the token is not reachable from the secondary, for example behind a load balancer.
ca_file (string: ""): Path to a PEM-format CA file that the secondary uses to verify the primary's API certificate when it unwraps the token. Applies to the wrapping token method.
ca_path (string: ""): Path to a directory of PEM-format CA files, for the same purpose as ca_file.

If you do not set ca_file or ca_path, the secondary uses the system CA roots. Refer to Enable performance secondary.

After this step, the secondary's previous root token stops working. Log in to the secondary with the auth method you created on the primary.

Step 4: Verify the replication status

Read the performance replication status on the secondary cluster. The status endpoint does not need a token.

$ vault read -format=json sys/replication/performance/status

Replication works when the secondary shows the following values:

mode is secondary.
connection_state is ready.
state is stream-wals. A new secondary can show merkle-diff or merkle-sync briefly while it copies the initial data.
known_primary_cluster_addrs lists the cluster address of the primary.
secondary_id matches the id you set in step 2.

Log in to the secondary and list the secrets engines. The list matches the shared mounts on the primary and does not include the primary's local mounts.

$ vault secrets list

Refer to Monitor replication for every status field. If a step fails, refer to Troubleshoot replication.

Step 5: Optional: Set a path filter

Set a path filter on the primary to control which mounts and namespaces replicate to this secondary. Use the secondary's secondary_id value from step 4 as the last part of the path.

$ vault write sys/replication/performance/primary/paths-filter/<secondary-id> \
    mode="deny" \
    paths="<mount-path>"

$ curl \
    --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"mode": "deny", "paths": ["<mount-path>", "<namespace-path>"]}' \
    $VAULT_ADDR/v1/sys/replication/performance/primary/paths-filter/<secondary-id>

Parameters:

mode (string: "allow"): allow replicates only the listed paths. deny replicates everything except the listed paths.
paths (array: []): The mount and namespace paths to filter.

Refer to Create paths filter. To confirm the filter, log in to the secondary and run vault secrets list again. Filtered mounts no longer appear.

Next steps

Finish the deployment:

Add a DR secondary to the primary and to each performance secondary. Refer to Enable DR replication.
Configure automated snapshots on each cluster. Refer to Snapshots.
Set up monitoring and alerts. Refer to Monitor replication.