Setting up Performance Replication

10min
Enterprise
Vault

Enterprise Only

Performance Replication requires Vault Enterprise Premium license.

If you're unfamiliar with Vault Replication concepts, please first look at the general information page. More details can be found in the replication internals document.

Vault replication also includes a complete API. For more information, please see the Vault Replication API documentation.

Activating Performance Replication

Before enabling performance replication, create a superuser policy and enable userpass auth method with tester user on the primary cluster.

On the primary cluster, create a superuser policy.

$ vault policy write superuser -<<EOF
path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
EOF

Enable the userpass auth method.

$ vault auth enable userpass

Create a new user named tester in userpass where the password is changeme and superuser policy is attached.

$ vault write auth/userpass/users/tester password="changeme" policies="superuser"

First, create the API request payload containing the superuser policy.

$ tee payload-1.json <<EOF
{
 "policy": "path \"*\" {\n  capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\", \"sudo\"]\n}\n"
}
EOF

Create base policy.

# Create base policy
$ curl --header "X-Vault-Token: <TOKEN>" \
    --request PUT \
    --data @payload-1.json \
    https://cluster-A.example.com:8200/v1/sys/policies/acl/superuser

Enable the userpass auth method.

$ curl --header "X-Vault-Token: <TOKEN>" \
    --request POST \
    --data '{"type": "userpass"}' \
    https://cluster-A.example.com:8200/v1/sys/auth/userpass

Create a new user named tester in userpass where the password is changeme and superuser policy is attached.

$ curl --header "X-Vault-Token: <TOKEN>" \
    --request POST \
    --data '{"password": "changeme", "policies": "superuser"}' \
    https://cluster-A.example.com:8200/v1/auth/userpass/users/tester

Open a web browser and launch the Vault UI (e.g. https://cluster-A.example.com:8200/ui) and then log in.
Click the Policies tab, and then select Create ACL policy.

Enter superuser in the Name text field. In the Policy, enter the following policy rule.

path "*" {
  capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

Click Create Policy to complete.
Click the Access tab, and select Enable new method.
Select the Username & Password radio button.
Click Next.
Leave the path as default and click Enable Method.
Click the Vault CLI shell icon (>_) to open a command shell.

Execute the following command in the CLI shell to create a new user.

vault write auth/userpass/users/tester password=changeme policies=superuser

Click the icon (>_) again to hide the shell.

Later, you can authenticate with Vault using the userpass auth method.

Activating the Primary

To activate the primary, run:

$ vault write -f sys/replication/performance/primary/enable

There is currently one optional argument: primary_cluster_addr. This specifies the cluster address that the primary gives to the secondaries.

Note

Use this parameter if the primary's cluster address is not directly accessible and must be accessed via an alternate path/address (e.g. through a TCP-based load balancer). Otherwise, the secondaries use the configured cluster address to connect to the primary. See the Vault High Availability with Consul tutorial for an example Vault server configuration.

Open a web browser and launch the Vault UI for the primary cluster (e.g. https://cluster-A.example.com:8200/ui) and then log in.

Select the arrow next to Status and click Enable under REPLICATION.
Select the Performance radio button.
Choose Primary. There is currently one optional argument: Primary cluster address. This specifies the cluster address that the primary gives to the secondaries.
Click Enable replication.

Fetching a Secondary Token

This action is performed on the Primary cluster.

To fetch a secondary bootstrap token, run:

$ vault write sys/replication/performance/primary/secondary-token id=<id>

The value for id is opaque to Vault and can be any identifying value you want; this can be used later to revoke the secondary and will be listed when you read replication status on the primary. You will get back a normal wrapped response, except that the token will be a JWT instead of UUID-formatted random bytes.

Activating a Secondary

These actions are performed on the secondary cluster.

To activate a secondary using the fetched token, run:

$ vault write sys/replication/performance/secondary/enable token=<token>

You must provide the full token value. Be very careful when running this command, as it will destroy all data currently stored in the secondary.

The secondary will use the address embedded in the bootstrap token, which is the primary's redirect address to make a connection to the primary. If the primary has no redirect address (for instance, if it's not in an HA cluster), you'll need to set the primary_api_addr parameter to specify the primary's API address at secondary enable time.

Once the secondary is activated and has bootstrapped, it will be ready for service and will maintain state with the primary. It is safe to seal/shutdown the primary and/or secondary; when both are available again, they will synchronize back into a replicated state.

Note: if the secondary is in an HA cluster, you will need to ensure that each standby is sealed/unsealed with the new (primary's) unseal keys. If one of the standbys takes over on active duty before this happens it will seal itself to remove it from rotation (e.g. if using Consul for service discovery), but if a standby does not attempt taking over it will throw errors. We hope to make this workflow better in a future update.

Launch the Vault UI for the secondary cluster (e.g. https://cluster-B.example.com:8200/ui).
Select the arrow next to Status and click Enable under REPLICATION.
Check the Performance(DR) radio button and select secondary under the Cluster mode. Then paste the token you copied from the primary in the Secondary activation token field.
Click Enable replication.
Warning
This will immediately clear all data in the secondary cluster.
Sign out of the secondary cluster UI.

Note

Refer to the Monitoring Vault Replication tutorial for replication health check.

Secondary cluster re-authentication

On a production system, after a secondary is activated, enabled auth methods should be used to get tokens with appropriate policies, as policies and auth method configuration are replicated.

From the sign in page, select Username from the Method drop-down list. Enter tester in the Username and changeme in the *Password** text fields.

Click Sign in.

Select the arrow next to Status and click Performance Secondary.

The details information about the current performance replication information displays.

Performance Secondary

The generate-root command can also be used to generate a root token local to the secondary cluster. After the secondary is activated, it will need to use the unseal or recovery keys from the Primary when generating a new root token or performing other commands that require unseal or recovery keys.

Dev-Mode Root Tokens

To ease development and testing, when both the primary and secondary are running in development mode, the initial root token created by the primary (including those with custom IDs specified with -dev-root-token-id) will be populated into the secondary upon activation. This allows a developer to keep a consistent ~/.vault-token file or VAULT_TOKEN environment variable when working with both clusters.

Also, reference the Performance Replication with Mount Filters tutorial.

Managing Vault Performance Replication

Vault's performance replication model is intended to allow horizontally scaling Vault's functions rather than to act in a strict Disaster Recovery (DR) capacity. For more information on Vault's disaster recovery replication, look at the general information page.

As a result, Vault performance replication acts on static items within Vault, meaning information that is not part of Vault's lease-tracking system. In a practical sense, this means that all Vault information is replicated from the primary to secondaries except for tokens and secret leases.

Because token information must be checked and possibly rewritten with each use (e.g. to decrement its use count), replicated tokens would require every call to be forwarded to the primary, decreasing rather than increasing total Vault throughput.

Secret leases are tracked independently for two reasons: one, because every such lease is tied to a token and tokens are local to each cluster; and two, because tracking large numbers of leases is memory-intensive and tracking all leases in a replicated fashion could dramatically increase the memory requirements across all Vault nodes.

We believe that this performance replication model provides significant utility for horizontally scaling Vault's functionality. However, it does mean that certain principles must be kept in mind.

Always Use the Local Cluster

First and foremost, when designing systems to take advantage of replicated Vault, you must ensure that they always use the same Vault cluster for all operations, as only that cluster will know about the client's Vault token.

Enabling a Secondary Wipes Storage

Replication relies on having a shared keyring between primary and secondaries and also relies on having a shared understanding of the data store state. As a result, when replication is enabled, all of the secondary's existing storage will be wiped. This is irrevocable. Make a backup first if there is a remote chance you'll need some of this data at some future point.

Generally, activating as a secondary will be the first thing that is done upon setting up a new cluster for replication.

Replicated vs. Local Backend Mounts

All backend mounts (of all types) that can be enabled within Vault default to being mounted as a replicated mount. This means that mounts cannot be enabled on a secondary, and mounts enabled on the primary will replicate to secondaries.

Mounts can also be marked local (via the -local flag on the Vault CLI or setting the local parameter to true in the API). This can only be performed at mount time; if a mount is local but should have been replicated, or vice versa, you must disable the backend and mount a new instance at that path with the local flag enabled.

Local mounts do not propagate data from the primary to secondaries, and local mounts on secondaries do not have their data removed during the syncing process. The exception is during initial bootstrapping of a secondary from a state where replication is disabled; all data, including local mounts, is deleted at this time (as the encryption keys will have changed so data in local mounts would be unable to be read).

Audit Devices

In normal Vault usage, if Vault has at least one audit device configured and is unable to successfully log to at least one device, it will block further requests.

Replicated audit mounts must be able to successfully log on all replicated clusters. For example, if using the file audit device, the configured path must be able to be written to by all secondaries. It may be useful to use at least one local audit mount on each cluster to prevent such a scenario.

Never Have Two Primaries

The replication model is not designed for active-active usage and enabling two primaries should never be done, as it can lead to data loss if they or their secondaries are ever reconnected.

Disaster Recovery

Local backend mounts are not replicated and their use will require existing DR mechanisms if DR is necessary in your implementation.

If you need true DR, look at the general information page for information on Vault's disaster recovery replication.