Use integrated storage for HA coordination
Challenge
In most common scenarios, you configure the Vault server to use a storage backend that supports high availability (HA); therefore, the storage backend stores the Vault data while maintaining the HA coordination. However, not all storage backends support HA (e.g. Amazon S3, Cassandra, MSSQL). In some cases, you may need to use a storage backend that does not have HA support which means that you can only have a single-node Vault deployment instead of an HA cluster.

Solution
When you need to use a storage backend that does not support HA, ha_storage
stanza can be specified along with the storage
stanza in the Vault server
configuration to handle the HA coordination. By doing so, you can add additional
Vault nodes for fault tolerance.
The ha_storage
must be an HA-supporting storage backend.
Note
If you wish to use Vault Integrated Storage as the ha_storage
,
you must run Vault 1.5 or later.
Terminology
HA coordination: This is used for intra-cluster high availability (in case one node fails). Without this enabled, operators cannot have multiple Vault nodes join a cluster, and there is no failover scenario in place. HA coordination helps manage the lock that nodes use to agree on who the leader is.
Storage Backend: This is the storage to where Vault writes it data. Some storage backends such as Integrated Storage and Consul support HA coordination by default. Others (e.g. Amazon S3, Cassandra, MSSQL) do not.
Integrated Storage: This is a purely Vault internal storage option and eliminates the need to manage a separate storage backend. It supports HA coordination by default.
Prerequisites
To perform the steps in this tutorial, you need Vault 1.5 or later.
Scenario Introduction
Your goal is to arrive at a Vault HA setup consisting of the following:
- 3 Vault servers: 1 active and 2 standby.
- Filesystem storage configured in each Vault server.
- Integrated storage configured in each Vault server for HA coordination.
The Vault Reference Architecture explains the recommended cluster architecture. The aim of this tutorial is to walk through the manual steps to create a Vault HA cluster for better understanding.
Create a new HA cluster
The /new_cluster/cluster.sh
script configures and starts three Vaults. Here's
the diagram explaining what the script would produce.

For demonstration, vault_1
listens to port 8210
, vault_2
listens to port
8220
, and vault_3
listens to port 8230
.
Retrieve the configuration by cloning the
hashicorp/learn-vault-raft
repository from GitHub.Change the working directory to
learn-vault-raft/raft-ha-storage/new_cluster
.Set the
cluster.sh
file to executable.Create the configuration for each Vault.
Open the server configuration file for
vault_1
to examine.The
storage
stanza is configured to use filesystem as the storage backend which does not support HA. All encrypted Vault data gets stored in the filesystem ($demo_home/vault-storage-file/
).To support HA, the
ha_storage
stanza is configured. As of Vault 1.5, Integrated Storage can be used as a HA storage. Therefore, HA coordination information (e.g. which Vault server is the active node) are stored in theha_storage
which is separate from the Vault data.config-vault_1.hcl1 2 3 4 5 6 7 8 9 101112131415161718
Similarly,
config-vault_2.hcl
andconfig-vault_3.hcl
look very much the same. They both point to the same file storage.Warning
Although the listener stanza disables TLS for this tutorial, Vault should always be used with TLS in production to provide secure communication between clients and the Vault server. It requires a certificate file and key file on each Vault host.
Setup
vault_1
.The script first terminates all Vault server instances that are running locally (if there is any), and then start a Vault server,
vault_1
.Once the server is running, the script initializes and unseals the server.
It logs in with the initial root token and enables the
kv-v2
secrets engine atkv
, and create some secrets at thekv/apikey
.Setup
vault_2
.When
vault_2
starts up and unseals, theHA Enabled
value from the status output is nowtrue
.After
vault_2
joins the raft cluster by invokingvault operator raft join
command. Notice that there is no target leader API address provided with the command when you are using theraft
as theha_storage
.The server status shows that the
Active Node Address
ishttp://127.0.0.1:8210
(vault_1
).Finally, setup
vault_3
.It executes the same workflow as the
setup vault_2
command.
Verification
The Vault data is stored in the filesystem as configured in the storage
stanza
($demo_home/vault-storage-file/
). Therefore, you can find the encrypted data
under the $demo_home/vault-storage-file/logical
folder.
In the raft-vault_1
, raft-vault_2
and raft-vault_3
directories,
information necessary to coordinate the HA is stored. So, let's verify.
Currently, vault_1
is the leader. Observe what happens when vault_1
steps
down.
Check the cluster information.
Now, vault_2
is the leader.
When you are done exploring, you can clean up the environment
using the cluster.sh
script.
Update an existing server
You have a Vault server which uses filesystem as its storage backend. Since filesystem storage backend does not support HA, you have a single node deployment. Now, you wish add two more nodes and turn it into a 3-node cluster.

Change your working directory to
vault-guides/operations/raft-ha-storage/existing_server
.Set the
cluster.sh
file to executable.Start a server with filesystem storage backend.
This command spins up a Vault server listening to port
8210
which uses file storage backend, initialize and unseal the Vault server.It also enables K/V v2 secrets engine at
kv
and writes some test data.Review the created server configuration file,
config-vault_1.hcl
.The
storage
stanza configures filesystem as the storage backend.The server status shows that
HA Enabled
isfalse
.Stop
vault_1
before updating its configuration.Update
vault_1
to define itsha_storage
.This command updates the
config-vault_1.hcl
withha_storage
stanza.config-vault_1.hcl1 2 3 4 5 6 7 8 9 10
The script restart
vault_1
with updated configuration and then unseal.In order for the update to take an effect, it invokes
sys/storage/raft/bootstrap
endpoint as you find in the output.You can see that the server status shows
HA Enabled
to betrue
in the output after bootstrapping.You can now add additional nodes,
vault_2
.This command creates a server configuration file,
config-vault_2.hcl
, start the server which listens to port8220
, and unseals the server.Remember that the server is using the
file
storage backend to persist Vault data; therefore, the Shamir's Secret Sharing based unseal keys, encryption key and initial root token were created and persisted at$demo_home/vault-storage-file/
when you initializedvault_1
.Once the sever is up and running, it joins the raft cluster by invoking
vault operator raft join
command. Notice that there is no target leader API address provided with the commands when you are using theraft
as theha_storage
.Open the generated
config-vault_2.hcl
file to examine.The
ha_storage
stanza is configured.config-vault_2.hcl1 2 3 4 5 6 7 8 9 101112131415161718
Add
vault_3
to the raft cluster.This command executes the same workflow as
setup vault_2
. You can see in the output that you now have a 3-nodes cluster.
You now have a three-node cluster.

Verification
Observe what happens when vault_1
steps down.
Check the cluster information.
Now, vault_2
is the leader.
In the vault_2.log
, you should find the following entries.
Clean up
When you are done you can quickly stop all services, remove all configuration
and remove all modifications to your local system with the same cluster.sh
script you used the setup.
Clean up your local workstation.
This tutorial demonstrated the ha_storage
configuration which was first
introduced in Vault 0.9.0. When Vault 1.4 introduced integrated storage, it could not be set as a ha_storage although it supports HA. As of Vault 1.5, integrated storage can be used as a ha_storage. This tutorial demonstrated the basic concept of how this works.