HashiConf Last chance to register for our cloud conference October 10-12. Register today
Vault
Integrated Storage

Use integrated storage for HA coordination

  • 14min
  • |
  • Vault

Challenge

In most common scenarios, you configure the Vault server to use a storage backend that supports high availability (HA); therefore, the storage backend stores the Vault data while maintaining the HA coordination. However, not all storage backends support HA (e.g. Amazon S3, Cassandra, MSSQL). In some cases, you may need to use a storage backend that does not have HA support which means that you can only have a single-node Vault deployment instead of an HA cluster.

Diagram

Solution

When you need to use a storage backend that does not support HA, ha_storage stanza can be specified along with the storage stanza in the Vault server configuration to handle the HA coordination. By doing so, you can add additional Vault nodes for fault tolerance.

The ha_storage must be an HA-supporting storage backend.

Note

If you wish to use Vault Integrated Storage as the ha_storage, you must run Vault 1.5 or later.

Terminology

  • HA coordination: This is used for intra-cluster high availability (in case one node fails). Without this enabled, operators cannot have multiple Vault nodes join a cluster, and there is no failover scenario in place. HA coordination helps manage the lock that nodes use to agree on who the leader is.

  • Storage Backend: This is the storage to where Vault writes it data. Some storage backends such as Integrated Storage and Consul support HA coordination by default. Others (e.g. Amazon S3, Cassandra, MSSQL) do not.

  • Integrated Storage: This is a purely Vault internal storage option and eliminates the need to manage a separate storage backend. It supports HA coordination by default.

Prerequisites

To perform the steps in this tutorial, you need Vault 1.5 or later.

Scenario Introduction

Your goal is to arrive at a Vault HA setup consisting of the following:

  • 3 Vault servers: 1 active and 2 standby.
  • Filesystem storage configured in each Vault server.
  • Integrated storage configured in each Vault server for HA coordination.

The Vault Reference Architecture explains the recommended cluster architecture. The aim of this tutorial is to walk through the manual steps to create a Vault HA cluster for better understanding.

Create a new HA cluster

The /new_cluster/cluster.sh script configures and starts three Vaults. Here's the diagram explaining what the script would produce.

Scenario

For demonstration, vault_1 listens to port 8210, vault_2 listens to port 8220, and vault_3 listens to port 8230.

  1. Retrieve the configuration by cloning the hashicorp/learn-vault-raft repository from GitHub.

    $ git clone https://github.com/hashicorp/learn-vault-raft.git

  2. Change the working directory to learn-vault-raft/raft-ha-storage/new_cluster.

    $ cd learn-vault-raft/raft-ha-storage/new_cluster

  3. Set the cluster.sh file to executable.

    $ chmod +x cluster.sh

  4. Create the configuration for each Vault.

    $ ./cluster.sh create config

[vault_1] Creating configuration
  - creating $demo_home/config-vault_1.hcl
  - creating $demo_home/raft-vault_1
[vault_2] Creating configuration
  - creating $demo_home/config-vault_2.hcl
  - creating $demo_home/raft-vault_2
[vault_3] Creating configuration
  - creating $demo_home/config-vault_3.hcl
  - creating $demo_home/raft-vault_3

  5. Open the server configuration file for vault_1 to examine.

    $ cat config-vault_1.hcl

    The storage stanza is configured to use filesystem as the storage backend which does not support HA. All encrypted Vault data gets stored in the filesystem ($demo_home/vault-storage-file/).

    To support HA, the ha_storage stanza is configured. As of Vault 1.5, Integrated Storage can be used as a HA storage. Therefore, HA coordination information (e.g. which Vault server is the active node) are stored in the ha_storage which is separate from the Vault data.

    config-vault_1.hcl
    1 2 3 4 5 6 7 8 9 101112131415161718ha_storage "raft" {
  path    = "$demo_home/ha-raft_1/"
  node_id = "vault_1"
}

storage "file" {
  path = "$demo_home/vault-storage-file/"
}

listener "tcp" {
  address = "127.0.0.1:8210"
  cluster_address = "127.0.0.1:8211"
  tls_disable = true
}

disable_mlock = true
api_addr = "http://127.0.0.1:8210"
cluster_addr = "http://127.0.0.1:8211"

    Similarly, config-vault_2.hcl and config-vault_3.hcl look very much the same. They both point to the same file storage.

    Warning

    Although the listener stanza disables TLS for this tutorial, Vault should always be used with TLS in production to provide secure communication between clients and the Vault server. It requires a certificate file and key file on each Vault host.

  6. Setup vault_1.

    $ ./cluster.sh setup vault_1

    The script first terminates all Vault server instances that are running locally (if there is any), and then start a Vault server, vault_1.

    [vault_1] starting Vault server @ vault_1
++ nc -w 1 localhost 8210
++ VAULT_API_ADDR=http://127.0.0.1:8210
++ vault server -log-level=trace -config $demo_home/config-vault_1.hcl

    Once the server is running, the script initializes and unseals the server.

    [vault_1] initializing and capturing the recovery key and root token
++ vault_1 operator init -format=json -key-shares=1 -key-threshold=1
++ export VAULT_ADDR=http://127.0.0.1:8210
++ VAULT_ADDR=http://127.0.0.1:8210
++ vault operator init -format=json -key-shares=1 -key-threshold=1

...snip...

++ vault operator unseal z8+w9bV70i4ZluxVZA53tn3eMBOC/zM+8XlAs9LX2yw=

    It logs in with the initial root token and enables the kv-v2 secrets engine at kv, and create some secrets at the kv/apikey.

    [vault_1] logging in and enabling the KV secrets engine
...
++ vault secrets enable -path=kv kv-v2
Success! Enabled the kv-v2 secrets engine at: kv/
...

[vault_1] storing secret 'kv/apikey' for testing
++ vault_1 kv put kv/apikey webapp=ABB39KKPTWOR832JGNLS02
...

  7. Setup vault_2.

    $ ./cluster.sh setup vault_2

    When vault_2 starts up and unseals, the HA Enabled value from the status output is now true.

    [vault_2] Unseal vault_2
++ cat $demo_home/unsealKey1
+ vault_2 operator unseal czIfir0r18oIfA9fXFlCwUAISfxy/w+BAK6Hiyvcs/I=
+ export VAULT_ADDR=http://127.0.0.1:8220
+ VAULT_ADDR=http://127.0.0.1:8220
+ vault operator unseal czIfir0r18oIfA9fXFlCwUAISfxy/w+BAK6Hiyvcs/I=
Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           1
Threshold              1
Version                1.10.0
Storage Type           file
Cluster Name           vault-cluster-f7fbc350
Cluster ID             9791a478-93ee-8ea2-1cd1-71a7411c5865
HA Enabled             true
HA Cluster             n/a
HA Mode                standby
Active Node Address    <none>

    After vault_2 joins the raft cluster by invoking vault operator raft join command. Notice that there is no target leader API address provided with the command when you are using the raft as the ha_storage.

    [vault_2] Join the raft cluster
+ vault_2 operator raft join
+ export VAULT_ADDR=http://127.0.0.1:8220
+ VAULT_ADDR=http://127.0.0.1:8220
+ vault operator raft join
Key       Value
---       -----
Joined    true

...snip...

[vault_2] Vault status
+ vault_2 status
+ export VAULT_ADDR=http://127.0.0.1:8220
+ VAULT_ADDR=http://127.0.0.1:8220
+ vault status
Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           1
Threshold              1
Version                1.10.0
Storage Type           file
Cluster Name           vault-cluster-f7fbc350
Cluster ID             9791a478-93ee-8ea2-1cd1-71a7411c5865
HA Enabled             true
HA Cluster             https://127.0.0.1:8211
HA Mode                standby
Active Node Address    http://127.0.0.1:8210

    The server status shows that the Active Node Address is http://127.0.0.1:8210 (vault_1).

  8. Finally, setup vault_3.

    $ ./cluster.sh setup vault_3

    It executes the same workflow as the setup vault_2 command.

    [vault_3] Join the raft cluster
+ vault_3 operator raft join
+ export VAULT_ADDR=http://127.0.0.1:8230
+ VAULT_ADDR=http://127.0.0.1:8230
+ vault operator raft join
Key       Value
---       -----
Joined    true

...snip...

[vault_3] List the raft cluster members
+ vault_3 operator raft list-peers
+ export VAULT_ADDR=http://127.0.0.1:8230
+ VAULT_ADDR=http://127.0.0.1:8230
+ vault operator raft list-peers
Node       Address           State       Voter
----       -------           -----       -----
vault_1    127.0.0.1:8211    leader      true
vault_2    127.0.0.1:8221    follower    true
vault_3    127.0.0.1:8231    follower    true

Verification

The Vault data is stored in the filesystem as configured in the storage stanza ($demo_home/vault-storage-file/). Therefore, you can find the encrypted data under the $demo_home/vault-storage-file/logical folder.

$ tree vault-storage-file
vault-storage-file
├── core
│   ├── _audit
│   ├── _auth
│   ├── _index-header-hmac-key
│   ├── _keyring
│   ├── _local-audit
│   ├── _local-auth
│   ├── _local-mounts
│   ├── _master
│   ├── _mounts
│   ├── _seal-config
│   ├── _shamir-kek
│   ├── cluster
│   │   ├── _feature-flags
│   │   └── local
│   │       └── _info
│   ├── hsm
│   │   └── _barrier-unseal-keys
│   ├── leader
│   │   └── _f732db3d-2333-b54d-492f-77dcc83c3ca7
│   ├── raft
│   │   └── _tls
│   ├── versions
│   │   └── _1.10.0
│   └── wrapping
│       └── _jwtkey
├── logical
│   ├── bd34b4dd-cdaa-2773-1172-bd7966b9a614
│   │   ├── _casesensitivity
│   │   ├── oidc_provider
│   │   │   ├── assignment
│   │   │   │   └── _allow_all
│   │   │   └── provider
│   │   │       └── _default
│   │   └── oidc_tokens
│   │       ├── named_keys
│   │       │   └── _default
│   │       └── public_keys
│   │           ├── _2bccaef4-be81-fd21-7eac-87f2c02cd063
│   │           └── _aa00eadc-b3ff-4ec1-8517-3d459693b5a0
│   └── ed92043b-160f-9812-cd7d-14cfc3681563
│       └── 0373a149-317e-0e6e-5288-5b3fa6d1f840
│           ├── _salt
│           ├── _upgrading
│           ├── archive
│           │   └── _metadata
│           ├── metadata
│           │   └── _p0EeUenIfcPM8fKsDyxO891tRE0n30mvMwrgUcTxueqaMYjhTxRT4swZONMB2kenLR3
│           ├── policy
│           │   └── _metadata
│           └── versions
│               └── 473
│                   └── _caa26f5e94333d8f9c9585a4859e066762f0361045a8b4952971950f5c60c
└── sys
    ├── policy
    │   ├── _control-group
    │   ├── _default
    │   └── _response-wrapping
    └── token
        ├── _salt
        ├── accessor
        │   └── _ec74c42ccdb7ea473dea614c24d0117502326648
        └── id
            └── _hda2b04d1aa1702f69d20b113614e5952ef1f62ec61d8ef4207ea0b6197e70354

28 directories, 36 files

In the raft-vault_1, raft-vault_2 and raft-vault_3 directories, information necessary to coordinate the HA is stored. So, let's verify.

Currently, vault_1 is the leader. Observe what happens when vault_1 steps down.

$ VAULT_ADDR=http://127.0.0.1:8210 vault operator step-down

Success! Stepped down: http://127.0.0.1:8210

Check the cluster information.

$ VAULT_ADDR=http://127.0.0.1:8210 vault operator raft list-peers

Node       Address           State       Voter
----       -------           -----       -----
vault_1    127.0.0.1:8211    follower    true
vault_2    127.0.0.1:8221    leader      true
vault_3    127.0.0.1:8231    follower    true

Now, vault_2 is the leader.

When you are done exploring, you can clean up the environment using the cluster.sh script.

Update an existing server

You have a Vault server which uses filesystem as its storage backend. Since filesystem storage backend does not support HA, you have a single node deployment. Now, you wish add two more nodes and turn it into a 3-node cluster.

Scenario 2

  1. Change your working directory to vault-guides/operations/raft-ha-storage/existing_server.

    $ cd ../existing_server

  2. Set the cluster.sh file to executable.

    $ chmod +x cluster.sh

  3. Start a server with filesystem storage backend.

    $ ./cluster.sh setup vault_1

    This command spins up a Vault server listening to port 8210 which uses file storage backend, initialize and unseal the Vault server.

    [vault_1] Creating configuration

...snip...

[vault_1] starting Vault server @ vault_1
++ nc -w 1 localhost 8210
++ VAULT_API_ADDR=http://127.0.0.1:8210

...snip...

[vault_1] initializing and capturing the recovery key and root token
++ vault_1 operator init -format=json -key-shares=1 -key-threshold=1
++ export VAULT_ADDR=http://127.0.0.1:8210
++ VAULT_ADDR=http://127.0.0.1:8210
++ vault operator init -format=json -key-shares=1 -key-threshold=1

    It also enables K/V v2 secrets engine at kv and writes some test data.

    [vault_1] logging in and enabling the KV secrets engine
...snip...

[vault_1] storing secret 'kv/apikey' for testing
++ vault_1 kv put kv/apikey webapp=ABB39KKPTWOR832JGNLS02
++ export VAULT_ADDR=http://127.0.0.1:8210
++ VAULT_ADDR=http://127.0.0.1:8210
++ vault kv put kv/apikey webapp=ABB39KKPTWOR832JGNLS02

  4. Review the created server configuration file, config-vault_1.hcl.

    $ cat config-vault_1.hcl

storage "file" {
    path = "$demo_home/vault-storage-file/"
}

listener "tcp" {
    address = "127.0.0.1:8210"
    cluster_address = "127.0.0.1:8211"
    tls_disable = true
}

disable_mlock = true
api_addr = "http://127.0.0.1:8210"
cluster_addr = "http://127.0.0.1:8211"

    The storage stanza configures filesystem as the storage backend.

  5. The server status shows that HA Enabled is false.

    $ VAULT_ADDR=http://127.0.0.1:8210 vault status
Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.10.0
Storage Type    file
Cluster Name    vault-cluster-0ff7cb74
Cluster ID      79d61de9-acb7-4f57-92f6-ea5647918045
HA Enabled      false

  6. Stop vault_1 before updating its configuration.

    $ ./cluster.sh stop vault_1

Found 1 Vault service(s) matching that name
[vault_1] stopping

  7. Update vault_1 to define its ha_storage.

    $ ./cluster.sh update

    This command updates the config-vault_1.hcl with ha_storage stanza.

    config-vault_1.hcl
    1 2 3 4 5 6 7 8 9 10ha_storage "raft" {
    path    = "$demo_home/raft-vault_1/"
    node_id = "vault_1"
}

storage "file" {
    path = "$demo_home/vault-storage-file/"
}

##...snip...

    The script restart vault_1 with updated configuration and then unseal.

    In order for the update to take an effect, it invokes sys/storage/raft/bootstrap endpoint as you find in the output.

    [vault_1] Bootstrap: vault write -f sys/storage/raft/bootstrap
++ vault_1 write -f sys/storage/raft/bootstrap
++ export VAULT_ADDR=http://127.0.0.1:8210
++ VAULT_ADDR=http://127.0.0.1:8210
++ vault write -f sys/storage/raft/bootstrap
Success! Data written to: sys/storage/raft/bootstrap

    You can see that the server status shows HA Enabled to be true in the output after bootstrapping.

    Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    1
Threshold       1
Version         1.10.0
Storage Type    file
Cluster Name    vault-cluster-0ff7cb74
Cluster ID      79d61de9-acb7-4f57-92f6-ea5647918045
HA Enabled      true
HA Cluster      https://127.0.0.1:8211
HA Mode         active
Active Since    2022-04-25T14:43:26.015345Z
    Scenario 2

  8. You can now add additional nodes, vault_2.

    $ ./cluster.sh setup vault_2

    This command creates a server configuration file, config-vault_2.hcl, start the server which listens to port 8220, and unseals the server.

    [vault_2] Creating configuration
...snip...

[vault_2] starting Vault server @ vault_2
++ nc -w 1 localhost 8220
++ VAULT_API_ADDR=http://127.0.0.1:8220
++ vault server -log-level=trace -config
...snip...

[vault_2] Unseal vault_2
++ cat $demo_home/unsealKey1
++ vault_2 operator unseal X9ig4qSZzWrjxLuRXGsVXaQQK3uL9NI7YlHHkEsdZUI=
++ export VAULT_ADDR=http://127.0.0.1:8220
++ VAULT_ADDR=http://127.0.0.1:8220
++ vault operator unseal X9ig4qSZzWrjxLuRXGsVXaQQK3uL9NI7YlHHkEsdZUI=
...snip...

    Remember that the server is using the file storage backend to persist Vault data; therefore, the Shamir's Secret Sharing based unseal keys, encryption key and initial root token were created and persisted at $demo_home/vault-storage-file/ when you initialized vault_1.

    Once the sever is up and running, it joins the raft cluster by invoking vault operator raft join command. Notice that there is no target leader API address provided with the commands when you are using the raft as the ha_storage.

    [vault_2] Join the raft cluster
++ vault_2 operator raft join
++ export VAULT_ADDR=http://127.0.0.1:8220
++ VAULT_ADDR=http://127.0.0.1:8220
++ vault operator raft join
Key       Value
---       -----
Joined    true

...snip...

[vault_2] List the raft cluster members
++ vault_2 operator raft list-peers
++ export VAULT_ADDR=http://127.0.0.1:8220
++ VAULT_ADDR=http://127.0.0.1:8220
++ vault operator raft list-peers
Node       Address           State       Voter
----       -------           -----       -----
vault_1    127.0.0.1:8211    leader      true
vault_2    127.0.0.1:8221    follower    true

  9. Open the generated config-vault_2.hcl file to examine.

    $ cat config-vault_2.hcl

    The ha_storage stanza is configured.

    config-vault_2.hcl
    1 2 3 4 5 6 7 8 9 101112131415161718ha_storage "raft" {
  path    = "$demo_home/raft-vault_2/"
  node_id = "vault_2"
}

storage "file" {
  path = "$demo_home/vault-raft-file/"
}

listener "tcp" {
  address = "127.0.0.1:8220"
  cluster_address = "127.0.0.1:8221"
  tls_disable = true
}

disable_mlock = true
api_addr = "http://127.0.0.1:8220"
cluster_addr = "http://127.0.0.1:8221"

  10. Add vault_3 to the raft cluster.

    $ ./cluster.sh setup vault_3

    This command executes the same workflow as setup vault_2. You can see in the output that you now have a 3-nodes cluster.

    [vault_3] Join the raft cluster
++ vault_3 operator raft join
++ export VAULT_ADDR=http://127.0.0.1:8230
++ VAULT_ADDR=http://127.0.0.1:8230
++ vault operator raft join
Key       Value
---       -----
Joined    true

...snip...

[vault_3] List the raft cluster members
++ vault_3 operator raft list-peers
++ export VAULT_ADDR=http://127.0.0.1:8230
++ VAULT_ADDR=http://127.0.0.1:8230
++ vault operator raft list-peers
Node       Address           State       Voter
----       -------           -----       -----
vault_1    127.0.0.1:8211    leader      true
vault_2    127.0.0.1:8221    follower    true
vault_3    127.0.0.1:8231    follower    true

You now have a three-node cluster.

Scenario

Verification

Observe what happens when vault_1 steps down.

$ VAULT_ADDR=http://127.0.0.1:8210 vault operator step-down

Success! Stepped down: http://127.0.0.1:8210

Check the cluster information.

$ VAULT_ADDR=http://127.0.0.1:8210 vault operator raft list-peers

Node       Address           State       Voter
----       -------           -----       -----
vault_1    127.0.0.1:8211    follower    true
vault_2    127.0.0.1:8221    leader      true
vault_3    127.0.0.1:8231    follower    true

Now, vault_2 is the leader.

In the vault_2.log, you should find the following entries.

...
[DEBUG] ha.raft: vote granted: from=vault_1 term=3 tally=2
[INFO]  ha.raft: election won: tally=2
[INFO]  ha.raft: entering leader state: leader="Node at 127.0.0.1:8221 [Leader]"
[INFO]  ha.raft: added peer, starting replication: peer=vault_1
[INFO]  ha.raft: added peer, starting replication: peer=vault_3
[INFO]  ha.raft: pipelining replication: peer="{Voter vault_1 127.0.0.1:8211}"
...

Clean up

When you are done you can quickly stop all services, remove all configuration and remove all modifications to your local system with the same cluster.sh script you used the setup.

Clean up your local workstation.

$ ./cluster.sh clean
Found 1 Vault service(s) matching that name
[vault_1] stopping

Found 1 Vault service(s) matching that name
[vault_2] stopping

Found 1 Vault service(s) matching that name
[vault_3] stopping

Removing configuration file $demo_home/config-vault_1.hcl

Removing configuration file $demo_home/config-vault_2.hcl

Removing configuration file $demo_home/config-vault_3.hcl

Removing raft storage file $demo_home/raft-vault_1
Removing raft storage file $demo_home/raft-vault_2
Removing raft storage file $demo_home/raft-vault_3
Removing raft storage file $demo_home/vault-storage-file
Removing key $demo_home/rootToken1
Removing key $demo_home/unsealKey1
Removing log file $demo_home/vault_1.log
Removing log file $demo_home/vault_2.log
Removing log file $demo_home/vault_3.log
Clean complete

This tutorial demonstrated the ha_storage configuration which was first introduced in Vault 0.9.0. When Vault 1.4 introduced integrated storage, it could not be set as a ha_storage although it supports HA. As of Vault 1.5, integrated storage can be used as a ha_storage. This tutorial demonstrated the basic concept of how this works.

Help and Reference

Previous

Migrate from Consul to integrated storage

 Next

Vault cluster lost quorum recovery