Deploying Vault Enterprise in a private datacenter
Overview
This section details the steps to create a Vault cluster manually in a private datacenter. This guide assumes that you have already read the Detailed design section of this guide and have a basic understanding of the Vault architecture and the steps required to deploy Vault in a private datacenter.
Throughout this section we will use the following names and IP addresses for the Vault servers:
DNS Name | IP Address | Node Type | Location |
---|---|---|---|
vault-1.domain | 10.0.0.1 | voter | zone1 |
vault-2.domain | 10.0.0.2 | voter | zone2 |
vault-3.domain | 10.0.0.3 | voter | zone3 |
vault-4.domain | 10.0.0.4 | non-voter | zone1 |
vault-5.domain | 10.0.0.5 | non-voter | zone2 |
vault-6.domain | 10.0.0.6 | non-voter | zone3 |
vault.domain | 10.0.0.100 | (load balancer) | (all zones) |
These names and IP addresses are used for illustrative purposes only. Use the names and IP addresses that are appropriate for your environment.
Prepare
Create or obtain the files that you will later deploy onto the Vault servers. The Vault binaries are pulled via curl
command during the Installation phase.
TLS
Create an X.509 certificate that will be installed onto each of the Vault servers. Refer to your organization's process on creating a new certificate that matches the DNS record you intend to direct users to when accessing Vault – in this case, the DNS record pointing at the load balancer: vault.domain
.
Three files will be needed:
- The certificate:
vault-public.pub
- The certificate's private key
vault-private.key
: - The bundle file from the certificate authority used to vend the certificate:
ca.pub
Note
Only one set of these files is required. These files will be copied onto all nodes.License
Obtain your active Vault Enterprise License file. If you do not have this file, contact your HashiCorp account team.
Server Configs
Using what you have learned from previous sections of this guide, create 6 configuration files, one for each server. The configuration for each server will be slightly different, depending on which redundancy zone it lives in, and whether it is a voting or a non-voting node.
Voting Node Example File
# config.hcl for vault1 (voter node example)
api_addr = "https://10.0.0.1:8200"
cluster_addr = "https://10.0.0.1:8201"
ui = true
license_path = "/etc/vault.d/vault.hclic"
listener "tcp" {
address = "10.0.0.1:8200"
cluster_address = “10.0.0.1:8201”
tls_client_ca_file = "/etc/vault.d/ca.pub"
tls_cert_file = "/etc/vault.d/vault-public.pub"
tls_key_file = "/etc/vault.d/vault-private.key"
}
storage "raft" {
path = "/var/lib/vault/data"
node_id = "vault1"
autopilot_redundancy_zone = "zone1"
retry_join {
leader_api_addr = "https://10.0.0.1:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
retry_join {
leader_api_addr = "https://10.0.0.2:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
retry_join {
leader_api_addr = "https://10.0.0.3:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
retry_join {
leader_api_addr = "https://10.0.0.4:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
retry_join {
leader_api_addr = "https://10.0.0.5:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
retry_join {
leader_api_addr = "https://10.0.0.6:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
}
seal "awskms" {
region = "us-west-2"
kms_key_id = "abcd1234-a123-456a-a12b-a123b4cd56ef"
}
telemetry {
disable_hostname = true
prometheus_retention_time = "12h"
}
Non-Voting Node Example File
# config.hcl for vault4 (non-voter node example)
api_addr = "http://10.0.0.4:8200"
cluster_addr = "https://10.0.0.4:8201"
ui = true
license_path = "/etc/vault.d/vault.hclic"
storage "raft" {
path = "/var/lib/vault/data"
node_id = "vault4"
autopilot_redundancy_zone = "zone1"
}
listener "tcp" {
address = "10.0.0.4:8200"
cluster_address = “10.0.0.4:8201”
tls_client_ca_file = "/etc/vault.d/ca.pub"
tls_cert_file = "/etc/vault.d/vault-public.pub"
tls_key_file = "/etc/vault.d/vault-private.key"
retry_join_as_non_voter = true
retry_join {
leader_api_addr = "https://10.0.0.1:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
retry_join {
leader_api_addr = "https://10.0.0.2:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
retry_join {
leader_api_addr = "https://10.0.0.3:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
retry_join {
leader_api_addr = "https://10.0.0.5:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
retry_join {
leader_api_addr = "https://10.0.0.6:8200"
leader_ca_cert_file = "/etc/vault.d/ca.pub"
leader_client_cert_file = "/etc/vault.d/vault-public.pub"
leader_client_key_file = "/etc/vault.d/vault-private.key"
leader_tls_servername = “vault.domain”
}
}
seal "awskms" {
region = "us-west-2"
kms_key_id = "abcd1234-a123-456a-a12b-a123b4cd56ef"
}
telemetry {
disable_hostname = true
prometheus_retention_time = "12h"
}
Set aside these files for later in the installation process.
Build
Servers
- Identify the availability zones within your datacenter where your Vault servers will live. For the rest of this document, we will refer to these as
zone1
,zone2
, etc. - Build the servers. Ensure there are 2 servers per each of the 3 redundancy zones, for a total of 6 servers. For the rest of this document, we will refer to these servers as
vault1
,vault2
, etc. - Ensure you can log in to each server as the root user (or a user with sudo privileges) via SSH.
Load Balancer
Configure your load balancer with a new frontend for Vault that will route incoming traffic on TCP port 8200 to the Vault servers on TCP port 8200. Refer to the Detailed design section of the guide for more information on how to configure the load balancer.
Snapshot Storage
Ensure that your snapshot storage location is created and that the Vault service can successfully write to it. If you will be using a remote network filesystem, ensure that it is mounted at the correct location on all Vault servers.
Configuration of Vault snapshots can be found in Vault: Operating Guide for Adoption.
Install Vault
Perform these steps on all Vault servers:
- Create a
vault
user. Create directories for Vault configuration owned by this user.
$ useradd --system --user-group --shell /bin/false vault
$ mkdir -p /etc/vault /var/lib/vault
$ chown -R vault:vault /etc/vault /var/lib/vault /usr/local/bin/vault
- Download the Vault Enterprise package from Hashicorp. Unzip the package and move the
vault
binary to a sharedPATH
location such as/usr/local/bin
, owned by thevault
user.
$ curl -O https://releases.hashicorp.com/vault/1.15.0+ent/vault_1.15.0+ent_linux_amd64.zip
$ unzip vault_1.15.0+ent_linux_amd64.zip
$ mv vault /usr/local/bin/
$ chown vault:vault /usr/local/bin/vault
Note
As of this writing, the current version of Vault Enterprise is1.15.0+ent
. Update the curl
and unzip
commands to reflect the desired version of Vault Enterprise.- Copy the files created in the Prepare section (license file, certificate files, and relevant config file) to this server at
/etc/vault
. Ensure that the files are owned by thevault
user and that the permissions are set to640
.
$ chown vault:vault /etc/vault.d/*
$ chmod 640 /etc/vault.d/*
$ ls -l /etc/vault.d/
-rw-r---- 1 vault vault 0 Sep 22 12:45 ca.pub
-rw-r---- 1 vault vault 0 Sep 22 12:45 config.hcl
-rw-r---- 1 vault vault 0 Sep 22 12:45 vault.hclic
-rw-r---- 1 vault vault 0 Sep 22 12:45 vault-public.pub
-rw-r---- 1 vault vault 0 Sep 22 12:45 vault-private.key
- Create a systemd unit file for the Vault service, then load it into systemd. Note that the
ExecStart
line runs thevault
binary pointing to yourconfig.hcl
file:
$ cat << EOF >> /etc/systemd/system/vault.service
[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://developer.hashicorp.com/vault/docs
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/config.hcl
StartLimitIntervalSec=60
StartLimitBurst=3
[Service]
Type=notify
EnvironmentFile=/etc/vault.d/vault.env
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/bin/vault server -config=/etc/vault.d/config.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
LimitNOFILE=65536
LimitMEMLOCK=infinity
LimitCORE=0
[Install]
WantedBy=multi-user.target
EOF
$ systemctl daemon-reload
$ systemctl enable vault
- Start the Vault service.
$ systemctl start vault
Initialize
Create GPG public keys
For each individual who will be receiving a key share, create their own unique GPG public key:
$ cat > alice_key.conf << EOF
%echo Generating a basic OpenPGP key for Alice
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Name-Real: Alice
Name-Comment: Alice is a Vault PGP user
Name-Email: alice@example.com
Expire-Date: 1
Passphrase: recede-yard-unwilling-shrouded
%commit
%echo done
EOF
$ gpg --full-gen-key --batch alice_key.conf
…
$ gpg --output alice_key.pub --export alice@example.com
Initialize Vault
Warning
The following section should be performed on only one of the nodes. Failure to follow this could result in multiple single-node clusters, rather than a single cluster containing all nodes.Pass in the GPG public key from each participating individual to initialize Vault with the API, CLI, or UI. Once Vault initializes, it returns the key shares, each one encrypted with the GPG public keys which you passed in at initialization time.
$ vault operator init \
-pgp-keys “alice_key.pub,bob_key.pub,carol_key.pub,dan_key.pub,frank_key.pub" \
-root-token-pgp-key "alice_key.pub"
Unseal Key 1: wcFMA7Np...JCbg=
Unseal Key 2: wcFMAw4D...5LY4=
Unseal Key 3: wcFMA8d6...G8r0=
Unseal Key 4: wcFMA4ly5...6ELY=
Unseal Key 5: wcFMA0eb...3hRg=
Initial Root Token: wcFMA7Np…2mGO
Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.
Vault does not store the generated root key (previously known as master key).
Without at least 3 keys to reconstruct the root key, Vault will remain permanently sealed!
It is possible to generate new unseal keys, provided you have a quorum of existing unseal keys shares. See "vault operator rekey" for more information.
Record the root token value from this output. You will need it in order to log into Vault initially.
Each individual should now record their unique unseal key and store it safely. This is their key share, which is now encrypted and base64-encoded.
Unseal Vault
To unseal Vault using encrypted key shares, each individual will need to both base64-decode and decrypt their unseal key before sending it to the Vault server. When decrypting with GPG each individual will also need to pass the unique passphrase that was supplied during the creation of their own GPG key:
$ cat encoded-encrypted-keyshare1 | base64 --decode > encrypted-keyshare1
$ gpg --passphrase recede-yard-unwilling-shrouded --decrypt encrypted-keyshare1 > keyshare1
$ vault operator unseal keyshare1
gpg: encrypted with 4096-bit RSA key, ID 91002A8F909C6714, created 2023-04-17
"Alice (Alice is a Vault PGP user) <alice@example.com>"
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 5
Threshold 3
Unseal Progress 1/3
…
Repeat this process until a sufficient number of keyshares have been supplied to Vault.
At this point the Vault server is now considered unsealed. The output of the final unseal command should look something like this:
gpg: encrypted with 4096-bit RSA key, ID 6ED85F446B6FAA27, created 2023-04-17
"Dan (Dan is a Vault PGP user) <dan@example.com>"
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 5
Threshold 3
…
This concludes the key ceremony.
Form Cluster
You should now have a functional initialized and unsealed cluster, which contains one active node: vault1
. You can confirm this by setting the VAULT_TOKEN environment variable to the root token generated during the init step and then running:
$ vault operator raft list-peers
You should now see that the cluster is comprised of six nodes: one active node, and five standby nodes. To view the voter/non-voter status of each node, you can run:
$ vault operator raft autopilot state
Congratulations, you now have a running cluster! You can now test that you are able to log into Vault:
$ export VAULT_TOKEN=<root_token_value>
$ vault login -no-store $VAULT_TOKEN