SSH secrets engine: One-time SSH password

21min
|
Vault
Video

Deployment

In a distributed cloud environment, tenant, and system is increasingly important part of online security. If an attacker gains access to your virtual machines, they can get control of most running applications, local data as well as its connected machines and systems.

The Vault SSH secrets engine provides secure authentication and authorization for access to machines via the SSH protocol. It supports signed SSH certificate and one-time SSH password modes. This tutorial demonstrates the one-time SSH password mode.

Personas

The end-to-end scenario described in this tutorial involves two personas:

operations with privileged permissions to setup SSH secrets engine
client trusted entity to request SSH OTP from Vault

Challenge

SSH servers provide a range of authentication methods, but most use password based authentication by default. If any user on the system has a fairly weak password, an attacker can more readily determine the password and create an unauthorized SSH connection.

Solution

Vault can create a one-time password (OTP) for SSH authentication on a network every time a client wants to SSH into a remote host using a helper command on the remote host to perform verification.

An authenticated client requests an OTP from the Vault server. If the client is authorized, Vault issues and returns an OTP. The client uses this OTP during the SSH authentication to connect to the desired target host.

When the client establishes an SSH connection, the OTP is received by the Vault helper which validates the OTP with the Vault server. The Vault server then deletes this OTP, ensuring that it is only used once.

Since the Vault server is contacted during SSH connection establishment, every login attempt and the correlating Vault lease information can be logged to an audit device.

Note

Vault SSH Helper version 0.1.6 and higher supports the Vault Enterprise namespaces feature.

Prerequisites

This lab was tested on macOS using an x86_64 based processor. If you are running macOS on an Apple silicon-based processor, use a x86_64 based Linux virtual machine in your preferred cloud provider.

To perform the tasks described in this tutorial, you need to have:

HCP or Vault Community Edition environment
Vagrant installed with a configured provider such as VirtualBox
jq installed

Policy requirements

Note

For the purpose of this tutorial, you can use root token to work with Vault. However, it is recommended that root tokens are only used for initial setup or in emergencies. As a best practice, use tokens with appropriate set of policies based on your role in the organization.

To perform all tasks demonstrated in this tutorial, your policy must include the following permissions:

# To enable secrets engines
path "sys/mounts/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

# To configure the SSH secrets engine
path "ssh/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# To enable the userpass auth method
path "sys/auth/userpass" {
  capabilities = [ "update" ]
}

# To create the policy for the test user
path "sys/policies/acl/test" {
  capabilities = [ "read", "update" ]
}

# To create the test user
path "auth/userpass/users/ubuntu" {
  capabilities = [ "create", "update" ]
}

If you are not familiar with policies, complete the policies tutorial.

Lab setup

Start Vault

Note

If you do not have access to an HCP Vault cluster, visit the Create a Vault Cluster on HCP tutorial.

Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the VAULT_ADDR environment variable to the copied address.
```
$ export VAULT_ADDR=<Public_Cluster_URL>
```
Return to the Overview page and click Generate token.
Within a few moments, a new token will be generated.
Copy the Admin Token.
Return to the terminal and set the VAULT_TOKEN environment variable.
```
$ export VAULT_TOKEN=<token>
```
Set the VAULT_NAMESPACE environment variable to admin.
```
$ export VAULT_NAMESPACE=admin
```
The admin namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.

Type vault status to verify your connectivity to the Vault cluster.

$ vault status

Key                      Value
---                      -----
Recovery Seal Type       shamir
Initialized              true
Sealed                   false
Total Recovery Shares    1
Threshold                1
Version                  1.9.2+ent
Storage Type             raft
...snipped...

The HCP Vault server is ready.

Start Ubuntu

Open another terminal session and initialize a new Ubuntu Vagrant box.
```
$ vagrant init generic/ubuntu2204
```
To successfully complete this tutorial, your Ubuntu box needs to be on the same network as your workstation. Edit the Vagrantfile to connect to your local network.
```
$ sed -ibak "s/# config.vm.network \"public_network\"/config.vm.network \"public_network\"/g" Vagrantfile
```
Start the Ubuntu box.
```
$ vagrant up
```

When prompted, select the number that corresponds to your active network interface. For example, if you are connected to your local network via your Wi-Fi (Wireless) adapter, enter the number 1 and press enter.

Example output:

==> default: Available bridged network interfaces:
1) en0: Wi-Fi (Wireless)
2) en5: USB Ethernet(?)
3) awdl0
4) llw0
5) en2: Thunderbolt 2
6) en1: Thunderbolt 1
7) en3: Thunderbolt 3
8) en4: Thunderbolt 4
9) bridge0
==> default: When choosing an interface, it is usually the one that is
==> default: being used to connect to the internet.
==> default:
    default: Which interface should the network bridge to?

You are ready to proceed with the tutorial.

Setup the SSH secrets engine

(Persona: operations)

On the Vault server, you must enable the SSH secrets engine and create a role.

Return to the terminal session where you created the VAULT_ADDR environment variable.

Enable the SSH secrets engine.

$ vault secrets enable ssh
Success! Enabled the ssh secrets engine at: ssh/

Create a role named otp_key_role with key_type set to otp.
```
$ vault write ssh/roles/otp_key_role \
    key_type=otp \
    default_user=vagrant \
    cidr_list=0.0.0.0/0
```
This role defaults to creating credentials for the vagrant user and will allow all remote hosts whose IP addresses fit within this CIDR range.

Return to the terminal session where you created the VAULT_ADDR environment variable.

Enable ssh secrets engine using /sys/mounts endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{"type":"ssh"}' \
    $VAULT_ADDR/v1/sys/mounts/ssh

The data defines the configuration parameters of the secrets engine.

Create an API request payload containing the parameters to set a role.
```
$ tee payload.json <<EOF
{
  "key_type": "otp",
  "default_user": "vagrant",
  "cidr_list": "0.0.0.0/0"
}
EOF
```
This defines a role with key_type set to otp and defaults to creating credentials for the vagrant user and will allow all remote hosts whose IP addresses fit within this CIDR range.

Create a role using the ssh/roles/otp_key_role endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/ssh/roles/otp_key_role

In a browser window, navigate to the Vault UI and login.
Select Enable new engine, select SSH, and click **Next**.
Click Enable Engine.
Select Create role.
Enter otp_key_role in the Role Name field, select otp from the Key type drop-down list, and then enter vagrant in the Default Username field.
Select Options to expand the optional parameter fields, and then enter 0.0.0.0/0 in the CIDR List field.
Click Create role.

Security: The tutorial uses a very permissive cidr_list value of 0.0.0.0/0. In production, we recommend that roles are defined with a range that is granular to the range of remote hosts. We also recommend that you create one role for each username to ensure isolation between usernames.

Setup the client authentication

(Persona: operations)

On the Vault server, you must create a policy to allow access to the SSH OTP role and then attach the policy to an authentication method.

Create a policy file named, test.hcl, that provides access to the ssh/creds/otp_key_role path.

$ tee test.hcl <<EOF
# To list SSH secrets paths
path "ssh/*" {
  capabilities = [ "list" ]
}
# To use the configured SSH secrets engine otp_key_role role
path "ssh/creds/otp_key_role" {
  capabilities = ["create", "read", "update"]
}
EOF

The path ssh/creds/otp_key_role is the path to the role created in the Setup the SSH secrets engine section.

Create a policy named test with the policy defined in test.hcl.
```
$ vault policy write test ./test.hcl
```
Enable the userpass auth method.
```
$ vault auth enable userpass
```

Create a user named ubuntu with the password "training" assigned the test policy.

$ vault write auth/userpass/users/ubuntu password="training" policies="test"

Create an API request payload containing the stringified test policy.

$ tee payload.json <<EOF
{
  "policy": "path \"ssh/creds/otp_key_role\" {\n capabilities = [ \"create\", \"read\", \"update\", \"list\" ]\n }"
}
EOF

Create a policy named test with the policy defined in test.hcl.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request PUT \
   --data @payload.json \
   $VAULT_ADDR/v1/sys/policies/acl/test

Enable the userpass auth method.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{"type": "userpass"}' \
    $VAULT_ADDR/v1/sys/auth/userpass

Create a user named ubuntu with the password "training" assigned the test policy.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
    --request POST \
    --data '{"password": "training", "policies": "test"}' \
    $VAULT_ADDR/v1/auth/userpass/users/ubuntu

Click the Policies tab, and then select Create ACL policy.
In the Name field enter test

Paste the following policy in the Policy text box.

# To list SSH secrets paths
path "ssh/*" {
  capabilities = [ "list" ]
}
# To use the configured SSH secrets engine otp_key_role role
path "ssh/creds/otp_key_role" {
  capabilities = ["create", "read", "update"]
}

Click Create policy to complete.
Click the Access tab, and select Enable new method.
Select the Username & Password radio button.
Click Next.
Leave the path as default and click Enable Method.
Click the Vault CLI shell icon (>_) to open a command shell.
Execute vault write auth/userpass/users/ubuntu password=training policies=test in the CLI shell to create a new user, ubuntu:
Click the icon (>_) again to hide the shell.

Install vault-ssh-helper

(Persona: operations)

SELinux users

If you use SELinux in enforcing mode, you need to define and enable a module that allows vault-ssh-helper to open and write log files, and communicate with the Vault server. Creating and enabling the SELinux module is beyond the scope of this tutorial. If you do not require SELinux enforcement, then operating SELinux in permissive mode also allows vault-ssh-helper to function.

On each Remote host, you must:

Install vault-ssh-helper
Create a configuration file for the vault-ssh-helper
Modify both the Pluggable Authentication Module (PAM) sshd configuration file and the sshd configuration file
Restart the sshd service

Note

The example scenario in this tutorial uses the Ubuntu Linux distribution. If you use a different distribution, you will need to update paths and file locations to match your distribution.

Return to the terminal where you performed vagrant up and connect to Ubuntu.
```
$ vagrant ssh
```

Download and install version 0.2.1 of vault-ssh-helper from releases.hashicorp.com.

$ wget https://releases.hashicorp.com/vault-ssh-helper/0.2.1/vault-ssh-helper_0.2.1_linux_amd64.zip

Unzip the binary from the archive into /usr/local/bin.

$ sudo unzip -q vault-ssh-helper_0.2.1_linux_amd64.zip -d /usr/local/bin

Set the vault-ssh-helper binary's permissions to executable.
```
$ sudo chmod 0755 /usr/local/bin/vault-ssh-helper
```

Set the vault-ssh-helper binary's user and group to root.

$ sudo chown root:root /usr/local/bin/vault-ssh-helper

Create a directory to store the configuration file.
```
$ sudo mkdir /etc/vault-ssh-helper.d/
```
The Vault SSH Helper reads a configuration at the path /etc/vault-ssh-helper.d/config.hcl with this format:
```
vault_addr = "<VAULT_EXTERNAL_ADDR>"
tls_skip_verify = false
ca_cert = "<PEM_ENCODED_CA_CERT>"
ssh_mount_point = "ssh"
namespace = "my_namespace"
allowed_roles = "*"
```
The vault_addr is the network address of the Vault server configured to generate the OTP.
tls_skip_verify enables or disables TLS verification.
ca_cert is the path to the PEM-encoded CA certificate files used to verify the Vault server's TLS certificate. When vault-ssh-helper is run with the -dev flag this is ignored.
ssh_mount_point is the Vault server path where the SSH secrets engine is enabled.
namespace is the namespace of the SSH mount point (Vault Enterprise and HCP Vault)
allowed_roles defines all * or a comma-separated list of allowed roles defined in the SSH secrets engines.
Refer to the documentation for the entire list of configuration properties.
Return to the terminal session where you created the VAULT_ADDR environment variable.
Retrieve and copy the value for VAULT_ADDR.
```
$ echo $VAULT_ADDR
```
Return to the terminal connected to the Ubuntu box.

Create a variable named VAULT_EXTERNAL_ADDR that contains the Vault server's external network address copied in the previous step.

$ VAULT_EXTERNAL_ADDR=<VAULT_EXTERNAL_ADDR>

Example:

$ VAULT_EXTERNAL_ADDR=https://vault-cluster-public-vault-abcdefg.123456.z1.hashicorp.cloud:8200

Create a Vault SSH Helper configuration file /etc/vault-ssh-helper.d/config.hcl.

$ sudo tee /etc/vault-ssh-helper.d/config.hcl <<EOF
vault_addr = "$VAULT_EXTERNAL_ADDR"
tls_skip_verify = false
ssh_mount_point = "ssh"
namespace = "admin"
allowed_roles = "*"
EOF

Backup the original PAM sshd configuration configuration file.
```
$ sudo cp /etc/pam.d/sshd /etc/pam.d/sshd.orig
```
Open the file in your preferred text editor.
```
$ sudo nano /etc/pam.d/sshd
```
The common-auth must be commented out or removed to disable the standard Unix authentication and replaced with authentication through vault-ssh-helper. Finally, a workaround for a bug that exists with some versions of pam_exec.so must also be included.
Refer to the documentation for details about these parameter settings.
Example:
```
# PAM configuration for the Secure Shell service

# Standard Un*x authentication.
#@include common-auth
auth requisite pam_exec.so quiet expose_authtok log=/var/log/vault-ssh.log /usr/local/bin/vault-ssh-helper -config=/etc/vault-ssh-helper.d/config.hcl
auth optional pam_unix.so not_set_pass use_first_pass nodelay

...
```
The vault-ssh-helper is configured to load the configuration file config.hcl with -config parameter and while running it logs output to /var/log/vault-ssh.log using the -log parameter.

Backup the original sshd_config configuration file.

$ sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.orig

Modify the sshd configuration file.
```
$ sudo nano /etc/ssh/sshd_config
```
Add or set the following:
```
KbdInteractiveAuthentication yes
UsePAM yes
PasswordAuthentication no
```
This enables the keyboard-interactive authentication and PAM authentication modules. The password authentication is disabled.
Note
Older version of Ubuntu use ChallengeResponseAuthentication instead of KbdInteractiveAuthentication.
Caution: You should ensure that there is another means of access to the server (such as a previously established SSH key for the root user) when disabling password authentication so that you are not prevented from logging in at all.
Restart the sshd service.
```
$ sudo systemctl restart sshd
```

Verify the configuration.

$ vault-ssh-helper -verify-only -config /etc/vault-ssh-helper.d/config.hcl
==> WARNING: Dev mode is enabled!
[INFO] using SSH mount point: ssh
[INFO] using namespace: admin
[INFO] vault-ssh-helper verification successful!

Note

These steps must be performed on all remote hosts.

Determine the IP address of the Ubuntu box.
```
$ ip -4 address show eth1 | grep inet | awk '{print $2}' | cut -d'/' -f1

192.168.100.102
```
Note
By following this tutorial, you should have two eth adapters. The eth0 adapter represents the internal VirtualBox network and the eth1 adapter represents the bridged connection to your local network.
Copy the IP address for eth1.
Disconnect from Ubuntu.
```
$ exit
```

Generate an OTP

(Persona: client)

A client configured to target a Vault server may now authenticate with the Vault server, generate an OTP and then use that OTP to establish an SSH connection.

Return to the terminal configured with the VAULT_ADDR environment variable.
Create a variable named REMOTE_HOST_IP that stores the Ubuntu box's IP address.
```
$ REMOTE_HOST_IP=<REMOTE_HOST_IP>
```
Example:
```
$ REMOTE_HOST_IP=192.168.100.102
```

Authenticate with Vault using the userpass auth method with the username ubuntu and password training. Store the token in an environment variable.
```
$ UBUNTU_TOKEN=$(vault login -method=userpass username=ubuntu password=training -format=json | jq -r '.auth | .client_token')
```

Generate an OTP, through the otp_key_role, for remote host given its IP address.

$ VAULT_TOKEN=$UBUNTU_TOKEN vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP

Example Output:

$ vault write ssh/creds/otp_key_role ip=$REMOTE_HOST_IP
Key                Value
---                -----
lease_id           ssh/creds/otp_key_role/234bb081-d22e-3762-3ae5-744110ea4d0a
lease_duration     768h
lease_renewable    false
ip                 192.168.100.102
key                f1cb47ad-6255-0be8-6bd8-5c4b3b01c8df
key_type           otp
port               22
username           vagrant

The output displays a key. Its value is the OTP to use during SSH authentication.

Authenticate with Vault using the userpass auth method with the username ubuntu and password training. Store the token in an environment variable.

$ UBUNTU_TOKEN=$(curl --request POST \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --data '{"password": "training"}' \
   $VAULT_ADDR/v1/auth/userpass/login/ubuntu | jq -r '.auth | .client_token')

Generate an OTP, through the otp_key_role, for remote host given its IP address.

$ curl --header "X-Vault-Token: $UBUNTU_TOKEN" \
   --header "X-Vault-Namespace: $VAULT_NAMESPACE" \
   --request POST \
   --data '{"ip": "'"$REMOTE_HOST_IP"'"}' \
   $VAULT_ADDR/v1/ssh/creds/otp_key_role | jq -r .data

Example Output:

{
     "ip": "192.168.100.102",
     "key": "6e472878-721a-b066-2cec-1bed0127ad44",
     "key_type": "otp",
     "port": 22,
     "username": "vagrant"
}

The output displays a key. Its value is the OTP to use during SSH authentication.

Return to the web browser with the Vault UI and sign out.
At the Sign in to Vault, select the Userpass tab, and enter ubuntu in the Username field, and training in the Password field.
Click Sign In.
Select ssh under Secrets Engines.
Select otp_key_role and enter vagrant in the Username field, and enter the target host's IP address (e.g. 192.168.100.102) in the IP Address field.
Click Generate.
Click Copy credentials. This copies the OTP (key value). Note the warning given that this is your only chance to record this information.

Initiate an SSH session

Connect to the Ubunut box as the vagrant user and the one time password provided by Vault.
```
$ ssh -o PubkeyAuthentication=no vagrant@$REMOTE_HOST_IP
```
Note
To demonstrate one time passwords, include the PubkeyAuthentication=no parameter to ensure your native SSH client does include any cached SSH keys.

When prompted enter yes to accept the fingerprint and for a Password: enter the OTP key from the previous step.

Example:

$ ssh -o PubkeyAuthentication=no vagrant@$REMOTE_HOST_IP

The authenticity of host '192.168.100.102 (192.168.100.102)' can't be established.
ECDSA key fingerprint is SHA256:N8CjNy6ahilQ3xvOO6FZKA3NkvpIOPi1LxpLvNx7xbQ.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes

Password: <Enter OTP>
Last login: Tue Aug 16 13:20:22 2022 from 10.0.2.2
vagrant@ubuntu2204:~$

You are now connected to the Ubuntu box using the one time password from Vault.

Cleanup

Disconnect from the Ubuntu box.
```
$ exit
```
Switch to the terminal where you performed vagrant up and destroy the Ubuntu box.
```
$ vagrant destroy --force
```
Stop the Vault dev mode server or delete your HCP Vault server.

Help and Reference

PKI secrets engine with managed keys

Configurable password generation policies