Configure SSH secrets engine to use PKCS 11 HSM using managed keys

Appropriate Vault Enterprise license required

SSH secrets engine can be pluged into Vault's centralised abstraction layer called managed keys to delegate crypto-operations operations to a trusted external KMS or HSM. This allows customers to leverage key management systems external to Vault for handling, storing, and interacting with private key material. You can leverage HSM for storing private key pair of CA created in SSH using signed certificates method. Also Vault delegates signing operation to HSM.

Requirements

To setup SSH secrets engine backed by PKCS 11 based HSM, you need:

A Vault Enterprise server running.
A PKCS 11 based HSM.

Vault setup

Configure a managed key.
A PKCS#11 HSM requires a shared library to configure Vault to communicate with the HSM. To declare this library, edit the Vault server configuration to add a kms_library stanza
config-vault.hcl
```
kms_library "pkcs11" {
    name = "myhsm"
    library = "/usr/vault/lib/libCryptoki2_64.so"
}
```
Use pkcs11 with IBM EP11 tokens.
Note
User needs to have Crypto Express adapters enabled in the LinuxOne and Linux on IBM Z instances as well as configure to use EP11 token. User also needs to install opencryptoki which can then talk to EP11 token. For more details, refe to the IBM EP11 Token page.
```
kms_library "pkcs11" {
    name = "myhsm"
    library = "/usr/local/lib/opencryptoki/libopencryptoki.so"
}
```
Restart or reload Vault's configuration with SIGHUP
```
$ kill -HUP $(pidof vault)
```
Each managed key requires type-specific configuration. You must identify the location of a key and PIN to access the HSM. This is similar to Vault's PKCS#11 auto-unseal mechanism. Configure the managed key.
```
$ vault write sys/managed-keys/pkcs11/ssh-CA-key \
    library=myhsm \
    slot=4 \
    pin=12345678 \
    key_label="hsm-CA-key" \
    allow_generate_key=true \
    mechanism=0x0001 \
    allow_store_key=true \
    key_bits=4096 \
    any_mount=false
```
The PKCS#11 specific parameters are library, referring to the previously configured kms_library stanza, slot, pin, key_label, and mechanism, which identifies the object in the HSM which will hold the key. Its mechanism is CKM_RSA_PKCS (RSA with PKCS#11 v1.5 signatures), and that will require a 4096 bit key. The allow_generate_key flag indicates that Vault is allowed to request that the HSM generate a key. The allow_store_key parameter indicates that a new key may be stored in the backend. Without this flag, you may configure the key yourself in the HSM and just point Vault at the result. The any_mount means any mount in the namespace may access the managed key. For this example, it is set to false so that you can demonstrate how to lock managed key access down to a specific mount.
Example output:
```
Success! Data written to: sys/managed-keys/pkcs11/ssh-CA-key
```

Read the key back to verify if named managed key was created successfully.

$ vault read sys/managed-keys/pkcs11/ssh-CA-key

Key                   Value
---                   -----
UUID                  686d9bef-a13d-4d8a-edbe-293b22c6eec6
allow_generate_key    true
allow_replace_key     false
allow_store_key       true
any_mount             false
key_bits              4096
key_label             hsm-CA-key
library               myhsm
mechanism             1
name                  ssh-CA-key
pin                   redacted
slot                  4
type                  pkcs11
usages                [3 4]

Configure SSH secrets engine

Enable the ssh secrets engine at the ssh-client-signer path.

$ vault secrets enable -path=ssh-client-signer ssh
Success! Enabled the ssh secrets engine at: ssh-client-signer/

Tune the secrets engine to use managed keys.
```
$ vault secrets tune --allowed-managed-keys=ssh-CA-key ssh-client-signer
Success! Tuned the secrets engine at: ssh-client-signer/
```
You configured the managed key with any_mount=false in the previous step. This command grants access to the ssh-CA-key to the SSH secrets engine's mount.
Configure SSH CA to use managed keys.
```
$ vault write ssh-client-signer/config/ca managed_key_name="ssh-CA-key"
Success! Data written to: ssh-client-signer/config/ca
```
RSA 4096 public private key pair gets created at this step while configuring SSH CA to use managed-key. And its key label is hsm-CA-key

Read the private key of CA created inside HSM.

$ vault read ssh-client-signer/config/ca

Example output:

Key           Value
---           -----
public_key    ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQChGcB66LAVB9FEPS0tJSiV6r/jBS8rUls7CWelMnMfGn9tPJrl0rzBtAhrcjfiPhMd/bh7dYvAQhVvggo1M83owVsAljGoKbeHbL6qg73vO7mDJAWz1MpDetJwkC1QJKsXaE6WT6vjIFnRy2tnQ9mxP6mjRYCYSa0AMuyjMuh9L6ZRJ5Msrf8LJqkrklkr2QBuxyrMZhZqQHpar+rkTccXU2s3as8IQIWBEaYIGN7o3Og1zAH4r7OeyKAft58WhkCwU3y9gmBD4UTrz8Ohs3liAw3DuLTJ5JdWK0I4QPsx2+cB2sTz+2/Dsy+ip0CpjhzY8PpbIYXuLlTb5kxwsNrWXvtX86TVFqpnX5FadOeNnF3XWGJmECElFeaFQv4pH/0P+wg67Sx3cqq3vGKjxW4gKcn9B7cPp8rgCS9UKl/S8Vg3KTLCr/KFsyIrv/m4C0Yu2Yp88GiLz9Q4QuUOowsi/PoMpSH1W57sn8rM/zxMzjIWu2scqSNy29seAhJl0YOtuUhbep4+3/lChvVdqTPpfi5UzWsaawTF5S3AB0DD2ehEw6P7ZU78vGo5I+Pod+As6qEDdmcIXDR0IHcnXII2MqoTEjoHU6NiHR2RpHkRmRlGHy7HAQ/vqdEPuF1X0WjfH/l8fMRyq0v75GFiYcfbNWCGRlZFjyt3Tk3nEH7cZw==

Add the public key to all target host's SSH configuration. This process can be manual or automated using a configuration management tool. The public key is accessible via the API and does not require authentication.
```
$ curl -o /etc/ssh/trusted-user-ca-keys.pem http://127.0.0.1:8200/v1/ssh-client-signer/public_key
```
```
$ vault read -field=public_key ssh-client-signer/config/ca > /etc/ssh/trusted-user-ca-keys.pem
```
Add the path where the public key contents are stored to the SSH configuration file as the TrustedUserCAKeys option.
```
# /etc/ssh/sshd_config
# ...
TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
```
Restart the SSH service to pick up the changes.

Create a named Vault role for signing client keys.

$ vault write ssh-client-signer/roles/sshrole -<<EOH
{
    "algorithm_signer": "rsa-sha2-256",
    "allow_user_certificates": true,
    "allowed_users": "*",
    "allowed_extensions": "permit-pty,permit-port-forwarding",
    "default_extensions": {
    "permit-pty": ""
    },
    "key_type": "ca",
    "ttl": "30m0s"
}
EOH

Client SSH authentication

The following steps are performed by the client (user) that wants to authenticate to machines managed by Vault. These commands are usually run from the client's local workstation.

Locate or generate the SSH public key. Usually this is ~/.ssh/id_rsa.pub. If you do not have an SSH keypair, generate one:
```
$ ssh-keygen -t rsa -C "user@example.com"
```

Ask Vault to sign your public key. This file usually ends in .pub and the contents begin with ssh-rsa ....

$ vault write ssh-client-signer/sign/sshrole -<<EOH
{
    public_key=@$HOME/.ssh/id_rsa.pub,
    "valid_principals": "root"
}
EOH

Example output:

Key             Value
---             -----
serial_number   c73f26d2340276aa
signed_key      ssh-rsa-cert-v01@openssh.com AAAAHHNzaC1...

The result will include the serial and the signed key. This signed key is another public key.

Save the resulting signed, public key to disk. Limit permissions as needed.
```
$ vault write -field=signed_key ssh-client-signer/sign/my-role \
    public_key=@$HOME/.ssh/id_rsa.pub > signed-cert.pub
```
If you are saving the certificate directly beside your SSH keypair, suffix the name with -cert.pub (~/.ssh/id_rsa-cert.pub). With this naming scheme, OpenSSH will automatically use it during authentication.
SSH into the host machine using the signed key. You must supply both the signed public key from Vault and the corresponding private key as authentication to the SSH call.
```
$ ssh -i signed-cert.pub -i ~/.ssh/id_rsa username@10.0.23.5
```

Reference

Signed SSH certificates

API

SSH secrets engine (API)
/sys/managed-keys endpoint