SSH Certificate Injection with HCP Boundary
Boundary 0.12 added support for credential management of SSH certificates. HCP Boundary introduces credential injection for SSH certificates into sessions, using the HashiCorp Vault SSH Secrets Engine. Currently users must bring their own Vault deployment to HCP Boundary.
This tutorial demonstrates SSH certificate injection into an SSH server configured using Docker. Learners may also bring their own target to use for this tutorial.
Tutorial overview
- Prerequisites
- Background
- Set up an SSH target
- Set up Vault
- Set up Boundary
- Inject certificates into sessions
Prerequisites
This tutorial assumes the learner has completed the HCP Boundary Getting Started tutorials. The learner should have a working Boundary cluster and org running on HCP.
Docker is installed (Note: Learners may also bring their own target, instead of using Docker)
A Boundary binary greater than 0.12.0 in your
PATH
A Vault binary greater than 1.12.0 in your
PATH
is recommended. Any version of Vault greater than 1.7 should work with this tutorial.The
jq
utility installed and in yourPATH
Installing the Boundary Desktop App provides an optional workflow at the end of this tutorial. The 1.2.0 version or above is required for Vault support.
This tutorial assumes basic knowledge of using Vault, including running a development server and managing policies, roles, and tokens. If you are new to using Vault, complete the Getting Started with Vault quick start tutorials before integrating Vault with Boundary.
This tutorial also extends the workflow for credential brokering via Vault. If new to this process, we recommend completing the Vault Credential Brokering Quickstart tutorial before moving forward.
Certificate injection background
SSH, also known as Secure Shell, is arguably the de facto tool for remote administration, enabling end users and administrators with a simple, powerful and secure access method. SSH offers a variety of benefits and configurations, such as:
- Encryption
- Support for multiple keys types and sizes
- Built in key exchange
- Ability to create a tunnel and spawn multiple channels
Boundary seamlessly provides authenticated users access to remote machines via their SSH client, regardless of the authentication mechanisms implemented by their administrators.
Boundary supports credential brokering, where Boundary controllers check out credentials from Vault and return them back to users and clients. In this workflow, Boundary acts as a broker of the credential.
Brokering lacks the ability to hide credentials from clients. With credential injection, controllers instead return credentials to Boundary workers and create a session to a target where the client never has access to the credential. This workflow is called credential injection because the credential is injected into a worker’s session, instead of being passed back to the client.
Starting in Boundary 0.10, users can inject SSH private keys, as well as usernames and passwords within Boundary. With public key authentication, keys for each user need to be distributed to all of the correct hosts. In contrast, SSH certificates require administrators to distribute a certificate authority’s (CA) public key to the hosts.
Boundary 0.12 added support for credential injection of Vault SSH certificates. Boundary uses Vault as a certificate authority for SSH certificates by integrating with the Vault SSH Secrets Engine.
Boundary administrators can use either the /issue
or /sign
endpoints to
generate certificates from Vault.
This tutorial assumes the reader has installed Boundary 0.12.0 or above, and a Vault binary greater than 1.7.0. Vault can be run in development mode locally, as demonstrated in this tutorial, or by supplying a Vault public address to HCP Boundary.
Set up Vault
Vault needs to be configured to generate secrets for the database target. To set up Vault you need to:
- Start Vault
- Set up a Boundary controller policy
- Enable the database secrets engine
- Configure the postgres database plugin
- Create a database admin role to generate credentials
- Create an analyst role to generate credentials
Public or private Vault access is supported by HCP Boundary. If using private Vault or a Vault dev server, a self-managed worker also needs to be deployed. Select one of the options for configuring Vault to continue.
Integrating private Vault with HCP Boundary requires a self-managed PKI worker to be deployed in the same network as Vault. If a Vault credential store is defined with a worker filter, Boundary’s controller will know that Vault requests should be routed to workers matching that filter.
Private Vault clusters running on HCP can be configured to access HCP Boundary via a self-managed worker placed in a peered network via a HashiCorp Virtual Network (HCVN).
For simplicity, this tutorial demonstrates integrating HCP Boundary with a private Vault cluster running locally in development mode with the PKI worker also running locally.
To learn more about HCVNs, refer to the HCP Virtual Network tutorial collection.
To learn more about deploying a self-managed PKI worker for HCP, refer to the Self-Managed Worker Registration with HCP Boundary tutorial.
Open a new terminal session and run Vault in development mode.
Leave Vault running in dev mode and open a new terminal window.
Next, export the required Vault environment variables for the address and token.
Then enable the ssh secrets engine at ssh-client-signer/
.
Deploy a self-managed worker
An HCP worker deployed on the same network as Vault is required for integrating private Vault clusters with HCP Boundary. Additionally, a self-managed worker is also needed to route traffic to targets on private networks, like the Docker container in this tutorial. To learn more about setting up self-managed workers, refer to the Self-Managed Worker Registration with HCP Boundary tutorial.
Note
If using both a public Vault endpoint and bringing your own target with a public IP address, skip to the [Set up an SSH target](#set-up-an-ssh-target) section. If bringing your own target on a private network, a self-managed worker must be deployed on the same VPC as the target.Download the boundary-worker binary
Download the HCP Worker binary to the ~/boundary/
directory.
You can manually download the latest binary for your operating system by navigating to the HCP Boundary boundary-worker releases page. The example below demonstrates downloading the binary using the command line.
Note
The binary version should match the version of the HCP control plane. Check the version of the control plane in the HCP Boundary portal, and download the appropriate version using wget. The example below installs the 0.12.2 version of the boundary-worker binary, which is the minimum version for SSH certificates support.Below is an example of downloading and unzipping the boundary-worker binary on Ubuntu, MacOS, and Windows.
The following command downloads the boundary-worker binary and unzips it to the current directory.
Once downloaded, verify the version of the boundary-worker Boundary.
Ensure the Version Number matches the version of the HCP Boundary control plane. They should match in order to get the latest HCP Boundary features.
Write the PKI worker config
Next, open the pki-worker.hcl
file with a text editor.
Update the worker config file:
1 2 3 4 5 6 7 8 9 101112131415
Update the cluster id in the pki-worker.hcl
file:
The <cluster-id>
on line 3 can be determined from the UUID in the HCP
Boundary Cluster URL. For example, if your Cluster URL is:
https://c3a7a20a-f663-40f3-a8e3-1b2f69b36254.boundary.hashicorp.cloud
,
then the cluster id is c3a7a20a-f663-40f3-a8e3-1b2f69b36254
The auth_storage_path
should match the full path to the ~/boundary/worker1
directory, such as /home/myusername/boundary/worker1
.
Save this file.
Start the PKI worker
With the worker config defined, start the worker server. Provide the full path
to the worker config file (such as /home/myusername/boundary/pki-worker.hcl
).
1 2 3 4 5 6 7 8 9 101112131415161718
The worker will start and begin attempting to connect to the upstream Controller.
The worker also outputs its authorization request as the Worker Auth
Registration Request token. This will also be saved to a file,
auth_request_token
, defined by the auth_storage_path
in the worker config.
Note the Worker Auth Registration Request:
value. This value can also be
located in the ~/boundary/worker1/auth_request_token
directory. Copy this
value.
Register the worker with HCP
PKI workers can be registered using the Boundary CLI or Admin Console Web UI.
PKI workers can be registered and managed via the Boundary Admin Web UI.
Authenticate to HCP Boundary as the admin user:
Log in to the HCP portal.
From the HCP Portal's Boundary page, click Open Admin UI - a new page will open.
Enter the admin username and password you created when you deployed the new instance and click Authenticate.
Once logged in, navigate to the Workers page.
Notice that only HCP KMS workers are listed.
Click New.
Scroll down to the bottom of the New PKI Worker page and paste the Worker Auth Registration Request key you copied earlier.
Click Register Worker.
Click Done and notice the new PKI worker on the Workers page.
Define the controller policy
As described in the OSS Vault Credential Brokering Quickstart tutorial, the following Vault policy must be defined for the Boundary controller to support credential brokering.
Write the controller policy to Vault.
Define the credential and ssh policies
Vault requires an SSH policy to issue, sign, and update credentials.
Write the ssh policy to Vault.
Lastly, create a role for testing the certificates.
Create a Vault token
Create the a Vault token to be used by Boundary later on, and export it as the
$CRED_STORE_TOKEN
variable**.
Note
If you plan on using the Admin Console UI to connect, save this value for creating a credential store later on.
Set up an SSH target
Next, set up a target to test certificate injection using Vault.
Note
Alternatively, you can bring your own target to test the certificate injection. If bringing your own target, expand the accordion below to learn about the settings needed for SSH.The example below demonstrates configuring a Debian or RHEL target.
First, generate the CA key for signing certificates and add it to Vault. Note that
the jq
utility is used to parse the public key. Refer back to the tutorial
prerequisites if you need to install jq
.
The command above has Vault generate a signing certificate, and then jq is used
to store the key at ca/ca-key.pub
.
Next, set up SSH on the target to trust the CA.
Copy the /ca
directory to the target at /ca
.
Next, execute the following commands on the target:
Lastly, restart ssh.
Next, skip to the Set up Boundary section.
Docker is used to deploy an openssh-server target for testing credential injection.
For the purposes of this tutorial, create an ssh keypair that will be used by Vault when signing certificates.
First, open a terminal session and create a new directory to store the keypair
in. This tutorial recommends creating the openssh
directory within your home
directory at ~/
, but you can place this directory elsewhere if desired, so
long as you deploy the Docker container in the same directory.
Now create two new directories, ca/
and custom-cont-init.d/
.
Set up the CA key
Next, generate the CA key for signing certificates and add it to Vault. Note that
the jq
utility is used to parse the public key. Refer back to the tutorial
prerequisites if you need to install jq
.
The command above has Vault generate a signing certificate, and then jq is used
to store the key at ca/ca-key.pub
. This directory will be copied onto the
docker container target in a moment, so the target has access to the CA's public
key.
Next, create a startup script to be used by the open-ssh container. This script sets appropriate permissions on the CA public key so the open-ssh server will trust the Vault CA.
Deploy the target
Ensure Docker is running, and then deploy the openssh container.
Supply the following options to docker run
:
- The absolute path to the
ca/
andcustom-cont-init.d/
directories created earlier after the--volume=
option,** for example at/Users/myusername/openssh/ca
and/Users/myusername/openssh/custom-cont-init.d/
.
This will deploy the openssh container and mount the ca/
and
custom-cont-init.d/
folder contents as shared volumes.
Update lines 10 and 11 with correct local paths, then execute the following
command from the openssh/
directory.
1 2 3 4 5 6 7 8 9 101112
Note that the username for the openssh server is admin
, and the target is
available on your localhost at port 2222
(127.0.0.1:2222
).
Leave this container running for the duration of the tutorial.
Set up Boundary
Start by logging in to HCP Boundary within the terminal.
Log in to the HCP portal.
From the HCP Portal's Boundary page, click Open Admin UI - a new page will open.
Enter the admin username and password you created when you deployed the new instance and click Authenticate.
Next, set up a new ssh testing org and project scope.
Note
You may use an existing org scope, or create a new org scope for the ssh-project. Refer to the [Manage Scopes](/boundary/tutorials/hcp-administration/hcp-manage-scopes) tutorial to learn more about creating and managing scopes.Navigate to to the Orgs page and click New Org.
Fill out the new org form with a Name of
testing-org
and Description ofSSH testing org
. Click Save.From within the new org, click New Project.
Fill out the new project form with a Name of
ssh-project
and Description ofSSH test machines
. Click Save.
Create a credential store
Next, create a new credential store within Boundary using the new token. The
vault
credential store type is used for Vault integration, but static
credential stores can also be used with credential injection.
Create a new Vault credential store.
Navigate to the Credential Stores page and click New Credential Store.
Fill out the New Credential Store form. Supply the Credential Store Token copied earlier for the Token value:
- Name:
certificates-store
- Type:
Vault
- Address:
http://127.0.0.1:8200
- Worker Filter:
"vault" in "/tags/type"
- Token: <CRED_STORE_TOKEN>
Click Save.
- Name:
Create credential library
Create a new credential library of type vault-ssh-certificate
within
Boundary using the credential store ID and passing the vault-path of
ssh-client-signer/sign/boundary-client
.
Click the newly created certificates-store. Navigate to the Credential Libraries tab. Click the Mange dropdown and select New Credential Library.
Select a Type of SSH Certificates and fill out the New Credential Library form:
- Name:
certificates-library
- Type:
SSH Certificates
- Vault Path:
ssh-client-signer/sign/boundary-client
- Username:
admin
- Key Type:
ecdsa
- Key Bits:
521
- Extensions: Key:
permit-pty
Click Save.
Note
The Vault `ssh-client-signer/sign` endpoint can also be used with Vault 1.12.0 and above.- Name:
Create an SSH target
Create a new target of type ssh
and set its address to 127.0.0.1
and default
port to 2222
.
Note
Learners may also bring their own target to test credential injection. If bringing your own target, enter the target's public IP for the `-address` field, and SSH connection port (likely `22` for SSH) for the `-default-port` option.Navigate to the Targets page and click New Target.
Fill out the New Target form. Select a Type of SSH.
- Name:
openssh-target
- Type:
SSH
- Target Address:
127.0.0.1
- Default Port:
2222
- Maximum Connections:
-1
- Egress worker filter:
"worker" in "/tags/type"
Click Save.
Note
The Egress worker filter may be omitted if not using a self-managed worker, such as when using both a public Vault endpoint and a target with a public IP address.
- Name:
Now associate the SSH target with the SSH credential library.
Click on the new
openssh-target
to view its details page.Select the Injected Application Credentials tab. Click +Add Injected Application Credentials.
Select the
vault-cert-library
credential of typeSSH Certificates
.Click Add Injected Application Credentials.
Inject certificates into sessions
Now you are ready to inject certificates directly into a shell session. You can
accomplish this using the Boundary Desktop App, or by using the boundary
connect
command.
To log into Boundary using the Desktop App, the BOUNDARY_ADDR
(Boundary
cluster address) and BOUNDARY_AUTH_METHOD_ID
(user Auth Method ID) values must be
gathered from the HCP Boundary Admin Console, as demonstrated in the HCP
Boundary Getting Started
tutorial.
Open the Boundary desktop app.
Enter the Boundary cluster URL (for example,
https://ffee961b-5fd8-4e68-ba1d-2bbb487b576e.boundary.hashicorp.cloud
) and
click Submit.
Authenticate using your HCP Boundary user credentials.
Under the Targets page, notice the target details for ssh-target.
Click Connect to initiate a session.
The Successfully Connected page will display the target ID (Target Connection details) and Proxy URL.
To start a session, open your terminal or SSH client. A session can be started using SSH and the Proxy URL from the Boundary desktop app.
For the openssh docker target, connect on 127.0.0.1 and provide the proxy
port using the -p
option. Enter yes
when prompted to establish a connection.
When finished, the user can close the connection to the server by entering
exit
, or the session can be canceled directly from the Boundary desktop app
under the Sessions view.
Cleanup and teardown
Locate the terminal session used to execute the vault dev
server command, and
execute ctrl+c
to stop Vault.
Unset the environment variables used in any active terminal windows for this tutorial.
Destroy the openssh-server container created for the tutorial.
Check your work by executing docker ps
and ensure there are no more containers
from the tutorial leftover. If unexpected containers still exist, execute
docker rm -f <CONTAINER_ID>
against each to remove them.