Add a containerized secrets plugin to Vault

Run your external secrets plugins in containers to increases the isolation between the plugin and Vault.

Before you start

• Your Vault instance must be running on Linux.
• Your Vault instance must have local access to the Docker Engine API. Vault uses the Docker SDK to manage containerized plugins.
• You must have gVisor installed. Vault uses runsc as the entrypoint to your container runtime.
• If you are using a container runtime other than gVisor, you must have a runsc-compatible container runtime installed.

Step 1: Install your container engine

Install one of the supported container engines:

• Docker
• Rootless Docker

Step 2: Configure your container runtime

Update your container engine to use runsc for Unix sockets between the host and plugin binary.

Step 3: Update the HashiCorp go-plugin library

You must build your plugin locally with v1.5.0+ of the HashiCorp go-plugin library to ensure the finished binary is compatible with containerization.

Use go install to pull the latest version of the plugin library from the hashicorp/go-plugin repo on GitHub:

$ go install github.com/hashicorp/go-plugin@latest

Step 4: Build the plugin container

Containerized plugins must run as a binary in the finished container and behave the same whether run in a container or as a standalone application:

1. Build your plugin binary so it runs on Linux.
2. Create a container file for your plugin with the compiled binary as the entry-point.
3. Build the image with a unique tag.

For example, to build a containerized version of the built-in key-value (KV) secrets plugin for Docker:

1. Clone the latest version of the KV secrets plugin from hashicorp/vault-plugin-secrets-kv.
   $ git clone https://github.com/hashicorp/vault-plugin-secrets-kv.git
   
2. Build the Go binary for Linux.
   $ cd vault-plugin-secrets-kv ; CGO_ENABLED=0 GOOS=linux \
   go build -o kv cmd/vault-plugin-secrets-kv/main.go
   
3. Create an empty Dockerfile.
   $ touch Dockerfile
   
4. Update the empty Dockerfile with your infrastructure build details and the compiled binary as the entry-point.
   FROM gcr.io/distroless/static-debian12
   COPY kv /bin/kv
   ENTRYPOINT [ "/bin/kv" ]
   
5. Build the container image and assign an identifiable tag.
   $ docker build -t hashicorp/vault-plugin-secrets-kv:mycontainer .
   

Step 5: Register the plugin

Registering a containerized plugin with Vault is similar to registering any other external plugin that is available locally to Vault.

1. Store the SHA256 of the plugin image:

   $ export SHA256=$(docker images \
       --no-trunc \
       --format="{{ .ID }}" \
       YOUR_PLUGIN_IMAGE_TAG | cut -d: -f2)
   

   For example:

   $ export SHA256=$(docker images \
       --no-trunc \
       --format="{{ .ID }}" \
       hashicorp/vault-plugin-secrets-kv:mycontainer | cut -d: -f2)
   

2. Register the plugin with vault plugin register and specify your plugin image with the oci_image flag:

   $ vault plugin register \
       -sha256="${SHA256}" \
       -oci_image=YOUR_PLUGIN_IMAGE_TAG \
       NEW_PLUGIN_TYPE NEW_PLUGIN_ID
   

   For example:

   $ vault plugin register \
       -sha256="${SHA256}" \
       -oci_image=hashicorp/vault-plugin-secrets-kv:mycontainer \
       secret my-kv-container
   

3. Enable the new plugin for your Vault instance with vault secrets enable and the new plugin ID:

   $ vault secrets enable NEW_PLUGIN_ID
   

   For example:

   $ vault secrets enable my-kv-container
   

Step 6: Test your plugin

Now that the container is registered with Vault, you should be able to interact with it like any other plugin. Try writing then fetching a new secret with your new plugin.

1. Use vault write to store a secret with your containerized plugin:

   $ vault write NEW_PLUGIN_ID/SECRET_PATH SECRET_KEY=SECRET_VALUE
   

   For example:

   $ vault write my-kv-container/testing subject=containers
   Success! Data written to: my-kv-container/testing
   

2. Fetch the secret you just wrote:

   $ vault read NEW_PLUGIN_ID/SECRET_PATH
   

   For example:

   $ vault read my-kv-container/testing
   ===== Data =====
   Key        Value
   ---        -----
   subject    containers
   

Use alternative runtimes

You can force Vault to use alternative runtimes provided the runtime is installed locally.

To use an alternative runtime:

1. Register and name the runtime with vault plugin runtime register. For example, to register the default Docker runtime (runc) as docker-rt:

   $ vault plugin runtime register  \
      -oci_runtime=runc             \
      -type=container docker-rt
   

2. Use the --runtime flag during plugin registration to tell Vault what runtime to use:

   $ vault plugin register  \
      -runtime=RUNTIME_NAME \
      -sha256="${SHA256}"   \
      -oci_image=YOUR_PLUGIN_IMAGE_TAG \
      NEW_PLUGIN_TYPE NEW_PLUGIN_ID
   

   For example:

   $ vault plugin register \
      -runtime=docker-rt   \
      -sha256="${SHA256}"  \
      -oci_image=hashicorp/vault-plugin-secrets-kv:mycontainer \
      secret my-kv-container
   

Troubleshooting

Invalid backend version error

If you run into the following error while registering your plugin:

invalid backend version error: 2 errors occurred:
        * error creating container: Error response from daemon: error while looking up the specified runtime path: exec: " /usr/bin/runsc": stat  /usr/bin/runsc: no such file or directory
        * error creating container: Error response from daemon: error while looking up the specified runtime path: exec: " /usr/bin/runsc": stat  /usr/bin/runsc: no such file or directory

it means that Vault cannot find the executable for runsc. Confirm the following is true before trying again:

1. You have gVisor installed locally to Vault.
2. The path to runsc is correct in you your Docker configuration.
3. Vault has permission to run the runsc executable.

If you still get errors when registering a plugin, the recommended workaround is to use the default Docker runtime (runc) as an alternative runtime.