Read secrets from Vault using Vault Agent

Nearly all requests to Vault must be accompanied by a valid client token. This includes all API requests, as well as via the Vault CLI and other libraries. Therefore, Vault clients must first authenticate with Vault to acquire a token, and Vault provides a number of different authentication methods to assist in the delivery of this initial token.

As an application developer, you must implement Vault API to perform the following:

• Authenticate with Vault to acquire a client token (refer to the above diagram).
  • If the application makes multiple requests to Vault, manage the lifecycle of the client token (e.g. renew or revoke the token TTL).
• Request secrets from Vault using the client token.
  • Manage the leases of dynamic secrets (if applicable).

Most programming languages and frameworks provide a way to implement the necessary HTTP requests to perform this work, and some language communities have implemented clients to assist with these operations.

Ultimately, your applications need changes to use Vault. This implies code changes to client applications, along with additional testing, error handling, and maintenance of the application.

package main

import (
    ...snip...
    vault "github.com/hashicorp/vault/api"
)

// Fetches a key-value secret (kv-v2) after authenticating via AppRole
func getSecretWithAppRole() (string, error) {
    config := vault.DefaultConfig() 

    client := vault.NewClient(config)
    wrappingToken := ioutil.ReadFile("path/to/wrapping-token") 
    unwrappedToken := client.Logical().Unwrap(strings.TrimSuffix(string(wrappingToken), "\n"))
   
    secretID := unwrappedToken.Data["secret_id"]
    roleID := os.Getenv("APPROLE_ROLE_ID")

    params := map[string]interface{}{
        "role_id":   roleID,
        "secret_id": secretID,
    }
    resp := client.Logical().Write("auth/approle/login", params)
    client.SetToken(resp.Auth.ClientToken)

    secret, err := client.Logical().Read("kv-v2/data/creds")
    if err != nil {
        return "", fmt.Errorf("unable to read secret: %w", err)
    }

    data := secret.Data["data"].(map[string]interface{})
    
    ...
}

What is Vault Agent

Vault Agent is a client daemon that behaves as a Vault client running alongside your applications to perform the previously mentioned authentication and secrets retrieval operations.

• Auto-Auth - Vault Agent can automatically authenticate to Vault, and manage the token renewal process for locally-retrieved dynamic secrets.
• Templating - Allows rendering of user supplied templates by Vault Agent, using the token generated by the Auto-Auth step. Templates allow secrets to be rendered to files, for simpler access by client applications.

This tutorial demonstrates the Vault Agent features which help to make your application to be Vault un-aware, and minimize required code changes.

Phase 1: Authentication

Vault Agent Auto-Auth can perform authentication and manage the token renewal process for locally-retrieved dynamic secrets. It stores the acquired client token in the configured sink location. The application can simply read the token and start making requests to Vault. Your application does not need to implement Vault API to authenticate with Vault.

Vault Agent will keep the resulting token renewed until renewal is no longer allowed or fails, at which point it will attempt to re-authenticate.

Phase 2: Read secrets from Vault

Vault Agent Templates allows Vault secrets to be rendered to files using Consul Template markup. Your application simply accesses the rendered output file; therefore, there is no need to implement Vault API in your application code.

Leveraging the Vault Agent Template feature, the application no longer needs to directly talk to Vault.

Phase 3: Scale

Vault Agent becomes a sidecar to every application, and an organization may have multiple applications requiring secrets that are stored in Vault.

You may need to:

• Run Vault Agent as a service on the client host (where applications run).
• Create multiple templated files (one per application).

Hands-on lab

In this tutorial, you are going to run a simple web application (workshop-webapp). It displays data read from keys.yaml. The challenge is that the web application must read data from Vault to display them. The application developer wants to avoid making any code change to the web application codebase so they decided to leverage Vault Agent.

Vault Agent Auto-Auth handles the authentication and token management (step 1~3 in the diagram). The auto_auth stanza in the Vault Agent configuration file defines the auth method and the targeted sink file location to store the token.

auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "role_id-webapp"
      secret_id_file_path = "secret_id-webapp"
      remove_secret_id_file_after_reading = false
    }
  }

  sink "file" {
    config = {
      path = "./agent-token"
    }
  }
}

Upon a successful authentication, the Vault Agent uses the acquired token to read secrets stored at external-apis/data/socials/twitter. Vault Agent Templates uses the Consul Template markup to define which key values to parse, and output a file.

{{ with secret "external-apis/data/socials/twitter" }}
api_key: {{ .Data.data.api_key }}
secret_key: {{ .Data.data.api_secret_key }}
{{ end }}

...snip...

template {
   source      = "/application/keys.ctmpl"
   destination = "/application/keys.yaml"
}

Launch the lab environment

Launch the interactive tutorial, and follow its step-by-step instruction to learn Vault Agent. Click the Show Terminal button to start.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Start interactive lab

Help and reference

• Blog post: Why Use the Vault Agent for Secrets Management?
• Video: Streamline Secrets Management with Vault Agent and Vault 0.11
• Vault Agent Templates documentation
• Consul Template - Templating Language
• Direct Application Integration

Other Vault Agent tutorials:

• Vault Agent with AWS
• Vault Agent with Kubernetes
• Vault Agent Templates
• Vault Agent Caching
• Vault Agent Windows Service
• Using HashiCorp Vault Agent with .NET Core