»Dynamic Secrets for Waypoint with Vault

When an app is deployed with Waypoint, its deployment can also be configured using Waypoint! This can be done with static application configuration, where static environment variables or files can be applied to your running deployment. Alternatively, configuration can be sourced dynamically from external sources, using a config sourcer plugin.

The HashiCorp Vault plugin for Waypoint is one of the config sourcer plugins that is built-in to the Waypoint binary. This plugin can be used to source static values from the Vault KV secrets engine, but also for dynamic secrets! A dynamic secret is one that is managed by Vault and has a lease associated with it, where the secret will expire after a set time to live (TTL). This is a huge benefit for the security of your application, as any possibly leaked secrets would automatically expire after a pre-configured amount of time, among other things including that developers and operators don’t need to know secrets for their deployed applications.

Prior to using Waypoint to deploy Nomad jobs, Nomad users have relied on Nomad’s integration with Vault to template secrets to their applications. Users may still make use of this integration, but in instances where Waypoint is used to deploy an app, it can be abstracted away by using Waypoint’s dynamic configuration integration with Vault, even when not deploying to a Nomad cluster. This use case goes over how this can be done in Waypoint.

An example application, located at this path in the waypoint-examples GitHub repository, will eventually need read-only access from a Postgres database, and should provide dynamic credentials to access it. To make this happen, a role will be created in the Postgres database with read-only access, using the psql command line tool. The IP, port, and user below should be filled in based on the configuration and environment in which your Postgres instance is running.

$ psql -h <POSTGRES_IP> -p <POSTGRES_PORT> -U <POSTGRES_USER> -c "CREATE ROLE \"ro\" NOINHERIT;"

$ psql -h <POSTGRESIP> -p <POSTGRES_PORT> -U postgres -c "GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"ro\";"

There is already a Vault cluster up and running, and a database secrets engine will be configured and enabled, so that Vault can connect to a database and generate credentials dynamically.

$ vault secrets enable database
Success! Enabled the database secrets engine at: database/

$ vault write database/config/postgresql \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@<POSTGRES_ADDR>/postgres?sslmode=disable" \
  allowed_roles=readonly \
  username="<POSTGRES_USER>" \
  password="<POSTGRES_PASSWORD>"

Success! Data written to: database/config/postgresql

In the database secrets engine, a connection and role have been configured. With this, authorized entities in Vault will be able to generate dynamic credentials using the secrets engine.

$ tee readonly.sql <<EOF
CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' INHERIT;
GRANT ro TO "{{name}}";
EOF

$ vault write database/roles/readonly \
  db_name=postgresql \
  creation_statements=@readonly.sql \
  default_ttl=1h \
  max_ttl=24h
Success! Data written to: database/roles/readonly

The final steps in Vault are to create a Vault policy, which will allow an entity to access the database secrets engine and generate a dynamic credential for the database, and to create a token with this policy attached to it:

$ tee waypoint-policy.vault.hcl <<EOF
path "database/creds/readonly" {
  capabilities = [ "read" ]
}
EOF

$ vault policy write waypoint waypoint-policy.vault.hcl
Success! Uploaded policy: waypoint-policy

$ vault token create -policy=waypoint -ttl=24h
Key                  Value
---                  -----
token                <VAULT_TOKEN>
token_accessor       <VAULT_TOKEN_ACCESSOR>
token_duration       24h
token_renewable      true
token_policies       ["default" "waypoint"]
identity_policies    []
policies             ["default" "waypoint"]

Now that Vault is configured to produce dynamic database credentials, Waypoint needs to be configured to connect to Vault. A configuration source for the Vault plugin will be added like so:

$ waypoint config source-set -type=vault -config=addr=http://127.0.0.1:8200 -config=token=<VAULT_TOKEN>
Configuration set for dynamic source "vault"!

To double-check that the token and address are what is expected, waypoint config source-get can be run:

$ waypoint config source-get -type=vault
  KEY  |           VALUE
--------+----------------------------
  addr  | http://127.0.0.1:8200
  token | <VAULT_TOKEN>

At this point, the config source isn't actually being used yet, but that can be made ready by adding it to the waypoint.hcl file, and committing it to the git repository for the project.

project = “waypoint-dynamic-config-vault”

app “my-app” {
  build {
    use “pack” {}

    registry {
      use “docker” {
        image = var.image_name
        tag   = var.image_tag
        auth {
          username = var.username
          password = var.password
        }
      }
    }
  }

  deploy {
    use “nomad” {
      // The application code is written such that the app will listen on port 80,
      // so Nomad should configure the job to listen on this port.
      service_port = 80
    }
  }

  config {
    env = {
      // These environment variables use dynamic configuration from Vault's dynamic
      // database secrets engine
      USERNAME = dynamic(“vault”, {
        path = “database/creds/my-role”
        key = “username”
      })

      PASSWORD = dynamic(“vault”, {
        path = “database/creds/my-role”
        key = “password”
      })

      HOST   = var.postgres_ip
      PORT   = var.postgres_port
      DBNAME = var.postgres_dbname
    }
  }
}

variable "postgres_ip" {
  description = "IP address at which the Postgres DB server is listening."
  type        = string
}

variable "postgres_port" {
  description = "Port on which the Postgres DB server is listening."
  type        = number
  default     = 5432
}

variable "postgres_dbname" {
  description = "Name of the Postgres DB to access."
  type        = string
  default     = “postgres”
}

variable "image_name" {
  decription = "Name of the Docker image for the app built by Waypoint. This will be used with the tag to push an image to a Docker repository."
  type = string
}

variable "image_tag" {
  decription = "Tag of the Docker image for the app built by Waypoint. This will be used with the name to push an image to a Docker repository."
  type = string
}

variable "registry_username" {
  description = "Username of user with push access to your Docker registry."
  type      = string
  sensitive = true
}

variable "registry_password" {
  description = "Password for the configured user with push access to your Docker registry."
  type      = string
  sensitive = true
}

In the configuration above, Waypoint has been informed that it must get a secret from Vault at the path database/creds/my-role. To learn more about how Waypoint does this, see the Waypoint Entrypoint docs.

The values for the keys username and password at the secret path will be configured as environment variables, USERNAME and PASSWORD, in the deployment of the app. We can confirm that the secret was successfully templated after running waypoint up, by checking the environment variables using waypoint exec, and checking the application logs using waypoint logs:

// To run this operation, input variables were exported to the environment, prefixed with “WP_VAR_”
// The output below is abbreviated for brevity
$ waypoint up
✓ Running build v2
✓ Building Buildpack with kaniko...
✓ Repository is available and ready: devopspaladin/hello-database:latest
✓ Executing kaniko...
…
✓ Image pushed to 'devopspaladin/hello-database:latest'
✓ Running push build v1
✓ Running deploy v2
✓ Job registration successful
✓ Allocation "4a0788ae-a6c0-bf55-4a84-46a4f0ff4a81" created: node "b9a637ea-6098-4b22-6f52-9d5b94581a75", group
"dynamic-go-01gj1s1sqj31r72qn38t09xj7p"
✓ Evaluation status changed: "pending" -> "complete"
✓ Evaluation "d1c03461-fddd-2e68-bb7b-d6e7e718f121" finished with status "complete"
✓ Deployment successfully rolled out!
✓ Job "dynamic-go-01gj1s1sqj31r72qn38t09xj7p" is reporting ready!
✓ Finished building report for Nomad platform

// The values for USERNAME and PASSWORD below are dynamic credentials generated by Vault!
$ waypoint exec env
…
USERNAME=v-token-readonly-wOdEtyXQ5MktQ5SwTnXo-1668702202
PASSWORD=e-4QVqEpUhGfmpTncKTy
…

$ waypoint logs
...
2022-11-17T16:41:36.491Z D60TJX: 2022/11/17 16:41:36 Successfully connected to the database! :)

With the application successfully deployed and configured with dynamic credentials, it can securely connect to the database, and no user ever needed to provide the app with an actual secret value. The Waypoint Entrypoint, built-in to the application image, can now dynamically source this Vault configuration wherever the app is deployed. If a user deployed to Docker on their laptop for some local development testing, or an EKS cluster in AWS, as long as the application container can access the Vault cluster, the secrets necessary to run the application will be provided by Waypoint.

»Dynamic Secrets for Waypoint with Vault

Further Reading