Register workers

Workers authenticate to Boundary using an activation token. They require an accessible directory defined by auth_storage_path for credential storage and rotation. Transport level communication between the worker and controller is secured through PKI.

Example (not safe for production!):

worker {
  auth_storage_path="/var/lib/boundary"
  initial_upstreams = ["10.0.0.1"]
}

There are three mechanisms that can be used to initially register a worker to the cluster, controller-led, worker-led, and registration through an external KMS.

Controller-led authorization

In this flow, the operator fetches an activation token from the controller's workers:create:controller-led action (on the CLI, this is via boundary workers create controller-led). That activation token is given to the worker via the controller_generated_activation_token parameter. This can be done either directly or via an env var or file by using env:// or file:// syntax:

worker {
  auth_storage_path="/var/lib/boundary"
  initial_upstreams = ["10.0.0.1"]
  controller_generated_activation_token = "neslat_........."
  # controller_generated_activation_token = "env://ACT_TOKEN"
  # controller_generated_activation_token = "file:///tmp/worker_act_token"
}

Once the worker starts, it reads this token and uses it to authorize to the cluster. Note that this token is one-time-use; it is safe to keep it here even after the worker has successfully authorized and authenticated, as it will be unusable at that point.

If this value is not present at worker startup time and the worker is not authorized, it will print and write out suitable information for the worker-led flow, described below. If the worker-led flow has not been used to authorize the worker, and the controller-generated activation token is provided and the worker restarted, it will make use of it.

Worker-led authorization

In this flow, the worker prints out an authorization request token to two places: the startup information printed to stdout, and a file called auth_request_token in the base of the configured auth_storage_path. This token can be submitted to a controller at the workers:create:worker-led path; on the CLI this would be via boundary workers create worker-led -worker-generated-auth-token. No values are needed in the configuration file.

KMS-led authorization and authentication

In this flow, the worker authenticates upstream, either to a controller or worker, using a shared KMS provided by the customer. This mechanism auto-registers the worker in addition to authenticating it, and does not require on-disk storage for credentials since each time it connects, it re-authenticates using the trusted KMS.

Workers using KMS-led authorization require a name field. This specifies a unique name of this worker within the Boundary cluster and must be unique across workers. The name value can be:

a direct name string (must be all lowercase)
a reference to a file on disk (file://) from which the name is read
an env var (env://) from which the name is read.

Workers using KMS-led authorization accept an optional description field. The description value can be:

a direct description string
a reference to a file on disk (file://) from which the name is read
an env var (env://) from which the name is read.

worker {
  name = "example-worker"
  description = "An example worker"
  public_addr = "5.1.23.198"
}

Workers using the KMS authorization flow also require a KMS block designated for worker-auth. This is the KMS configuration for authentication between the workers and controllers and must be present. Example (not safe for production!):

kms "aead" {
  purpose = "worker-auth"
  aead_type = "aes-gcm"
  key = "X+IJMVT6OnsrIR6G/9OTcJSX+lM9FSPN"
  key_id = "global_worker-auth"
}

The upstream controller or worker must have a kms block that references the same key and purpose. If both a controller and worker are running as the same server process, only one stanza is needed.

Complete configuration example

listener "tcp" {
    purpose = "proxy"
    tls_disable = true
    address = "127.0.0.1"
}

worker {
  # Name attr must be unique across workers
  name = "demo-worker-1"
  description = "A default worker created for demonstration"

  # Workers must be able to reach upstreams on :9201
  initial_upstreams = [
    "10.0.0.1",
    "10.0.0.2",
    "10.0.0.3",
  ]

  public_addr = "myhost.mycompany.com"

  tags {
    type   = ["prod", "webservers"]
    region = ["us-east-1"]
  }
}

# must be same key as used on controller config
kms "aead" {
    purpose = "worker-auth"
    aead_type = "aes-gcm"
    key = "X+IJMVT6OnsrIR6G/9OTcJSX+lM9FSPN"
    key_id = "global_worker-auth"
}

initial_upstreams are used to connect to upstream Boundary clusters.

Downstream worker authentication in multi-hop deployments

In multi-hop deployments, Boundary distinguishes between upstream and downstream workers:

An upstream worker connects directly to controllers or to another upstream worker. It authenticates using a KMS key with the worker-auth purpose.
A downstream worker connects to an upstream worker instead of directly to controllers. It authenticates using the same KMS-based mechanism, but its trust anchor is the upstream worker rather than a controller.

You configure the downstream-worker-auth KMS purpose only on upstream workers. This configuration tells the upstream worker which KMS keys to trust when it validates authentication tokens presented by downstream workers.

Downstream workers use a worker-auth KMS stanza that references the same key material or an equivalent key in the same KMS backend. The upstream worker uses a corresponding downstream-worker-auth stanza that points to that key.

This model lets you:

Use a single KMS key for controller-to-worker trust, worker-auth
Use different KMS keys, or even different KMS backends, to isolate downstream networks, downstream-worker-auth
Delegate authentication of edge workers to trusted upstream workers without exposing controller KMS keys in those environments.

Multiple `downstream-worker-auth` keys per upstream worker

You can specify multiple kms blocks with the purpose = "downstream-worker-auth" attribute on a single upstream worker. Each block represents a separate trust domain for downstream workers.

Some typical use cases include:

Different networks or environments - You could configure one key per data center or per cloud account, for example.
Different sensitivity levels - You could configure separate keys for production or non-production downstream workers.
Gradual migration - You could bring new downstream workers online using a new key, while existing workers continue to use the old key so that you gradually phase out the older key.

A downstream worker must use a worker-auth stanza that references one of the keys configured with downstream-worker-auth on its upstream. The upstream worker rejects any downstream workers that present tokens derived from unknown keys.

Operational considerations for `downstream-worker-auth`

When you design multi-hop topologies that use downstream-worker-auth, keep the following considerations in mind:

Key rotation - Rotating a downstream-worker-auth key requires coordinating the changes on both the upstream and all affected downstream workers. Staggered roll outs in which you add a new key, migrate workers, and then remove the old key minimize downtime.
Blast radius - Because a downstream-worker-auth key defines a trust domain, isolating different environments with distinct keys limits the impact of a compromised key.
KMS backends - Upstream workers can use different KMS backends for worker-auth and downstream-worker-auth. For example, controllers might use an on-premises KMS, while edge workers use a cloud KMS in the target region.
Auditability - Authentication events for workers are recorded in the event log. You can correlate them with the name and the KMS key used to help you verify which trust domain a worker belonged to at a given point in time.

Multi-hop example configuration

The following example shows a controller, an ingress worker that connects to the controller, and an egress worker that connects only to the ingress worker:

Controller configuration:

# Controller uses worker-auth to authenticate workers that connect directly.
kms "awskms" {
  purpose    = "worker-auth"
  key_id     = "arn:aws:kms:us-east-1:111111111111:key/controller-workers"
  region     = "us-east-1"
}

controller {
  # ...
}

Ingress (upstream) worker configuration:

worker {
  name             = "ingress-1"
  initial_upstreams = ["10.0.0.10:9201"] # controller
  public_addr      = "ingress-1.example.internal"
}

# Used to authenticate the ingress worker to the controllers
kms "awskms" {
  purpose    = "worker-auth"
  key_id     = "arn:aws:kms:us-east-1:111111111111:key/controller-workers"
  region     = "us-east-1"
}

# Used by the ingress worker to authenticate downstream workers
kms "awskms" {
  purpose    = "downstream-worker-auth"
  key_id     = "arn:aws:kms:us-east-1:111111111111:key/edge-workers"
  region     = "us-east-1"
}

Egress (downstream) worker configuration:

worker {
  name              = "egress-1"
  initial_upstreams = ["ingress-1.example.internal:9201"]
  public_addr       = "egress-1.example.internal"
}

# Used to authenticate the egress worker to its upstream (ingress-1)
kms "awskms" {
  purpose    = "worker-auth"
  key_id     = "arn:aws:kms:us-east-1:111111111111:key/edge-workers"
  region     = "us-east-1"
}

In this topology:

The controller and ingress worker share the worker-auth key controller-workers.
The ingress worker and egress worker share a different key, edge-workers.
The ingress worker has both purposes configured, worker-auth and downstream-worker-auth, allowing it to act as a client to controllers and as an authenticating upstream for additional workers.

KMS registration

When you use controller or worker-led authentication, a worker’s generated activation token is stored in clear-text on disk. When you use an external KMS, you can encrypt a worker's credentials by including an optional KMS stanza with the purpose worker-auth-storage.

Example (not safe for production!):

kms "aead" {
  purpose = "worker-auth-storage"
  aead_type = "aes-gcm"
  key = "X+IJMVT6OnsrIR6G/9OTcJSX+lM9FSPN"
  key_id = "worker-auth-storage"
}

Note

You can only set the name and description fields for workers that use the KMS-led authorization and authentication method.

Resources

For more on how tags{} in the above configuration are used to facilitate routing to the correct target, refer to Route traffic through a worker.