Boundary health endpoints

Boundary provides health monitoring through the /health path using a listener with the "ops" purpose. By default, a listener with that purpose runs on port 9203. Refer to the example configuration section for an example listener stanza in a configuration file.

Controllers and workers both serve a health endpoint, but they behave differently. The controller endpoint reports its status using the HTTP status code. The worker endpoint always returns 200 OK, and reports its status in the response body.

Requirements

Both endpoints require a listener with the ops purpose in the configuration file of the server you want to monitor.

Controller health requirements

To enable the controller health endpoint, any Boundary instance must be started with a controller. That is, a controller block and a purpose = "api" listener must be defined in Boundary's configuration file. Additionally, a purpose = "ops" listener must also be defined in Boundary's configuration file. Under these conditions, Boundary exposes the ops server, which hosts the controller health API.

Worker health requirements

To enable the worker health endpoint, define a purpose = "ops" listener in the worker's own configuration file. The worker serves its own health endpoint, so you do not query the controller to determine the health of a worker.

If a single Boundary instance runs both a controller and a worker, the instance serves the controller health endpoint. In that case, you request the worker information from the controller endpoint using the worker_info parameter.

Shutdown grace period

When the controller health endpoint is enabled, you can configure the controller to respond with 503 Service Unavailable when it receives a shutdown signal. The controller then waits a configurable amount of time before it starts the shutdown process.

This feature supports load balancing by reducing the risk of an outgoing Boundary instance causing disruption to incoming requests.

In this state, Boundary still processes requests as normal, but reports as unhealthy through the controller health endpoint. In load-balanced environments, the load balancer removes the unhealthy instance from the pool of instances eligible to handle requests. Removing the instance reduces the likelihood that it receives a request during shutdown.

This feature is disabled by default, even if the controller health endpoint is enabled. You can enable it by defining graceful_shutdown_wait_duration in the controller block of Boundary's configuration file. The value should be set to a string that is parseable by ParseDuration.

Health endpoint status codes

The controller health service introduces a single read-only endpoint:

All controller responses return empty bodies. GET /health does not support any input.

The worker health service uses the same path, but it accepts the worker_info query parameter and returns its status in the response body:

Check the health endpoint using wget

The Boundary Docker image includes wget. You can use it to check the health endpoint. Enterprise edition users can check the health of controllers and workers. HCP Boundary users can check the health of their self-managed workers.

Use the following command to check whether a worker is connected to its upstream:

$ wget -q -O - http://localhost:9203/health?worker_info=1 | grep -q 'READY'

• -q - Runs in quiet mode, which suppresses the progress output that wget writes to stderr.
• -O - - Writes the response body to stdout instead of saving it to a file.

The grep -q 'READY' command exits with code 0 if the worker's upstream connection is in the READY state, and a non-zero code otherwise. Because the worker endpoint always returns 200 OK, you must inspect the response body rather than rely on the HTTP status code.

Use the following command to print the full response, which includes the worker's state, active session count, and connection state:

$ wget -q -O - http://localhost:9203/health?worker_info=1

Example response

The following example shows the response from a worker that is connected to its upstream:

$ wget -q -O - http://localhost:9203/health?worker_info=1
{"worker_process_info":{"state":"active","active_session_count":0,"upstream_connection_state":"READY"}}

You can make the same request using curl:

$ curl -s "http://localhost:9203/health?worker_info=1" | jq
{
   "worker_process_info":{
      "state":"active",
      "active_session_count":0,
      "upstream_connection_state":"READY"
   }
}

If you omit the worker_info parameter, the worker returns an empty JSON object:

$ curl -s "http://localhost:9203/health"
{}

The response contains the following fields:

• state - The operational state of the worker. Possible worker states include active, shutdown, and unknown.
• active_session_count - The number of active sessions on the worker.
• session_connections - A map of session IDs to the number of open connections for each session. Boundary omits this field when the worker has no active connections.
• upstream_connection_state - The connection state of the worker. This value indicates whether the worker can connect to an upstream address or connection. It can be any of the following states:
  • CONNECTING - The channel is trying to make a connection and is waiting for name resolution or the connection establishment.
  • READY - The channel has successfully established a connection, and any attempts to communicate have succeeded.
  • TRANSIENT_FAILURE - The channel suffered a transient failure such as a time out or socket error. A channel in this state switches to the CONNECTING state and tries to establish a connection.
  • IDLE - The channel is not attempting to create a connection because there are no calls.
  • SHUTDOWN - The channel is shutting down. Any new calls fail immediately. Pending calls may continue running until Boundary cancels them.

Boundary omits fields that are set to their zero value from the response. A worker that has never connected to an upstream may return only the fields that have values.

Example configuration

Health checks are available for a controller or worker defined with a purpose = "ops" listener stanza. For details on what fields are allowed in this stanza, refer to the documentation about TCP Listener.

An example controller listener stanza:

controller {
  name = "boundary-controller"
  database {
      url = "postgresql://<username>:<password>@10.0.0.1:5432/<database_name>"
  }
}

listener "tcp" {
    purpose = "api"
    tls_disable = true
}

listener "tcp" {
    purpose = "ops"
    tls_disable = true
}

An example worker listener stanza:

worker {
  auth_storage_path = "/var/lib/boundary"
  initial_upstreams = ["10.0.0.1:9201"]
}

listener "tcp" {
    address = "0.0.0.0:9202"
    purpose = "proxy"
}

listener "tcp" {
    address = "0.0.0.0:9203"
    purpose = "ops"
    tls_disable = true
}

To enable a shutdown grace period, update the controller block with a defined wait duration:

controller {
  name = "boundary-controller"
  database {
      url = "env://BOUNDARY_PG_URL"
  }
  graceful_shutdown_wait_duration = "10s"
}

A complete example can be found under the Controller configuration docs.

More information

Refer to the following topics for more information:

• Monitor metrics for the controller and worker metrics that Boundary exports
• Troubleshoot workers for common worker problems and their resolutions
• Manage workers for the worker life cycle operations