HashiConf 2025 Don't miss the live stream of HashiConf Day 2 happening now View live stream
Nomad
Nomad Home

Connect nodes into a cluster

In order to create a Nomad cluster out of individual nodes, you need to introduce them to one another. There are several ways to perform this:

  • Manual bootstrap
  • Cloud Auto-Join
  • Consul

This guide describes each method and provides configuration snippets, which you can use as starting points for your own configuration.

You may also use client node introduction tokens to restrict which clients join your cluster.

Manual clustering

Manually bootstrapping a Nomad cluster does not rely on additional tooling, but does require operator participation in the cluster formation process. When bootstrapping, Nomad servers and clients must be started and informed with the address of at least one Nomad server.

As you can tell, this creates a chicken-and-egg problem where one server must first be fully bootstrapped and configured before the remaining servers and clients can join the cluster. This requirement can add additional provisioning time as well as ordered dependencies during provisioning.

First, you need to bootstrap a single Nomad server and capture its IP address. Place this address in the configuration once you have that nodes IP address.

For Nomad servers, this configuration may look something like this:

server {
  enabled          = true
  bootstrap_expect = 3

  # This is the IP address of the first server provisioned
  server_join {
    retry_join = ["<known-address>:4648"]
  }
}

Alternatively, you can supply a server's address after the servers have all been started by running the server join command on the servers individually to cluster the servers. All servers can join one other server, and then rely on the gossip protocol to discover the rest.

$ nomad server join <known-address>

For Nomad clients, the configuration may look something like:

client {
  enabled = true
  server_join {
    retry_join = ["<known-address>:4647"]
  }
}

The client node's server list can be updated at run time using the node config command.

$ nomad node config -update-servers <IP>:4647

The port corresponds to the RPC port. If no port is specified with the IP address, the default RPC port of 4647 is assumed.

As servers are added or removed from the cluster, this information is pushed to the client. This means only one server must be specified because, after initial contact, the full set of servers in the client's region are shared with the client.

Use cloud auto-join

The retry_join parameter accepts a unified interface using the go-discover library for doing automatic cluster joining using cloud metadata. To use retry-join with a supported cloud provider, specify the configuration on the command line or configuration file as a key=value key=value ... string. Values are taken literally and must not be URL encoded. If the values contain spaces, backslashes or double quotes they need to be double quoted and the usual escaping rules apply.

{
  "retry_join": ["provider=my-cloud config=val config2=\"some other val\" ..."]
}

Consult the cloud provider-specific configurations in the cloud-autojoin documentation. This can be combined with static IP or DNS addresses or even multiple configurations for different providers. In order to use discovery behind a proxy, you will need to set HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables per Golang net/http library.

Use Consul to automatically cluster nodes

To automatically bootstrap a Nomad cluster, Nomad can leverage another HashiCorp open source tool, Consul. Bootstrapping Nomad is easiest against an existing Consul cluster. The Nomad servers and clients will become informed of each other's existence when the Consul agent is installed and configured on each host. As an added benefit, integrating Consul with Nomad provides service and health check registration for applications which later run under Nomad.

Consul models infrastructures as datacenters and multiple Consul datacenters can be connected over the WAN so that clients can discover nodes in other datacenters. Since Nomad regions can encapsulate many datacenters, you should be running a Consul cluster in every Nomad region and connecting them over the WAN. Refer to the Consul tutorial for both bootstrapping a single datacenter and connecting multiple Consul clusters over the WAN.

If a Consul agent is installed on the host prior to Nomad starting, the Nomad agent will register with Consul and discover other nodes.

For servers, you must inform the cluster how many servers you expect to have. This is required to form the initial quorum, since Nomad is unaware of how many peers to expect. For example, to form a region with three Nomad servers, you would use the following Nomad configuration file:

# /etc/nomad.d/server.hcl

# data_dir tends to be environment specific.
data_dir = "/opt/nomad/data"

server {
  enabled          = true
  bootstrap_expect = 3
}

This configuration would be saved to disk and then run:

$ nomad agent -config=/etc/nomad.d/server.hcl

A similar configuration is available for Nomad clients:

# /etc/nomad.d/client.hcl

datacenter = "dc1"

# data_dir tends to be environment specific.
data_dir = "/opt/nomad/data"

client {
  enabled = true
}

The agent is started in a similar manner:

$ sudo nomad agent -config=/etc/nomad.d/client.hcl

Nomad clients should always run as root (or with sudo). The above configurations include no IP or DNS addresses between the clients and servers. This is because Nomad detected the existence of Consul and utilized service discovery to form the cluster.

Consul auto-join internals

This section discusses the internals of the Consul and Nomad integration at a very high level. Reading is only recommended for those curious to the implementation.

As discussed in the previous section, Nomad merges multiple configuration files together, so the -config may be specified more than once:

$ nomad agent -config=base.hcl -config=server.hcl

In addition to merging configuration on the command line, Nomad also maintains its own internal configurations (called "default configs") which include reasonable base defaults. One of those default configurations includes a "consul" block, which specifies reasonable defaults for connecting to and integrating with Consul. In essence, this configuration file resembles the following:

# You do not need to add this to your configuration file. This is an example
# that is part of Nomad's internal default configuration for Consul integration.
consul {
  # The address to the Consul agent.
  address = "127.0.0.1:8500"

  # The service name to register the server and client with Consul.
  server_service_name = "nomad"
  client_service_name = "nomad-client"

  # Enables automatically registering the services.
  auto_advertise = true

  # Enabling the server and client to bootstrap using Consul.
  server_auto_join = true
  client_auto_join = true
}

Refer to the consul stanza documentation for the complete set of configuration options.

Use client node introduction tokens

Use client introduction tokens to restrict which clients join your cluster. The client node introduction feature is like multi-factor authentication for your Nomad clusters. It does not replace mTLS, but instead adds an additional layer of security that prevents an unauthorized or misconfigured client from joining a Nomad cluster.

When you generate a client introduction token, you may specify the following optional parameters to further secure cluster access:

  • Node pool: The node pool that clients with this token may join. This token is not valid with any other node pool.
  • Node name: The token is scoped to the node with this name. No other node may use this token to join the cluster.
  • TTL: Token expiration. The token is not valid after expiration.

Create a client introduction token

Follow these steps to use client node introduction tokens:

  1. Create an ACL node policy.
  2. Create an ACL role from the policy.
  3. Generate a client introduction token.
  4. Update your Nomad server configuration to use client introduction.
  5. Start the Nomad server.
  6. Monitor client join failures.

Prerequisites

You bootstrapped the ACL system and configured the CLI to use your management token. Refer to the Bootstrap the ACL system guide for instructions.

Create an ACL policy

This example creates the node:write policy required to generate tokens.

  1. Create a policy file called client-introduction.hcl.

    node {
 policy = "write"
}

  2. Add the policy to the cluster.

    nomad acl policy apply client-introduction client-introduction.hcl

Create an ACL role

Create an ACL role with the client-introduction policy.

nomad acl role create --tls-skip-verify -name="client-introduction" \
-policy="client-introduction"

The output describes the ACL role, including its attached policies and the Raft index at its creation.

ID           = cf0b4a43-b00f-cc30-b656-b34d66151b04
Name         = client-introduction
Description  = <none>
Policies     = client-introduction
Create Index = 117
Modify Index = 117

Generate a client introduction token

Use the nomad node intro create command to generate the client introduction token, which is a JSON Web Token (JWT).

This example scopes the token to a node pool named staging and writes the result to a file called intro_token.jwt.

nomad node intro create -node-pool=staging  > intro_token.jwt

The intro-token.jwt file contains the JWT.

 "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgy..."

Update your Nomad server configuration

Configure your Nomad server's server.client_introduction block. This example sets strict enforcement, which means the server rejects any client without a valid token. Refer to the server.client_introduction block documentation for additional enforcement options.

data_dir = "/opt/nomad/"
acl {
  enabled = true
}
server {
  enabled          = true
  bootstrap_expect = 1
  client_introduction {
    enforcement          = "strict" # Default = "warn"
    default_identity_ttl = "5m"     # Default = "5m"
    max_identity_ttl     = "30m"    # Default = "30m"
  }
}
tls {
  http = true
  rpc  = true
  ca_file   = "/opt/nomad/tls/nomad-agent-ca.pem"
  cert_file = "/opt/nomad/tls/global-server-nomad.pem"
  key_file  = "/opt/nomad/tls/global-server-nomad-key.pem"
}

No additional configuration is required on the client node.

Start the Nomad server

You have the following options for passing the client introduction token to the Nomad server:

This example starts the server with the client introduction token passed in the -client-intro-token parameter.

nomad agent -config /etc/nomad.d/nomad.hcl \
-client-intro-token "eyJhbGciOiJSUzI1NiIsImtpZCI6IjljZDgy..."

Monitor client join failures

You have the following options to determine when client registration fails:

  • Check the server logs for [ERROR] nomad.client: node registration without introduction token messages.
  • Monitor the nomad.client.introduction.enforcement counter, which increments when a client tries to join without a valid client introduction token.
Edit this page on GitHub