Create a Nomad cluster

11min
|
Nomad
Terraform

Nomad is a flexible scheduler that can run on your local machine as a single node cluster or as a multi-node cluster on virtualized or physical infrastructure.

In this tutorial, you will create a cluster locally on your machine or on one of the major cloud providers, verify connectivity to the cluster, and interact with Nomad via the UI.

Prerequisites

Docker Desktop installed locally

Nomad supports many task drivers but the example application in this tutorial uses Docker. Docker Desktop provides the Docker engine, the CLI client, and a graphical interface. Other third-party options are available for running Docker on your machine but we recommend Docker Desktop for the simplicity and ease of use.

Nomad uses a socket file to communicate with Docker and by default, this file is /var/run/docker.sock on Linux, Mac, and other Unix platforms, and /./pipe/docker_engine on Windows. Docker Desktop creates a symlink to this socket during installation.

If the Nomad client fails to detect the Docker driver with the error Constraint "missing drivers": 1 nodes excluded by filter, the issue may be this missing symlink. Additional information about this symlink for Mac and Windows is available.

Clone the code repository

The example repository includes Terraform configurations for starting a cluster on the cloud as well as the jobspec files for the example application.

Clone the code repository.

$ git clone https://github.com/hashicorp-education/learn-nomad-getting-started

Navigate to the repository folder.

$ cd learn-nomad-getting-started

Check out the v1.1 tag of the repository as a local branch named nomad-getting-started.

$ git checkout -b nomad-getting-started v1.1
Switched to a new branch 'nomad-getting-started'

Create the cluster

Start a Nomad cluster. You will use the Nomad CLI to access your cluster regardless of its location. Select where you want to create and run your cluster.

Open your terminal and start the development agent. This creates a Nomad cluster of one node that acts as both the server and client.

Warning

This tutorial uses the network interface attached to the default route on your machine and as a result, workloads will be accessible to other machines on the same network. This is for development and illustration purposes. We recommend using a firewall for production environments or when connected to public networks to limit unauthorized workload access.

The -bind flag set to 0.0.0.0 instructs Nomad to listen on all network interfaces present on the machine. The -network-interface flag instructs Nomad to use a network interface other than the default loopback interface (localhost). In this example, Nomad automatically gets the network interface attached to the default route on the machine with the GetDefaultInterfaces function but you can also provide the name of an interface. This flag is necessary for the example application as each service runs in a different container and cannot reach the other services on the loopback address.

Note

You may encounter a permission denied error when creating the allocation directory. If so, run the nomad agent command without sudo.

Leave Nomad running in this terminal session.

$ sudo nomad agent -dev \
  -bind 0.0.0.0 \
  -network-interface='{{ GetDefaultInterfaces | attr "name" }}'

In a second terminal session, export the cluster address.

$ export NOMAD_ADDR=http://localhost:4646

Each of the following cloud setups use Terraform to create and configure the server and client nodes. During this process, Terraform installs the dependencies and the Nomad binary, and configures Nomad with user startup scripts.

Be sure that you have Terraform installed and configured.

This process will take about 5 minutes depending on the cloud provider.

Warning

We do not recommend this guide for a production deployment of Nomad. It is a method for quickly creating a multi-node cluster for learning, development, and testing purposes. When you are ready to move on to a production deployment, check out the cluster setup collection and the reference architecture guide for best practices.

Choose the cloud provider below that you want to use.

Make sure that you have your AWS access credentials set as environment variables.

Navigate to the aws folder.

$ cd aws

Rename the example variables file to terraform.tfvars.

$ mv terraform.tfvars.example terraform.tfvars

Update the region variable in terraform.tfvars with your AWS region preference. Save the file.

region      = "us-east-2"

Make sure that you have the gcloud CLI tool installed on your machine.

Navigate to the gcp folder.

$ cd gcp

Rename the example variables file to terraform.tfvars.

$ mv terraform.tfvars.example terraform.tfvars

$ gcloud auth login
Your browser has been opened to visit:

https://accounts.google.com/o/oauth2/auth?response_type=code[...]

You are now logged in as [YOUR_GCP_ACCOUNT].
Your current project is [YOUR_CURRENT_PROJECT].  You can change this setting by running:
  $ gcloud config set project PROJECT_ID

Set the project, region, and zone configurations in gcloud.

Note

If you already have a project in your GCP account, these configurations will be set for you as part of the login step. If not, first create a project.

Set project to the project ID.

$ gcloud config set project <GCP_PROJECT_ID>
Updated property [core/project].

Then, set region to the associated region.

$ gcloud config set compute/region <GCP_REGION>
Updated property [compute/region].

Finally, set zone to the associated zone. Note that the zone must be in the region set above.

$ gcloud config set compute/zone <GCP_ZONE>
Updated property [compute/zone].

List the configurations with gcloud.

$ gcloud config list
[compute]
region = us-east1
zone = us-east1-b
[core]
account = [GCP_ACCOUNT]
disable_usage_reporting = True
project = hc-3ff63253e6a54756b207e4d4727

Copy the values for project, region, and zone into terraform.tfvars. In this example, those would be hc-3ff63253e6a54756b207e4d4727, us-east1, and us-east1-b. Save the file.

region      = "us-east1"
zone        = "us-east1-b"
project     = "hc-3ff63253e6a54756b207e4d4727"

Make sure that you have the az CLI tool installed on your machine.

Navigate to the azure folder.

$ cd azure

Rename the example variables file to terraform.tfvars.

$ mv terraform.tfvars.example terraform.tfvars

Open your terminal, log in to Azure with az, and follow the prompts to complete the login process.

$ az login
A web browser has been opened at https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize.
Please continue the login in the web browser. If no web browser is available or if the web browser
fails to open, use device code flow with `az login --use-device-code`.
[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29",
    "id": "0e3e2e88-47a3-4107-a2b2-f325314dfb67",
    "isDefault": true,
    "managedByTenants": [
      {
        "tenantId": "c9ed8610-2016-4bf5-b919-437a07bf2464"
      }
    ],
    "name": "[SUBSCRIPTION_NAME]",
    "state": "Enabled",
    "tenantId": "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29",
    "user": {
      "name": "[USER_EMAIL]",
      "type": "user"
    }
  }
]

Copy the values for id and tenantId and paste them into the terraform.tfvars file as values for subscription_id and tenant_id. For this example, the value for subscription_id would be 0e3e2e88-47a3-4107-a2b2-f325314dfb67 and tenant_id would be 1e472a2a-7ab3-9bd1-2016-a32fd04dfb29. Save the file.

location = "LOCATION"
subscription_id = "0e3e2e88-47a3-4107-a2b2-f325314dfb67"
tenant_id = "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29"
client_id = "CLIENT_ID"
client_secret = "CLIENT_SECRET"

Next, create an Azure service principal by providing the value of subscription_id to the --scopes argument.

$ az ad sp create-for-rbac \
  --role="Contributor" \
  --scopes="/subscriptions/0e3e2e88-47a3-4107-a2b2-f325314dfb67"

Creating 'Contributor' role assignment under scope '/subscriptions/0e3e2e88-47a3-4107-a2b2-f325314dfb67'
The output includes credentials that you must protect. Be sure that you do not include these credentials in your code or check the credentials into your source control. For more information, see https://aka.ms/azadsp-cli
{
  "appId": "ab3cb7b2-c932-4eb7-89ce-a369de998a37",
  "displayName": "azure-cli-2022-12-02-15-40-24",
  "password": "UVq8Q~7VPT9hIVYQ6QCtmCfUyNOTLoaIsze8IdwS",
  "tenant": "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29"
}

Copy the values for appId and password and paste them into the terraform.tfvars file as values for client_id and client_secret. For this example, the value for client_id would be ab3cb7b2-c932-4eb7-89ce-a369de998a37 and client_secret would be UVq8Q~7VPT9hIVYQ6QCtmCfUyNOTLoaIsze8IdwS. Save the file.

location = "LOCATION"
subscription_id = "0e3e2e88-47a3-4107-a2b2-f325314dfb67"
tenant_id = "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29"
client_id = "ab3cb7b2-c932-4eb7-89ce-a369de998a37"
client_secret = "UVq8Q~7VPT9hIVYQ6QCtmCfUyNOTLoaIsze8IdwS"

Update the location variable with your Azure location preference. Save the file.

location = "eastus"
subscription_id = "0e3e2e88-47a3-4107-a2b2-f325314dfb67"
tenant_id = "1e472a2a-7ab3-9bd1-2016-a32fd04dfb29"
client_id = "ab3cb7b2-c932-4eb7-89ce-a369de998a37"
client_secret = "UVq8Q~7VPT9hIVYQ6QCtmCfUyNOTLoaIsze8IdwS"

Deploy the Nomad cluster

Initialize Terraform to download required plugins and set up the workspace.

$ terraform init

Provision the resources. Respond yes to the prompt to confirm the operation. The provisioning takes a couple of minutes.

$ terraform apply

# ...

Apply complete!

Outputs:

IP_Addresses = <<EOT

It will take a little bit for setup to complete and the UI to become available.
Once it is, you can access the Nomad UI at:

http://3.14.8.189:4646/ui

Set the Nomad address, run the bootstrap, export the management token, set the token variable, and test connectivity:

export NOMAD_ADDR=http://3.14.8.189:4646 && \
nomad acl bootstrap | grep -i secret | awk -F "=" '{print $2}' | xargs > nomad-management.token && \
export NOMAD_TOKEN=$(cat nomad-management.token) && \
nomad server members

Copy the token value and use it to log in to the UI:

cat nomad-management.token

Verify cluster availability

Once the provisioning is complete, the Nomad web UI will become available in another couple of minutes.

Open the web UI by visiting the URL in the Terraform output. You may see an error from your browser letting you know that the site cannot be reached or a No cluster leader error message from the Nomad UI after refreshing the page. These are both normal during the setup and mean the cluster is not yet ready. Wait a few minutes and refresh your browser again to see the Nomad UI.

The Nomad UI will show a Not Authorized message once setup is complete. This means the ACL system has not been set up yet. Continue to the next section to set it up.

A screenshot of the Nomad UI displaying the not authorized message on first load

Set up Nomad

Once the cluster is ready and the web UI is available, you can bootstrap the ACL system.

Export the cluster address as the NOMAD_ADDR environment variable.

$ export NOMAD_ADDR=$(terraform output -raw nomad_ip)

Bootstrap the ACLs and save the management token to a file.

$ nomad acl bootstrap | \
    grep -i secret | \
    awk -F "=" '{print $2}' | \
    xargs > nomad-management.token

Export the token as the NOMAD_TOKEN environment variable.

$ export NOMAD_TOKEN=$(cat nomad-management.token)

Your CLI is now set up to interact with the cluster.

Verify connectivity

Verify connectivity to the cluster by checking the status of the nodes. The output you see may be different than the example output depending on where your cluster is running and how many nodes it has.

$ nomad node status
ID        DC   Name             Class   Drain  Eligibility  Status
13416cb7  dc1  user-C05G17CLKD  <none>  false  eligible     ready

$ nomad node status
ID        DC   Name             Class   Drain  Eligibility  Status
83a1527e  dc1  ip-172-31-36-72  <none>  false  eligible     ready
4aeff813  dc1  ip-172-31-32-8   <none>  false  eligible     ready

Browse the web UI

Navigate to the Nomad UI in your web browser by visiting http://localhost:4646/ui(opens in new tab).

Open the Nomad UI from your terminal. This ui command reads the NOMAD_ADDR and NOMAD_TOKEN environment variables, opens the UI in your browser, and logs in.

$ nomad ui -authenticate
Opening URL "http://3.14.8.189:4646/ui" with one-time token

Alternatively, you can navigate to the Nomad UI in your web browser, click the Sign In link in the top right of the page, and log in with the bootstrap token saved in the NOMAD_TOKEN variable or the nomad-management.token file by setting the Secret ID to the token's value.

A screenshot of the Nomad UI displaying the sign in page with the Secret ID field

Visit the Servers page from the left navigation to see the server nodes in the cluster and the Clients page to see the client nodes. These pages will show only one node each if you are running a local development agent.

A screenshot of the Nomad UI displaying the list of servers in the cluster

A screenshot of the Nomad UI displaying the list of clients in the cluster

Visit the Topology page to see an overview of the clients, the resources available on each, and the total resources available to the cluster.

A screenshot of the Nomad UI displaying the cluster topology

Next steps

In this tutorial you created a Nomad cluster. Continue on to the next tutorial by clicking on the Next button below and learn how to submit and run a job.

Install Nomad

Deploy and update a job