Nomad
Service Discovery on Nomad

Deploy an App with Nomad Service Discovery

  • 14min

  • Nomad

Service discovery is a method of automatically detecting services running on a host machine with the intent of making them available to other services connected to the host. Tools that offer these functionalities expose the service's address and port for communication by way of a service name or endpoint. Consul is one of these tools that takes care of routing requests sent to the service name endpoint and forwarding them on to the application running on a specific host and port.

Diagram showing service endpoint and Consul forwarding the request to the application host on a specific port

Nomad offers an easy-to-use integration with Consul that abstracts away a lot of the configuration necessary for service discovery, requiring only a few lines of configuration in the Nomad job spec file.

Consul service definition in a Nomad job file
job "example-job" {
  task "simple-task" {
    service {
      name = "simple-service"
      port = "8080"
    }
  }
}

However, this means that Consul must either be installed separately alongside Nomad on the host or running somewhere else, such as in HCP. Including Consul in your environment adds another layer of complexity to the deployment and, depending on the application, additional complexity that may not be worth the effort.

Nomad v1.3 added built-in native service discovery (SD), making Consul or other third-party tools unnecessary. Nomad's service discovery is not meant to be a replacement for those tools. Rather, it is an alternative for easier testing and deployments with simpler architectures.

Read on to see how you can easily connect your services together that are running in Nomad.

Prerequisites

For this tutorial, you will need:

Note: This tutorial creates AWS resources that may not qualify as part of the AWS free tier. Be sure to follow the Cleanup process at the end of this tutorial so you don't incur any additional unnecessary charges.

Service discovery usage

The Nomad job spec file defines services and service queries but you can also interact with the catalog and find information about services through the Nomad CLI tool. Each of these options are shown in the examples below.

Service definition

The service block enables a Nomad service for discovery by setting the provider attribute to nomad. By default, the value is set to consul implicitly.

Service definition example
job "example-job" {
  task "simple-task" {
    service {
      name = "simple-service"
      port = "8080"
      provider = "nomad"
    }
  }
}

Service query

A template block can specify a function to query a service's address and port. This is a similar procedure for querying services registered in Consul. The following example uses the nomadService function.

Service query example
job "example-job" {
    group "db" {
        network {
            port "db" {
                static = 5432
            }
        }
        task "db" {
            service {
                name = "database"
                provider = "nomad"
                port = "db"
                address  = attr.unique.platform.aws.public-ipv4
            }
        }
    }
    group "product-api" {
        task "product-api" {
        ## ...
            template {
                data        = <<EOH
{{ range nomadService "database" }}
DB_CONNECTION="host={{ .Address }} port={{ .Port }} user=user password=password dbname=db_name"
{{ end }}
EOH
                destination = "local/env.txt"
                env         = true
            }
        }
    }
}

Service commands

Once a service is registered, you can interact with it via the service command of the Nomad CLI.

Listing registered services
$ nomad service list
Service Name  Tags
database      []
frontend      []
nginx         []
payments-api  []
product-api   []
public-api    []
Getting more information about a registered service
$ nomad service info nginx
Job ID     Address                                        Tags  Node ID   Alloc ID
hashicups  ec2-18-233-100-132.compute-1.amazonaws.com:80  []    340e27c3  8e7e0db5

Load balancing

Nomad balances service workloads by selecting an allocation of the referenced service (e.g. database) when the upstream service (e.g. product-api) allocation is created. Nomad selects from available allocations of the referenced service that have not previously been selected from any of the upstream service’s currently active allocations.

Load balancing setup example
job "example-job" {
    group "db" {
        network {
            port "db" {
                static = 5432
            }
        }
        task "db" {
            service {
                name = "database"
                provider = "nomad"
                port = "db"
                address  = attr.unique.platform.aws.public-ipv4
            }
        }
    }
    group "product-api" {
        task "product-api" {
        ## ...
            template {
                data = <<EOH
{{ $allocID := env "NOMAD_ALLOC_ID" }}
{{ $service := nomadService 1 $allocID "database" }}
[product-api]
server = {{ $service.Address }}
ports = [ {{ $service.Port }} ]
                EOH

            destination = "config.toml"
            }
        }
    }
}

HashiCups with Nomad SD

In this tutorial, you will deploy HashiCups, a demo coffee shop application made up of several microservices running on Docker. Each of the services configured in the HashiCups job spec file use the native service discovery feature to retrieve downstream service connection information.

Additionally, you will modify the application after the initial deployment to integrate the load balancing feature of the native service discovery.

You can read more about the HashiCups application in the HashiCups overview.

HashiCups frontend and backend services

Clone the example repository

The example repository contains Terraform configuration to set up a cluster and a Nomad job spec file (hashicups.hcl) containing the entire HashiCups application.

Clone the example repository.

$ git clone https://github.com/hashicorp/learn-nomad-sd.git

Navigate to the cloned repository folder.

$ cd learn-nomad-sd

Check out the v0.1 tag of the repository as a local branch named nomad-hashicups-sd.

$ git checkout tags/v0.1 -b nomad-hashicups-sd
Switched to a new branch 'nomad-hashicups-sd'

Deploy Nomad cluster

Rename the example variables file from terraform.tfvars.example to terraform.tfvars.

$ mv terraform.tfvars.example terraform.tfvars

The module in main.tf requires an Amazon Machine Image (AMI) for provisioning the cluster's servers and clients. Copy the AMI ID that you created as part of the prerequisites and replace the placeholder value for ami in the terraform.tfvars with it.

terraform.tfvars
name                      = "nomad"
key_name                  = "us-east-1-ssh-key"
region                    = "us-east-1"
ami                       = "ami-012345678901234567"
allowlist_ip              = "0.0.0.0/0"
nomad_consul_token_id     = "123e4567-e89b-12d3-a456-426614174000"
nomad_consul_token_secret = "123e4567-e89b-12d3-a456-426614174000"

Replace the region and key_name values with the AWS region that contains the AMI from above and SSH key registered in that region. Replace the allowlist_ip value with a range of IP addresses if you want to limit access to the virtual machines.

terraform.tfvars
name                      = "nomad"
key_name                  = "us-east-1-ssh-key"
region                    = "us-east-1"
ami                       = "ami-012345678901234567"
allowlist_ip              = "0.0.0.0/0"
nomad_consul_token_id     = "123e4567-e89b-12d3-a456-426614174000"
nomad_consul_token_secret = "123e4567-e89b-12d3-a456-426614174000"

Finally, generate UUIDs for the nomad_consul_token_id and nomad_consul_token_secret and replace the placeholder values for them in terraform.tfvars.

terraform.tfvars
name                      = "nomad"
key_name                  = "us-east-1-ssh-key"
region                    = "us-east-1"
ami                       = "ami-012345678901234567"
allowlist_ip              = "0.0.0.0/0"
nomad_consul_token_id     = "123e4567-e89b-12d3-a456-426614174000"
nomad_consul_token_secret = "123e4567-e89b-12d3-a456-426614174000"

Provision the Nomad cluster

Initialize Terraform so that it retrieves any required plugins and sets up the workspace.

$ terraform init
Initializing modules...
Downloading github.com/hashicorp/learn-nomad-cluster-setup for nomad-cluster...
- nomad-cluster in .terraform/modules/nomad-cluster

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Finding latest version of hashicorp/template...
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)
- Installing hashicorp/aws v4.25.0...
- Installed hashicorp/aws v4.25.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

Provision the resources. Respond yes to the prompt to confirm the operation and then press the Enter key to start the process. This will take a few minutes to provision.

$ terraform apply
Apply complete! Resources: 15 added, 0 changed, 0 destroyed.

Outputs:

IP_Addresses = <<EOT

Client public IPs: 3.98.57.19, 3.96.66.175, 35.183.78.6

Server public IPs: 99.79.73.235, 3.99.241.240, 15.223.77.39

The Consul UI can be accessed at http://ar-nomad-server-lb-2016482221.ca-central-1.elb.amazonaws.com:8500/ui
with the bootstrap token: 8f94ee77-bc50-4ba1-bf75-132ed6b9366e

EOT
consul_bootstrap_token_secret = "8f94ee77-bc50-4ba1-bf75-132ed6b9366e"
lb_address_consul_nomad = "http://ar-nomad-server-lb-2016482221.ca-central-1.elb.amazonaws.com"

Set up access to Nomad

Run the post-setup.sh script to complete the setup process.

Note: The Consul UI may take some time to finish setting up and becoming responsive. If the post-setup.sh script fails, wait a minute, make sure the Consul UI is up, and run rm nomad.token && ./post-setup.sh to delete the empty token file and try again.

$ ./post-setup.sh
The Nomad user token has been saved locally to nomad.token and deleted from the Consul KV store.

Set the following environment variables to access your Nomad cluster with the user token created during setup:

export NOMAD_ADDR=$(terraform output -raw lb_address_consul_nomad):4646
export NOMAD_TOKEN=$(cat nomad.token)


The Nomad UI can be accessed at http://ar-nomad-server-lb-2016482221.ca-central-1.elb.amazonaws.com:4646/ui
with the bootstrap token: 0853c07b-5718-8b0a-dad8-e3d6715bbd9f

Copy the export commands from the output, paste them into your terminal, and press the Enter key.

$ export NOMAD_ADDR=$(terraform output -raw lb_address_consul_nomad):4646 && export NOMAD_TOKEN=$(cat nomad.token)

Finally, verify connectivity to the cluster by running a Nomad command.

$ nomad node status
ID        DC   Name              Class   Drain  Eligibility  Status
0c40d12b  dc1  ip-172-31-12-109  <none>  false  eligible     ready
d51c20d9  dc1  ip-172-31-7-192   <none>  false  eligible     ready
bacfe7b3  dc1  ip-172-31-7-84    <none>  false  eligible     ready

Add security groups for HashiCups

The Nomad cluster module sets up the minimum amount of ingress rules necessary for Nomad and Consul to function by opening up access to Consul on port 8500 and Nomad on port 4646 through AWS security groups. Additional rules for the HashiCups ports (80, 3000, 5432, 8080, 8081, and 9090) need to be added for the application to run properly.

Open the secgroups.tf file, add the following configuration to the end of it, and save the file. This attaches the security group defined in the same file to the Nomad clients running in the cluster.

resource "aws_network_interface_sg_attachment" "sg_attachment" {
  for_each = toset(module.nomad-cluster.nomad_client_network_ids)
  security_group_id    = aws_security_group.clients_security_group.id
  network_interface_id = each.value
}

Run Terraform again to update the client machines. Respond yes to the prompt to confirm the operation.

$ terraform apply
Terraform will perform the following actions:

  # aws_network_interface_sg_attachment.sg_attachment["eni-04fa961abdea856cc"] will be created
  + resource "aws_network_interface_sg_attachment" "sg_attachment" {
      + id                   = (known after apply)
      + network_interface_id = "eni-04fa961abdea856cc"
      + security_group_id    = "sg-00dfa5b3b913d7418"
    }

  # aws_network_interface_sg_attachment.sg_attachment["eni-08c88cf16277e295d"] will be created
  + resource "aws_network_interface_sg_attachment" "sg_attachment" {
      + id                   = (known after apply)
      + network_interface_id = "eni-08c88cf16277e295d"
      + security_group_id    = "sg-00dfa5b3b913d7418"
    }

  # aws_network_interface_sg_attachment.sg_attachment["eni-0abedc09c1e159842"] will be created
  + resource "aws_network_interface_sg_attachment" "sg_attachment" {
      + id                   = (known after apply)
      + network_interface_id = "eni-0abedc09c1e159842"
      + security_group_id    = "sg-00dfa5b3b913d7418"
    }

Plan: 3 to add, 0 to change, 0 to destroy.

aws_network_interface_sg_attachment.sg_attachment["eni-08c88cf16277e295d"]: Creating...
aws_network_interface_sg_attachment.sg_attachment["eni-0abedc09c1e159842"]: Creating...
aws_network_interface_sg_attachment.sg_attachment["eni-04fa961abdea856cc"]: Creating...
aws_network_interface_sg_attachment.sg_attachment["eni-0abedc09c1e159842"]: Creation complete after 0s [id=sg-00dfa5b3b913d7418_eni-0abedc09c1e159842]
aws_network_interface_sg_attachment.sg_attachment["eni-04fa961abdea856cc"]: Creation complete after 0s [id=sg-00dfa5b3b913d7418_eni-04fa961abdea856cc]
aws_network_interface_sg_attachment.sg_attachment["eni-08c88cf16277e295d"]: Creation complete after 0s [id=sg-00dfa5b3b913d7418_eni-08c88cf16277e295d]

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Deploy HashiCups

The hashicups.hcl job spec file configures the HashiCups application.

Submit the job file to Nomad. Some services may report as unhealthy at first but will become healthy soon after.

Note: If you are running a Nomad cluster that is not on AWS, you will need to use a different attribute for each of the services' addresses because the attr.unique.platform.aws.public-ipv4 attribute is specific to AWS. For local dev clusters, you can use attr.unique.network.ip-address. Check the Client Detail page in the Nomad UI to see a full list of the available attributes.

$ nomad job run hashicups.hcl
==> 2022-08-10T14:57:06-04:00: Monitoring evaluation "972d3a90"
    2022-08-10T14:57:06-04:00: Evaluation triggered by job "hashicups"
    2022-08-10T14:57:06-04:00: Allocation "2b71c82b" created: node "bacfe7b3", group "product-api"
    2022-08-10T14:57:06-04:00: Allocation "39b4797d" created: node "bacfe7b3", group "db"
    2022-08-10T14:57:06-04:00: Allocation "8c7efb80" created: node "d51c20d9", group "nginx"
    2022-08-10T14:57:06-04:00: Allocation "9c6ce57a" created: node "d51c20d9", group "public-api"
    2022-08-10T14:57:06-04:00: Allocation "ddc0691d" created: node "d51c20d9", group "frontend"
    2022-08-10T14:57:06-04:00: Allocation "e75d82a7" created: node "bacfe7b3", group "payments-api"
    2022-08-10T14:57:07-04:00: Evaluation within deployment: "852e99c4"
    2022-08-10T14:57:07-04:00: Evaluation status changed: "pending" -> "complete"
==> 2022-08-10T14:57:07-04:00: Evaluation "972d3a90" finished with status "complete"
==> 2022-08-10T14:57:07-04:00: Monitoring deployment "852e99c4"
  ✓ Deployment "852e99c4" successful
    
    2022-08-10T14:58:42-04:00
    ID          = 852e99c4
    Job ID      = hashicups
    Job Version = 0
    Status      = successful
    Description = Deployment completed successfully
    
    Deployed
    Task Group    Desired  Placed  Healthy  Unhealthy  Progress Deadline
    db            1        1       1        0          2022-08-10T19:07:41Z
    frontend      1        1       1        0          2022-08-10T19:07:38Z
    nginx         1        2       1        1          2022-08-10T19:08:40Z
    payments-api  1        1       1        0          2022-08-10T19:07:41Z
    product-api   1        2       1        1          2022-08-10T19:08:14Z
    public-api    1        1       1        0          2022-08-10T19:07:28Z

When the deployment is complete, list the HashiCups services.

$ nomad service list
Service Name  Tags
database      []
frontend      []
nginx         []
payments-api  []
product-api   []
public-api    []

Inspect the nginx service, copy the public address listed, and open it in your browser to see the HashiCups application.

$ nomad service info nginx
Job ID     Address                                                Tags  Node ID   Alloc ID
hashicups  ec2-3-96-66-175.ca-central-1.compute.amazonaws.com:80  []    bacfe7b3  382cdc86

Enable service load balancing

To see the service load balancing available in Nomad, you will modify the HashiCups job spec to create an additional instance of the frontend and nginx services. Each of the nginx services will communicate with a different instance of frontend.

Modify the HashiCups job

Open hashicups.hcl. Add the count attribute to the frontend service group.

hashicups.hcl
group "frontend" {
  count = 2
  network {
    port "frontend" {
      static = var.frontend_port
    }
  }
  # ...
}

Then, add the same attribute to the nginx group.

hashicups.hcl
group "nginx" {
  count = 2
  network {
    port "nginx" {
      static = var.nginx_port
    }
  }
  # ...
}

Finally, in the template block of the nginx group, add a line for the $allocID variable and modify the nomadService to match the highlighted example below. Save the file.

hashicups.hcl
group "nginx" {
  # ...
  task "nginx" {
    # ...
    template {
        data =  <<EOF
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=STATIC:10m inactive=7d use_temp_path=off;
upstream frontend_upstream {
  {{ $allocID := env "NOMAD_ALLOC_ID" }}
  {{ range nomadService 1 $allocID "frontend" }}
    server {{ .Address }}:{{ .Port }};
  {{ end }}
}
    }
  }
  # ...
}

Submit the job file to Nomad.

$ nomad job run hashicups.hcl
==> 2022-08-10T15:01:24-04:00: Monitoring evaluation "601e50cd"
    2022-08-10T15:01:24-04:00: Evaluation triggered by job "hashicups"
    2022-08-10T15:01:24-04:00: Allocation "e75d82a7" modified: node "bacfe7b3", group "payments-api"
    2022-08-10T15:01:24-04:00: Allocation "2eef23d1" created: node "0c40d12b", group "nginx"
    2022-08-10T15:01:24-04:00: Allocation "db819fd3" created: node "0c40d12b", group "frontend"
    2022-08-10T15:01:24-04:00: Allocation "a5bbf109" modified: node "d51c20d9", group "product-api"
    2022-08-10T15:01:24-04:00: Allocation "39b4797d" modified: node "bacfe7b3", group "db"
    2022-08-10T15:01:24-04:00: Allocation "9c6ce57a" modified: node "d51c20d9", group "public-api"
    2022-08-10T15:01:24-04:00: Allocation "ddc0691d" modified: node "d51c20d9", group "frontend"
    2022-08-10T15:01:25-04:00: Evaluation within deployment: "ea8bfb0f"
    2022-08-10T15:01:25-04:00: Evaluation status changed: "pending" -> "complete"
==> 2022-08-10T15:01:25-04:00: Evaluation "601e50cd" finished with status "complete"
==> 2022-08-10T15:01:25-04:00: Monitoring deployment "ea8bfb0f"
  ✓ Deployment "ea8bfb0f" successful
    
    2022-08-10T15:01:54-04:00
    ID          = ea8bfb0f
    Job ID      = hashicups
    Job Version = 1
    Status      = successful
    Description = Deployment completed successfully
    
    Deployed
    Task Group    Desired  Placed  Healthy  Unhealthy  Progress Deadline
    db            1        1       1        0          2022-08-10T19:11:34Z
    frontend      2        2       2        0          2022-08-10T19:11:48Z
    nginx         2        2       2        0          2022-08-10T19:11:53Z
    payments-api  1        1       1        0          2022-08-10T19:11:34Z
    product-api   1        1       1        0          2022-08-10T19:11:34Z
    public-api    1        1       1        0          2022-08-10T19:11:34Z

When the deployment is complete, inspect the nginx service. Note that there are two instances. Copy each address and open them in a different browser tab.

$ nomad service info nginx
Job ID     Address                                                Tags  Node ID   Alloc ID
hashicups  ec2-35-183-78-6.ca-central-1.compute.amazonaws.com:80  []    0c40d12b  2eef23d1
hashicups  ec2-3-96-66-175.ca-central-1.compute.amazonaws.com:80  []    bacfe7b3  61934d47

Scroll down to the bottom of the page. The text in the footer shows which instance of the frontend service the Nginx instance is using. Note that they are different as Nomad attaches each one to a specific nginx instance during the job creation process.

Note: If you notice that both pages are displaying the same frontend instance name, your browser may be caching the result. In this case, try a hard refresh of the page or open it in a private browser window. If that still doesn't work, you can try stopping and deleting the job with nomad job stop -purge hashicups and resubmitting with nomad job run hashicups.hcl.

Screenshot of the HashiCups application highlighting the footer text that shows frontend instance zero

Next Steps

In this tutorial, you learned about the native service discovery present in Nomad and deployed an application that uses it.

Continue onto the next tutorial in the collection to learn how to convert from Nomad to Consul service discovery.

Back to Collection
Next