Queue batch jobs

Configure a batch job queue on a node pool so that Nomad schedules the pool's batch jobs one at a time, in the order that you choose. Enable a first in, first out (FIFO) queue or a dynamic priority queue, submit jobs to the queue, and monitor the queue.

Experimental

Experimental features are tested but unproven. Until the feature is verified through heavy production use, proceed with caution.

Background

A batch job queue holds the batch jobs that you register in a node pool and sends them to the scheduler one at a time. A FIFO queue runs jobs in the order that you register them. A dynamic priority queue adjusts each job's priority so that tenants whose jobs hold a smaller share of the allocated resources run first, which gives each tenant a fair share of the node pool's resources. Refer to Batch job queue for details about how the queue orders jobs and the tradeoffs of serialized scheduling.

Requirements

Nomad clients that belong to a node pool that you create. You cannot modify the built-in default and all node pools, so you cannot configure a queue on them. Set the node_pool parameter in the client configuration to assign clients to a node pool. Refer to Create and use node pools for more information.
When ACLs are enabled, the following permissions:
- A token with the write capability in a node_pool policy for the node pool to create or update the node pool.
- A token with the list-jobs namespace capability to run nomad queue jobs.
- A token with operator:read permissions to run nomad queue status.

Enable a FIFO queue

A FIFO queue sends batch jobs to the scheduler in the order that you register them. The fifo block has no parameters.

This example shows how to create batch queue for a node pool named batch.

Create a node pool specification file named batch.nomad.hcl with a batch_queue block that contains an empty fifo block.
batch.nomad.hcl
```
node_pool "batch" {
  batch_queue {
    fifo {}
  }
}
```

Apply the node pool specification with the nomad node pool apply command.

$ nomad node pool apply batch.nomad.hcl
Successfully applied node pool "batch"!

Verify that the queue is active with the nomad queue jobs command. An active FIFO queue with no jobs returns an empty table.
```
$ nomad queue jobs -node-pool batch
Batch Queue Workloads
JobID  Status  Position  CreatedAt
```
When the command returns No batch job queue configured for node pool: batch, the node pool does not have an active queue. Check the batch_queue block in your node pool specification.

Enable a dynamic priority queue

A dynamic priority queue calculates an adjusted priority for each job from its tenant's share of allocated resources, the time the job waited, and the resources the job requests.

This example creates a dynamic priority batch queue for a node pool named batch. Each namespace is a tenant.

Create a node pool specification file named batch.nomad.hcl with a batch_queue block that contains a dynamic_priority block.

batch.nomad.hcl

node_pool "batch" {
  description = "Clients that run batch workloads"

  batch_queue {
    dynamic_priority {
      calc_interval = "10s"

      tenant_fairshare {
        tenant_type   = "namespace"
        cpu_weight    = 25
        memory_weight = 25
      }

      age {
        weight = 10
        max    = "1h"
      }

      job_size {
        cpu_weight    = 10
        cpu_max       = 1000
        memory_weight = 10
        memory_max    = 1024
      }
    }
  }
}

When you set a weight in any of the blocks, set its maximum too. For example, set max when you set weight in the age block.

To group jobs by a job metadata value instead of by namespace, set tenant_type to metadata and name the metadata key in metadata_key. With metadata tenants, jobs that do not set the metadata key skip the queue, and Nomad schedules them as usual.

tenant_fairshare {
  tenant_type   = "metadata"
  metadata_key  = "customer"
  cpu_weight    = 25
  memory_weight = 25
}

Refer to the following node pool specification sections for parameter descriptions:

Apply the node pool specification.

$ nomad node pool apply batch.nomad.hcl
Successfully applied node pool "batch"!

Verify that the queue is active. An active dynamic priority queue with no jobs returns an empty table.

$ nomad queue jobs -node-pool batch
Batch Queue Workloads
JobID  Tenant  Status  Adjusted Priority  Base Priority  Position  Fairshare  Age  Cpu  Memory  CreatedAt

Submit batch jobs to the queue

The queue holds batch jobs that run in its node pool. Set the job's node_pool parameter to the node pool with the queue.

job "team-a-job1" {
  namespace = "team-a"
  type      = "batch"
  node_pool = "batch"

  group "work" {
    task "sleep" {
      driver = "raw_exec"

      config {
        command = "/bin/sleep"
        args    = ["20"]
      }

      resources {
        cpu    = 600
        memory = 256
      }
    }
  }
}

Register the job with the nomad job run command. While a job waits in the queue, nomad job status reports its status as pending.

$ nomad job status -namespace team-a
ID           Type   Priority  Status   Submit Date
team-a-job1  batch  50        running  2026-10-13T16:37:24-05:00
team-a-job2  batch  50        pending  2026-10-13T16:37:27-05:00
team-a-job3  batch  50        pending  2026-10-13T16:37:27-05:00

Monitor the queue

Use the nomad queue jobs command to list the jobs in the queue. Add the -p flag to list jobs in queue order. Without -p, the command sorts jobs by job ID.

The job at position 0 is the job that the queue sent to the scheduler. The Fairshare, Age, Cpu, and Memory columns show the adjustments that make up each job's adjusted priority. In this example, the team-a jobs hold all of the allocated resources and the team-b jobs hold none, so the team-b jobs move ahead of team-a-job3.

$ nomad queue jobs -node-pool batch -p
Batch Queue Workloads
JobID        Tenant  Status   Adjusted Priority  Base Priority  Position  Fairshare  Age  Cpu  Memory  CreatedAt
team-a-job2  team-a  placing  61                 50             0         0          0    4    7       2026-10-13T16:37:27-05:00
team-b-job1  team-b  queued   111                50             1         50         0    4    7       2026-10-13T16:37:28-05:00
team-b-job2  team-b  queued   111                50             2         50         0    4    7       2026-10-13T16:37:29-05:00
team-b-job3  team-b  queued   111                50             3         50         0    4    7       2026-10-13T16:37:30-05:00
team-a-job3  team-a  queued   61                 50             4         0          0    4    7       2026-10-13T16:37:27-05:00

For a dynamic priority queue, use the nomad queue status command to view each tenant's share of the allocated resources.

$ nomad queue status -node-pool batch
Batch Queue Status
Tenant  Resource  Current / Total  Percentage
team-a                             100
        cpu       600.00 / 600.00
        memory    256.00 / 256.00
team-b                             0
        cpu       0.00 / 600.00
        memory    0.00 / 256.00

To track queues over time, collect the nomad.queue.jobs and nomad.queue.watcher.since_job_submitted server metrics. Both metrics have a node_pool label.

Remove a job from the queue

Stop a queued job with the nomad job stop command. Nomad removes the job from the queue and cancels its registration evaluation.

$ nomad job stop -namespace team-a team-a-job3

A job that waits for resources stays at the front of the queue and holds the jobs behind it. Stop the job to let the queue continue.

To keep a job that Nomad cannot place because of its constraints from holding the queue, set constraint_timeout in the batch_queue block. After the timeout, the queue sends the next job to the scheduler. Without a timeout, the job holds the queue until Nomad places it or you stop it.

batch_queue {
  constraint_timeout = "5m"
  fifo {}
}

Change or remove the queue

To change the queue type or its parameters, update the batch_queue block in the node pool specification and apply it again with nomad node pool apply. Nomad replaces the node pool's queue and restores the queued jobs into the new queue.

When the new queue uses metadata tenants, queued jobs that do not set the metadata key leave the queue, and Nomad sends their evaluations directly to the scheduler.

To remove the queue, delete the batch_queue block from the node pool specification and apply it again. Nomad then sends the evaluations for the queued jobs and for new batch jobs in the node pool directly to the scheduler.