resources block in the job specification

The resources block describes the requirements a task needs to execute. Resource requirements include memory, CPU, and more.

job "docs" {
  group "example" {
    task "server" {
      resources {
        cpu    = 100
        memory = 256

        device "nvidia/gpu" {
          count = 2
        }
      }
    }
  }
}

Parameters

• cpu (int: 100) - Specifies the CPU required to run this task in MHz.

• cores (int: <optional>) - Specifies the number of CPU cores to reserve specifically for the task. This may not be used with cpu. The behavior of setting cores is specific to each task driver (e.g. docker, exec).

• memory (int: 300) - Specifies the memory required in MB.

• memory_max (int: <optional>) - Optionally, specifies the maximum memory the task may use, if the client has excess memory capacity, in MB. See Memory Oversubscription for more details.

• numa (Numa: <optional>) - Specifies the NUMA scheduling preference for the task. Requires the use of cores.

• device (Device: <optional>) - Specifies the device requirements. This may be repeated to request multiple device types.

• secrets (int: <optional>) - Specifies the size of the secrets/ directory in MB, on platforms where the directory is a tmpfs. If set, the scheduler adds the secrets value to the memory value when allocating resources on a client, and this value will be included in the allocated resources shown by the nomad alloc status and nomad node status commands. If unset, the client will allocate 1 MB of tmpfs space and it will not be counted for scheduling purposes or included in allocated resources. You should not set this value if the workload will be placed on a platform where tmpfs is unsupported, because it will still be counted for scheduling purposes.

Examples

The following examples only show the resources blocks. Remember that the resources block is only valid in the placements listed above.

Cores

This example specifies that the task requires 2 reserved cores. With this block, Nomad finds a client with enough spare capacity to reserve 2 cores exclusively for the task. Unlike the cpu field, the task does not share CPU time with any other tasks managed by Nomad on the client.

resources {
  cores = 2
}

If cores and cpu are both defined in the same resource block, validation of the job fails.

Refer to How Nomad Uses CPU for more details on Nomad's reservation of CPU resources.

Memory

This example specifies the task requires 2 GB of RAM to operate. 2 GB is the equivalent of 2048 MB:

resources {
  memory = 2048
}

Devices

This example shows a device constraints as specified in the device block which require two nvidia GPUs to be made available:

resources {
  device "nvidia/gpu" {
    count = 2
  }
}

Memory oversubscription

Setting task memory limits requires balancing the risk of interrupting tasks against the risk of wasting resources. If a task memory limit is set too low, the task may exceed the limit and be interrupted; if the task memory is too high, the cluster is left underutilized.

To help maximize cluster memory utilization while allowing a safety margin for unexpected load spikes, Nomad allows job authors to set two separate memory limits:

• memory: the reserve limit to represent the task’s typical memory usage — this number is used by the Nomad scheduler to reserve and place the task

• memory_max: the maximum memory the task may use, if the client has excess available memory, and may be terminated if it exceeds

If a client's memory becomes contended or low, the operating system will pressure the running tasks to free up memory. If the contention persists, Nomad may kill oversubscribed tasks and reschedule them to other clients. The exact mechanism for memory pressure is specific to the task driver, operating system, and application runtime.

The memory_max limit attribute is currently supported by the official raw_exec, exec2, exec, docker, podman, and java task drivers. Consult the documentation of community-supported task drivers for their memory oversubscription support.

Memory oversubscription is opt-in. Nomad operators can enable Memory Oversubscription in the scheduler configuration. Enterprise customers can use Resource Quotas to limit the memory oversubscription and enable or disable memory oversubscription per node pool.

To avoid degrading the cluster experience, we recommend examining and monitoring resource utilization and considering the following suggestions:

• Set oom_score_adj for Linux host services that aren't managed by Nomad, e.g. Docker, logging services, and the Nomad agent itself. For Systemd services, you can use the OOMScoreAdj field.

• Monitor hosts for memory utilization and set alerts on Out-Of-Memory errors

• Set the client reserved with enough memory for host services that aren't managed by Nomad as well as a buffer for the memory excess. For example, if the client reserved memory is 1GB, the allocations on the host may exceed their soft memory limit by almost 1GB in aggregate before the memory becomes contended and allocations get killed.