Save 10-15% Register for HashiConf and save big when you buy 2+ tickets Get your passes
Nomad
Nomad Home

parameterized block in the job specification

Placementjob -> parameterized

A parameterized job is used to encapsulate a set of work that can be carried out on various inputs much like a function definition. When the parameterized block is added to a job, the job acts as a function to the cluster as a whole.

The parameterized block allows job operators to configure a job that carries out a particular action, define its resource requirements and configure how inputs and configuration are retrieved by the tasks within the job.

To invoke a parameterized job, nomad job dispatch or the equivalent HTTP APIs are used. When dispatching against a parameterized job, an opaque payload and metadata may be injected into the job. These inputs to the parameterized job act like arguments to a function. The job consumes them to change its behavior, without exposing the implementation details to the caller.

To that end, tasks within the job can add a dispatch_payload block that defines where on the filesystem this payload gets written to. An example payload would be a task's JSON configuration.

Further, certain metadata may be marked as required when dispatching a job so it can be used to inject configuration directly into a task's arguments using interpolation. An example of this would be to require a run ID key that could be used to lookup the work the job is suppose to do from a management service or database.

Each time a job is dispatched, a unique job ID is generated. This allows a caller to track the status of the job, much like a future or promise in some programming languages. The dispatched job cannot be updated after dispatching; to update the job definition you need to update the parent job.

job "docs" {
  parameterized {
    payload       = "required"
    meta_required = ["dispatcher_email"]
    meta_optional = ["pager_email"]
  }
}

See the multiregion documentation for additional considerations when dispatching parameterized jobs.

Requirements

Parameters

  • meta_optional (array<string>: nil) - Specifies the set of metadata keys that may be provided when dispatching against the job.

  • meta_required (array<string>: nil) - Specifies the set of metadata keys that must be provided when dispatching against the job.

  • payload (string: "optional") - Specifies the requirement of providing a payload when dispatching against the parameterized job. The maximum size of a payload is 16 KiB. The options for this field are:

    • "optional" - A payload is optional when dispatching against the job.

    • "required" - A payload must be provided when dispatching against the job.

    • "forbidden" - A payload is forbidden when dispatching against the job.

Examples

The following examples show non-runnable example parameterized jobs:

Required inputs

This example shows a parameterized job that requires both a payload and metadata:

job "video-encode" {
  # ...

  type = "batch"

  parameterized {
    payload       = "required"
    meta_required = ["dispatcher_email"]
  }

  group "encode" {
    # ...

    task "ffmpeg" {
      driver = "exec"

      config {
        command = "ffmpeg-wrapper"

        # When dispatched, the payload is written to a file that is then read by
        # the created task upon startup
        args = ["-config=${NOMAD_TASK_DIR}/config.json"]
      }

      dispatch_payload {
        file = "config.json"
      }
    }
  }
}

Metadata interpolation

job "email-blast" {
  # ...

  type = "batch"

  parameterized {
    payload       = "forbidden"
    meta_required = ["CAMPAIGN_ID"]
  }

  group "emails" {
    # ...

    task "emailer" {
      driver = "exec"

      config {
        command = "emailer"

        # The campaign ID is interpolated and injected into the task's
        # arguments
        args = ["-campaign=${NOMAD_META_CAMPAIGN_ID}"]
      }
    }
  }
}

Use periodic with parameterized

Nomad uses an internal hierarchy when scheduling a job that is both parameterized and periodic.

parameterized => periodic => batch

Nomad processes a periodic with parameterized job in the following order:

  1. Nomad does not dispatch a periodic job with null parameters in the periodic configuration. This forces the parameterized job to take precedence over the periodic job.
  2. After Nomad dispatches the parameterized job and gives it parameters, Nomad uses the periodic configuration.
  3. Nomad dispatches new jobs according to the periodic configuration that uses thee parameters from the triggering parameterized job.

In this example, the periodic job does not trigger any new jobs until the operator dispatches the parameterized job at least once. After that, the dispatched child periodically triggers more children with the given parameters.

  periodic {
    crons = [
      "*/40 * * * * * *"
    ]
  }
  parameterized {
    payload       = "required"
    meta_required = ["dispatcher_email"]
    meta_optional = ["pager_email"]
  }

There are three columns plus comments in this example output, which is for the preceding periodic, parameterized example job. Scroll to the last column to review the comments.

$ nomad job status
ID                                                     Type                            Submit Date
sync                                                   batch/periodic/parameterized    2024-11-07T10:43:30+01:00 // Original submitted job
sync/dispatch-1730972650-247c6e97                      batch/periodic                  2024-11-07T10:44:10+01:00 // First dispatched job with parameters A
sync/dispatch-1730972650-247c6e97/periodic-1730972680  batch                           2024-11-07T10:44:40+01:00 // Cron job with parameters A
sync/dispatch-1730972650-247c6e97/periodic-1730972860  batch                           2024-11-07T10:47:40+01:00 // Cron job with parameters A
sync/dispatch-1730972760-f79a96e1                      batch/periodic                  2024-11-07T10:46:00+01:00 // Second dispatched job with parameters B
sync/dispatch-1730972760-f79a96e1/periodic-1730972800  batch                           2024-11-07T10:46:40+01:00 // Cron job with parameters B
sync/dispatch-1730972760-f79a96e1/periodic-1730972860  batch                           2024-11-07T10:47:40+01:00 // Cron job with parameters B

The output illustrates the following:

  1. The sync job is the parameterized and periodic job. However, Nomad doesn't run the periodic job until you submit with the required parameters.
  2. The sync/dispatch-1730972650-247c6e97 job contains parameters and triggers the batch jobs, named sync/dispatch-1730972650-247c6e97/periodic-1730972680 and sync/dispatch-1730972650-247c6e97/periodic-1730972860.
  3. The sync/dispatch-1730972760-f79a96e1 job assigns new parameters and triggers new batch jobs that use those parameters.

If you need to force the periodic job, force the the corresponding parameterized job.

This example forces the first dispatched job with parameters A from the preceding example.

$ nomad job periodic force sync/dispatch-1730972650-247c6e97
Edit this page on GitHub