Devices

Nomad has built-in support for scheduling compute resources such as CPU, memory, and networking. Nomad device plugins are used to support scheduling tasks with other devices, such as GPUs. They are responsible for fingerprinting these devices and working with the Nomad client to make them available to assigned tasks.

For a real world example of a Nomad device plugin implementation, see the Nvidia GPU plugin.

Authoring Device Plugins

Authoring a device plugin in Nomad consists of implementing the DevicePlugin interface alongside a main package to launch the plugin.

The device plugin skeleton project exists to help bootstrap the development of new device plugins. It provides most of the boilerplate necessary for a device plugin, along with detailed comments.

Lifecycle and State

A device plugin is long-lived. Nomad will ensure that one instance of the plugin is running. If the plugin crashes or otherwise terminates, Nomad will launch another instance of it.

However, unlike task drivers, device plugins do not currently have an interface for persisting state to the Nomad client. Instead, the device plugin API emphasizes fingerprinting devices and reporting their status. After helping to provision a task with a scheduled device, a device plugin does not have any responsibility (or ability) to monitor the task.

Device Plugin API

The base plugin must be implemented in addition to the following functions.

Fingerprint(context.Context) (<-chan *FingerprintResponse, error)

The Fingerprint function is called by the client when the plugin is started. It allows the plugin to provide Nomad with a list of discovered devices, along with their attributes, for the purpose of scheduling workloads using devices. The channel returned should immediately send an initial FingerprintResponse, then send periodic updates at an appropriate interval until the context is canceled.

Each fingerprint response consists of either an error or a list of device groups. A device group is a list of detected devices that are identical for the purpose of scheduling; that is, they will have identical attributes.

Stats(context.Context, time.Duration) (<-chan *StatsResponse, error)

The Stats function returns a channel on which the plugin should emit device statistics, at the specified interval, until either an error is encountered or the specified context is cancelled. The StatsReponse object allows dimensioned statistics to be returned for each device in a device group.

Reserve(deviceIDs []string) (*ContainerReservation, error)

The Reserve function accepts a list of device IDs and returns the information necessary for the client to make those devices available to a task. Currently, the ContainerReservation object allows the plugin to specify environment variables for the task, as well as a list of host devices and files to be mounted into the task's filesystem. Any orchestration required to prepare the device for use should also be performed in this function.