Nomad APM Plugin

The Nomad APM plugin allows querying the Nomad API for metric data. This provides an immediate starting point without addition applications but comes at the price of efficiency and flexibility. When using this APM, it is advised to monitor Nomad carefully ensuring it is not put under excessive load pressure.

Since Nomad does not store historical metric data and only reports instant metrics values, the Nomad APM plugin does not support the query_window policy option and is not able to aggregate metrics over time, such as computing averages or percentiles. An external APM is required to support uses cases like these.

Agent Configuration Options

apm "nomad-apm" {
  driver = "nomad-apm"
}

When using a Nomad cluster with ACLs enabled, following ACL policy will provide the appropriate permissions for obtaining task group metrics:

namespace "default" {
  policy       = "read"
  capabilities = ["read-job"]
}

In order to obtain cluster level metrics, the following ACL policy will be required:

node {
  policy = "read"
}

namespace "default" {
  policy       = "read"
  capabilities = ["read-job"]
}

Policy Configuration Options - Task Groups

The Nomad APM allows querying Nomad to understand the current resource usage of a task group.

check {
  source = "nomad-apm"
  query  = "avg_cpu"
  # ...
}

Querying Nomad task group metrics is be done using the <operation>_<metric> syntax, where valid operations are:

• avg - returns the average of the metric value across allocations in the task group.

• min - returns the lowest metric value among the allocations in the task group.

• max - returns the highest metric value among the allocations in the task group.

• sum - returns the sum of all the metric values for the allocations in the task group.

The metric value can be:

• cpu - CPU usage as reported by the nomad.client.allocs.cpu.total_percent metric.

• cpu-allocated - the percentage of CPU used out of the total CPU allocated for the allocation.

• memory - Memory usage as reported by the nomad.client.allocs.memory.usage metric.

• memory-allocated - the percentage of memory used out of the total memory allocated for the allocation.

Policy Configuration Options - Client Nodes

The Nomad APM allows querying Nomad to understand the current allocated resource as a percentage of the total available.

The plugin identifies the pool of Nomad clients to query from the pool identifiers in the policy target block. Set at least one of the following options in the target block.

• node_class matches clients by their node class. To match clients that do not have a node class, set node_class to autoscaler-default-pool.

• datacenter matches clients by their datacenter.

• node_pool matches clients by their node pool.

When you set more than one identifier, the pool includes only the clients that match every identifier. For example, the following policy queries the clients in the dc1 datacenter that also have the autoscale node class.

policy {
  # ...
  check {
    source = "nomad-apm"
    query  = "percentage-allocated_cpu"
    # ...
  }

  target "..." {
    # ...
    node_class = "autoscale"
    datacenter = "dc1"
    # ..
  }
}

Querying Nomad client node metrics is be done using the <operation>_<metric> syntax, where valid operations are:

• percentage-allocated - returns the allocated percentage of the desired resource.

The metric value can be:

• cpu - allocated CPU as reported by calculating total allocatable against the total allocated by the scheduler.

• memory - allocated memory as reported by calculating total allocatable against the total allocated by the scheduler.