Acquisition complete HashiCorp officially joins the IBM family. Learn more
Nomad
Nomad Home

Nomad 1.10.x release notes

We are pleased to announce the following Nomad updates.

Release highlights

Dynamic host volumes

The dynamic host volumes feature brings a persistent storage option to your workload allocations.

Nomad dynamic host volumes manage storage for stateful workloads without requiring a restart of the Nomad nodes to apply configuration changes. You create dynamic host volumes via the CLI or API and then configure the job with the volume and volume_mount blocks in the job specification.

Host volumes mount paths from the Nomad client into allocations. Nomad is aware of host volume availability and makes use of it for job scheduling. However, Nomad does not know about the volume's underlying characteristics, so you can use host volumes for both local persistent storage and for highly available networked storage.

Dynamic host volume governance Enterprise

Providing guardrails to platform consumers is an important aspect of the storage provisioning workflow when leveraging host volumes across a shared Nomad cluster. Nomad Enterprise supports these new capabilities to provide governance when provisioning host volumes:

  • Sentinel dynamic host volume objects

    During volume creation, Nomad can evaluate all of the details within the dynamic host volume specification against Sentinel policies that define and enforce specific patterns.

    For example, a policy that enforces the storage tier based on the environment or namespace specified would allow reserving more expensive NVME storage for specific workloads. Being able to apply policy to the volume specification gives you a method to enforce specific patterns while providing platform consumers with more flexibility around self-service volume provisioning. Refer to the Sentinel policy reference for more information.

  • Resource quota support

    Nomad’s resource quota system now includes coverage for host volume capacity limits that you can apply to provisioned storage within a specific namespace. This helps control storage consumption within a namespace based on the maximum capacities defined during creation or when making updates to the maximum capacities over the lifecycle of the volume. Refer to the Resource quota specification for more information.

  • Namespace and node pool validation:

    Dynamic host volumes live within the context of a specific namespace when created. When Nomad provisions volumes in a namespace targeting a specific node pool, Nomad evaluates the namespace node pool configuration to ensure that volume creation aligns with job placement rules for node pools. Refer to the Namespace specification for details on node_pool_config parameters.

Resources

Refer to the following resources to learn more about dynamic host volumes:

OpenID Connect (OIDC) enhancements

Nomad 1.10 extends Nomad's OIDC SSO login feature with Private Key JWT and Proof Key for Code Exchange (PKCE).

Private Key JWT

Private Key JWT, also called client assertions, is a more secure alternative for client secrets. Instead of sending a simple secret, Nomad builds a JWT and signs it with a value that the OIDC provider verifies. In this approach, Nomad asserts a valid OIDC client without sending any secret information over the network.

Proof Key for Code Exchange (PKCE)

PKCE adds an extra layer of security to any OIDC auth method for both client secrets and client assertions.

Set the ACL auth method OIDCEnablePKCE parameter to true to turn on this extra security.

Note that not all OIDC providers support PKCE. In addition to enabling PKCE in Nomad, you may need to enable it in your OIDC provider's configuration.

Resources

Container Storage Interface (CSI) enhancements

We added the following:

  • CSI volume and plugin events to the event stream

  • Volume capabilities to the nomad volume status command output

  • The ability to use a volume ID prefix search and wildcard namespace with the nomad volume delete command. Refer to the GitHub pull request for details. Example usage:

    $ nomad volume create ./internal-plugin.volume.hcl
==> Created host volume internal-plugin with ID aeea91a0-06df-c16e-5403-ff82a2f28fd4
✓ Host volume "aeea91a0" ready

  2025-01-31T15:55:14-05:00
  ID        = aeea91a0-06df-c16e-5403-ff82a2f28fd4
  Name      = internal-plugin
  Namespace = default
  Plugin ID = mkdir
  Node ID   = b4611abd-d4a8-c83a-b05e-7d9f5b44a179
  Node Pool = default
  Capacity  = 0 B
  State     = ready
  Host Path = /run/nomad/dev/data/host_volumes/aeea91a0-06df-c16e-5403-ff82a2f28fd4

$ nomad volume delete -type host aeea91a0
Successfully deleted volume "aeea91a0-06df-c16e-5403-ff82a2f28fd4"!

UI URL hints added to CLI commands

We added UI URL hints to the end of common CLI commands and a -ui flag to automatically open the generated link in your browser.

Showing UI URL hints is enabled by default. You have two options for turning off this feature:

  • Server: Modify the show_cli_hints parameter in your agent's ui block configuration.

  • CLI: Set the NOMAD_CLI_SHOW_HINTS environment variable to 0 or false.

    $ nomad status
No running jobs

==> View and manage Nomad jobs in the Web UI: https://localhost:4646/ui/jobs

$ export NOMAD_CLI_SHOW_HINTS=0
$ nomad status
No running jobs

Breaking changes

Go SDK API change for quota limits

In Nomad 1.10.0, the Go API for quotas has a breaking change. The QuotaSpec.RegionLimit field is now of type QuotaResources instead of Resources. The QuotaSpec.VariablesLimit field is deprecated in lieu of QuotaSpec.RegionLimit.Storage.Variables and will be removed in Nomad 1.12.0.

Loading binaries from plugin_dir without configuration

Plugins stored within the plugin_dir will now only be loaded when they have a corresponding plugin block in the agent configuration file. Nomad now skips any plugin found without a corresponding configuration block.

Vault and Consul integration changes

Nomad 1.10.0 removes the previously deprecated token-based authentication workflow for Vault and Consul. Nomad clients must now use a task's workload identity to authenticate to Vault and Consul and obtain a token specific to the task.

This table lists removed Vault fields and the new workflow.

FieldConfigurationNew Workflow
vault.allow_unauthenticatedAgentTasks should use a workload identity. Do not use a Vault token.
vault.task_token_ttlAgentWith workload identity, tasks receive their TTL configuration from the Vault role.
vault.tokenAgentNomad agents use the workload identity when making requests to authenticated endpoints.
vault.policiesJob specificationConfigure and use a Vault role.

Before upgrading to Nomad 1.10, perform the following tasks:

  1. Configure Vault and Consul to work with workload identity.
  2. Migrate all workloads to use workload identity.

Refer to the following guides for more information:

Consul template implicit workload identity removal

Nomad no longer creates an implicit Consul identity for workloads that don't register services with Consul. Tasks that require Consul tokens for template rendering must include a consul block or specify an identity.

Nomad 1.8 deprecated disconnect fields removed

In Nomad 1.8, we introduced the disconnect block to replace the max_client_disconnect, stop_after_client_disconnect, and prevent_reschedule_on_list fields. In Nomad 1.10, we removed these fields, and Nomad will ignore them if specified. Jobs should migrate to using the disconnect block prior to upgrading.

Remote task driver support removed

In Nomad 1.10.0, we removed all support for remote task driver capabilities. Nomad no longer detaches drivers with the RemoteTasks capability when an allocation is lost. Also, Nomad does not detach remote tasks when a node is drained. Workloads running as remote tasks should be migrated prior to upgrading.

Sentinel apply command requires scope Enterprise

To prevent accidentally adding policies for volumes to the job scope, the nomad sentinel apply command now requires the -scope option. Refer to the GitHub pull request for details.

Affinity and spread updates are non-destructive

We fixed a scheduler bug so that updates to affinity and spread blocks are no longer destructive. After a job update that changes only these blocks, existing allocations remain running with their job version incremented. If you were relying on the previous behavior to redistribute workloads, you can force a destructive update by changing fields that require one, such as the meta block.

Deprecations

Quota specification variable_limits deprecated Enterprise

The quota specification's variable_limits field is deprecated. We replaced it with a new storage block with a variables field, under the region_limit block. Existing quotas will be automatically migrated during server upgrade. We will remove the variables_limit field from the quota specification in Nomad 1.12.0.

Upgrade details

For more detailed information, refer to the upgrade details page and the GitHub releases changelogs.

Known issues

None.

Changelogs

These links take you to the changelogs on the GitHub website.

Edit this page on GitHub