Identity and access management

The access control lists (ACL) system in Nomad provides authentication and authorization, giving Nomad administrators fine-grained control over what users and applications can do within the Nomad environment.

Structuring ACLs will be organization-dependent, but there is set of good general practices to follow:

• Create role-based policies:
  • Define roles like "developer", "operator", and "admin" instead of one-off tokens.
• Use the principle of least privilege:
  • Grant minimal permissions necessary for each role.
• Use namespaces:
  • Use namespaces to isolate different tenants, teams, applications and/or environments.
• Use wildcards judiciously:
  • Be cautious with wildcards to avoid overly broad permissions.
• Avoid static/long-lived tokens:
  • Use either short lived Nomad ACL tokens, the Vault Nomad secrets engine(opens in new tab) for time-bound dynamic tokens, or Nomad single sign-on with OIDC(opens in new tab) which allows trusting external OIDC-capable identity providers, such as Okta, Auth0, EntraID, etc. for human access, or GitHub Actions, GitLab CI, Vault, etc. for machine access.
• Implement a robust offboarding process for any static tokens:
  • Ensure quick removal of access when users leave/applications no longer need access.
• Regular access audits:
  • Review ACL policies and any static tokens periodically.
• Enable audit logs:
  • Use audit logs to monitor and detect suspicious activity.
• Use Sentinel(opens in new tab):
  • Sentinel is a policy as code framework that allows you to define even more fine-grained policies that are enforced at job deploy time, and can be used to enforce extra security or business logic (such as to only authorize Docker containers from a specific Docker registry, or disallow deploys on Friday afternoon, etc.).
• Treat namespaces and ACL policies as code:
  • Store them in a version control system, ideally manage them via Terraform(opens in new tab), and use CI/CD pipelines to deploy them to Nomad. This ensures that changes are reviewed and audited, and that they are applied consistently across all environments.

ACL management

ACLs are composed of capabilities, policies, roles and tokens. Policies define what actions are allowed or denied either by assigning policies which contain high level read/write access, or by listing explicit capabilities, roles define which policies are attached to a token, and tokens are used to authenticate to Nomad (and can be brokered by Vault or an SSO identity provider). Policies are written in HashiCorp Configuration Language (HCL) or JSON, and can also be managed via Terraform.

It is generally recommended to use Terraform to manage namespaces, ACL policies and roles, as it allows for easier version control and code review, as well as for an easier understanding of all the policies in use in the Nomad cluster, drift detection and remediation. It can also be used to enable a self-service model using Terraform modules, where common combinations of namespace+associated policies can be rendered easily available to or requested by users.

An example of a policy that allows submitting jobs and a few related activities in a specific namespace, using raw HCL for Nomad or Terraform is below.

Raw HCL can also be used.

# Allow reading resources within the "dev" namespace as well as submitting
# jobs to this namespace, without allowing access to view log output or inspect
# the filesystem.
namespace "dev" {
  capabilities = ["submit-job", "list-jobs", "read-job", "parse-job"]
}

Such a policy can be created in Nomad via the CLI or API as follows.

nomad acl policy apply -description "Allow reading resources within the 'dev' namespace as well as submitting jobs to this namespace, without allowing access to view log output or inspect the filesystem." dev-policy dev.hcl

Alternatively, the same policy can be created as follows via Terraform using the nomad_acl_policy(opens in new tab) resource from the Nomad Terraform provider(opens in new tab).

resource "nomad_acl_policy" "dev" {
  name        = "dev-policy"
  description = "Allow reading resources within the 'dev' namespace as well as submitting jobs to this namespace, without allowing access to view log output or inspect the filesystem."

  rules_hcl = <<EOT
namespace "dev" {
  capabilities = [
    "submit-job",
    "list-jobs",
    "read-job",
    "parse-job"
  ]
}
EOT
}

Once that policy is created, it can be attached to an ACL role, or can directly be used to create a token with it. Preferably, SSO should be used to authenticate users and applications to roles.

SSO via OIDC

Using an external identity provider (IdP) for authentication and authorization is a recommended pattern for security and simplicity. Nomad supports OIDC (OpenID Connect) for Single Sign-On (SSO) with external identity providers such as Okta, Auth0, EntraID, etc. for human auth, and GitHub Actions, GitLab CI, etc. for machine auth. This allows users, pipelines and applications to authenticate to Nomad using their existing identity, and Nomad to trust the IdP to provide the necessary information about the user, such as their username, groups, metadata, etc. This is more secure than static tokens, as it allows for centralized management of users and their access, which is short-lived and time bound, and allows for easy revocation of access when users leave the organization or no longer need access.

To enable OIDC, you need to configure the necessary ACL policies and ACL roles in Nomad, and create an auth method (via the CLI(opens in new tab) or preferably, the nomad_acl_auth_method(opens in new tab) Terraform resource).

An example tutorial with Auth0 can be found here(opens in new tab), but the same general principles apply to any OIDC provider.

Workload Identity (also via OIDC)

The ACL system is what enables Nomad to provide Workload Identity to each workload running in Nomad, and serve as an OIDC identity provider. This allows to uniquely and securely authenticate and authorize workloads running in Nomad without any static credentials needed. This enables workloads running in Nomad to authenticate to Nomad itself, Vault, assume a role in AWS IAM/GCP IAM, or any other OIDC/JWT compatible system.

Internal

By default, each allocation in Nomad gets an identity which gives it implicit access to the Nomad API. This identity is unique to the allocation, and gives it read-only access to Nomad Variables in the respective path for that allocation's job/group/task (nomad/jobs/job/task_group/task). Additional explicit ACL policies can be attached to jobs/groups/tasks to enhance that identity, and provide granular access to alternative Nomad Variable paths, or the Nomad Task API which allows for secure and local access to the Nomad API from within tasks (perfect for load balancers, control tasks, automations, etc.).

Third party OIDC-compatible systems

To use Workload Identity for auth to OIDC-compatible third parties (such as Vault, AWS IAM, GCP IAM, etc.), you need to configure the OIDC issuer in Nomad, and configure trust between the external system and Nomad as an Identity Provider (IdP). The exact flow and configuration will depend on the third paty system trusting Nomad:

• Vault: Nomad workloads can authenticate to the OIDC/JWT Vault Authentication method
• AWS IAM: Nomad workloads can assume AWS IAM roles
• GCP Workload Identity Federation: GCP IAM identities can be federated to Nomad workloads