Secrets and variables

Understanding and implementing Nomad Enterprise variables and secrets

Nomad Enterprise provides a way to store and manage variables. It uses a path-based structure for organizing and accessing data, similar to HashiCorp Vault and replicates between servers.

Use different paths within Nomad Enterprise to store variables, including job, group, and task levels, or any custom path. Variables defined at the job level are not accessible to other jobs. Nomad Enterprise isolates group-level variables to their specific group, and task-level variables are only accessible by specific tasks. Along with ACLs, this scoped accessibility helps maintain the principle of least privilege and reduces the blast radius for compromised variables.

Nomad variables are useful for storing sensitive information such as API keys, credentials, and other small pieces of data that tasks may require in a key-value format. They are truly encrypted, not merely base64 encoded. Encryption ensures that sensitive information remains secure both at rest and in transit.

The encryption keys are managed by Nomad, providing an additional layer of security compared to storing secrets in plain text or using weak encoding methods.

Nomad versus HCL variables

Nomad Enterprise supports two types of variables: Nomad variables and HCL (HashiCorp Configuration Language) variables. While both serve the purpose of storing and using data within Nomad Enterprise, they have distinct characteristics and use cases.

Nomad variables are static k/v formatted variables. Nomad Enterprise stores, encrypts and replicates these and they be dynamically accessed at runtime by your application. Replication makes them suitable for storing sensitive information such as API keys or credentials. Use the command-line tool, API, or web UI to create, update, and delete Nomad variables.

Example use of Nomad variables

Access variables by using the template and nomadVarList function.

template {
  data        = <<EOH
{{ range "path/to/filter/" }}
  {{ . }}
{{ end }}
EOH
}

template {
  data        = <<EOH
{{ range nomadVarList "nomad/jobs/somejob" }}
  {{ .Path }}
{{ end }}
EOH
}

Because of the use of templates, the job does not need to be resubmitted to update the variable value. Nomad Enterprise will use the template to detect a change in the variable and automatically restart by default, which makes it ideal for application level configuration data. Here is a diagram to show the flow of the templating engine:

HCL variables

Job specifications define HCL variables and typically use them for configuration values known at the time of job submission. These variables are part of the job file and Nomad Enterprise processes these when parsing the job, so any changes to the variable require resubmitting the job which redeploys everything within the job. This makes it more suitable for job file data over application data such as defining a job name or a namespace.

For more information, visit Input Variables documentation page.

Example use of HCL variables

Define HCL variables just like Terraform configuration.

variable "image" {
  description = "The docker image to use"
  type        = string
  default     = "someimage:1.5.3"
}

job "myapp" {

#...

task "app" {
    driver = "docker"
    config {
        image = var.image
    }

#...

Choosing between Nomad variables and HCL variables

When deciding between Nomad variables and HCL variables, consider your existing workflows and the specific needs of your deployment. If you are already familiar with Terraform workflows, using HCL variables in Nomad Enterprise can ease the adoption process, as you do not need to learn a new system for managing variables.

The Nomad job specification allows job files to be readable and understandable. Use HCL variables for non-sensitive configurations like image tags or defining a namespace is often sufficient. However, for sensitive data or values that need to be dynamically updated, Nomad variables offer better security and flexibility.

If you require dynamic templates for your CI/CD pipelines, consider using Nomad Pack instead of extensively using HCL variables. Over-reliance on HCL variables can limit your pipeline's scalability and flexibility in the long run.

Here is a table summarizing the differences:

Nomad variables versus Vault variables

Using Nomad variables for secrets is beneficial for organizations in the early stages of adoption or those seeking a straightforward secrets management for your applications without needing to implement additional systems until later.

Storing secrets within Nomad Enterprise is ideal for small static secrets such as API keys, credentials, or configuration values that do not change frequently. Nomad variables consume server resources and are not designed to handle large volumes of changing data such as batch job results. As such, scaled organizations or those with dynamic secret needs must consider using Vault for more robust and scalable secrets management.

Offloading secrets management to Vault not only provides enhanced capabilities but also frees up compute resources on the Nomad server nodes. However, it is important to note that a dedicated secrets management solutions like HashiCorp Vault exceeds the capabilities of Nomad's. Nomad Enterprise does not support dynamic secrets generation or advanced features such as PKI (Public Key Infrastructure) management.

While Nomad Enterprise's built-in secrets management is sufficient for many use cases and provides an excellent starting point. Consider implementing Vault alongside Nomad if you have more complex requirements or have a growing Nomad Enterprise footprint.

In short, storing Nomad variables as secrets is a great option for static secrets and ease of adoption, but for dynamic secrets, PKI, and more advanced secrets management features, Vault is the recommended solution.

How to use variables

Creating a variable

You can create and store a variable in three different ways: command-line tool, API, and UI. You can define variables at different levels - job, group, or task. This hierarchical approach lets you control access to variables, limiting their visibility to specific scopes. For instance, a variable created at the task level is only accessible within that task, while a job-level variable is usable across groups and tasks within that job.

Store variables you may want accessible by all jobs in a path of your choosing. If you do this, ensure the variable either is appropriate to be globally accessed or if not, it has the appropriate ACL policies attached.

Command-line tool

Store variables tutorial covers how to create and store variables

API

v1/var/ endpoint creates, reads, and lists variables

UI

Go to the job for which you want to create a variable and click Create a variable Input the required k/v Click "Save Variables"

ACLs

Always protect access to variables with Access Control Lists (ACLs).

The workload identity for each task automatically grants read and list access to variables found at Nomad-owned paths with the prefix nomad/jobs/, followed by the job ID, task group name, and task name.

We recommend creating proper ACL policies for all variables stored outside the standard workload identity created ACLs. Visit the Identity and Access Management portion of the Operating Guide, Task Access to Variables for more information on adding additional policies for a particular variable path or the Access Control tutorial.

Accessing variables from tasks

Access variables and secrets from tasks by using template and nomadVarList function, however if you want a step by step tutorial visit the Access Variables tutorial.

We recommend rendering templates containing sensitive information in the secrets/ task working directory, which offers enhanced protection by being inaccessible outside the task, and unreadable to the API, command-line tool, or stored in-memory. For variables that are not sensitive, optionally store these under local or tmp directory. Nomad Enterprise uses plain text for these.

For more information on the filesystem architecture of Nomad, visit the Filesystem concepts page.

Secret environment variable example

This example shows accessing a variable from the api-app job path and assigning the value to the API_KEY environment variable. Nomad Enterprise renders this to the tasks secrets/ directory.

Groups and tasks within the job can access the api-key variable and no other jobs or requests outside of the job can access it. The rendered template is only accessible to the task itself.

template {
  data        = <<EOH
{{ range nomadVarList "nomad/jobs/api-app" }}
  API_KEY = {{ .api-key }}
{{ end }}
EOH
    destination = "secrets/vars.env"
    env = true
}

Non-sensitive environment variable example

This example shows a variable we store under the nomad/jobs path, which makes it accessible to all jobs within Nomad.

template {
  data        = <<EOH
{{ range nomadVarList "nomad/jobs" }}
  VAR = {{ .key }}
{{ end }}
EOH
    destination = "local/vars.env"
    env = true
}

Configuration file example

Nomad secrets and variables are not only for environment variables. Also use these for configuration files.

template {
    data        = <<EOH
<!DOCTYPE html>
<html lang="en">
  <head><meta charset="utf-8"><title>Hello Variables</title></head>
  <body>
    <p>The task has access to the key variable! The key variable is:</p>
    <ul>

      {{- range nomadVarList "nomad/jobs/nginx" }}
      <li>{{ .key }}</li>
      {{- end }}

    </ul>
    <p><a href="/">View the index page.</a></p>
  </body>
</html>
EOH
        destination = "local/index.html"
      }

Locks

Nomad Enterprise can block the update of a variable for a period of time by setting a lock on it. Upon locking a variable, everyone can read it, but only the lock-holder can update it.

Variable locks are great for stateful applications clustered together that utilize leader election. View the variable locks concepts document for more details.