Secrets and Variables

Understanding and Implementing Nomad Variables and Secrets

Nomad provides a way to store and manage variables. It uses a path-based structure for organizing and accessing data, similar to HashiCorp Vault and is replicated between servers.

Variables can be stored at different paths within Nomad, including job, group, and task levels, or any custom path. Variables defined at the job level are not accessible to other jobs, group-level variables are isolated to their specific group, and task-level variables are only accessible by specific tasks. Along with ACL's, this scoped accessibility helps maintain the principle of least privilege and reduces the blast radius if a variable is compromised.

Nomad variables are useful for storing sensitive information such as API keys, credentials, and other small pieces of data that tasks may require in a key-value format. They are truly encrypted, not merely base64 encoded. Encryption ensures that sensitive information remains secure both at rest and in transit.

The encryption keys are managed by Nomad, providing an additional layer of security compared to storing secrets in plain text or using weak encoding methods.

Nomad vs HCL Variables

Nomad supports two types of variables: Nomad variables and HCL (HashiCorp Configuration Language) variables. While both serve the purpose of storing and using data within Nomad, they have distinct characteristics and use cases.

Nomad variables are static k/v formatted variables which are stored by Nomad and can be dynamically accessed at runtime by your application. They are encrypted and replicated across Nomad servers, making them suitable for storing sensitive information such as API keys or credentials. Nomad variables can be created, updated, and deleted using Nomad's CLI, API, or web UI.

Example use of Nomad variables

Variables are accessed by using the template and nomadVarList function.

template {
  data        = <<EOH
{{ range "path/to/filter/" }}
  {{ . }}
{{ end }}
EOH
}

template {
  data        = <<EOH
{{ range nomadVarList "nomad/jobs/somejob" }}
  {{ .Path }}
{{ end }}
EOH
}

Because templates are used, the variable can be updated without resubmitting the job. The template will detect a change in the variable and automatically restart by default. which makes it ideal for application level configuration data. Here is a diagram to show the flow of the templating engine:

HCL Variables

HCL variables are defined within job specifications and are typically used for configuration values that are known at the time of job submission. These variables are part of the job file and are processed when the job is parsed, so any changes to the variable will require resubmitting the job which redeploys everything within the job.

This makes it more suitable for job file data over application data such as defining a job name or a namespace.

For more information, visit Input Variables documentation page.

Example use of HCL variables

HCL variables are defined just like Terraform configuration

variable "image" {
  description = "The docker image to use"
  type        = string
  default     = "someimage:1.5.3"
}

job "myapp" {

#...

task "app" {
    driver = "docker"
    config {
        image = var.image
    }

#...

Choosing Nomad Variables versus HCL Variables

When deciding between Nomad variables and HCL variables, consider your existing workflows and the specific needs of your deployment.

If you're already familiar with Terraform workflows, using HCL variables in Nomad can ease the adoption process, as you won't need to learn a new system for managing variables.

Nomad job files are designed to be easily readable and understandable, so using HCL variables for simple, non-sensitive configurations like image tags or defining a namespace is often sufficient. However, for sensitive data or values that need to be dynamically updated, Nomad variables offer better security and flexibility.

If you require dynamic templates for your CI/CD pipelines, consider using Nomad Pack instead of extensively using HCL variables. Over-reliance on HCL variables can limit your pipeline's scalability and flexibility in the long run.

Here is a table summarizing the differences:

Nomad Variables vs Vault

Using Nomad variables for secrets is beneficial for organizations in the early stages of adoption or those seeking a simple secrets management for your applications without immediately needing to implement additional systems.

Storing secrets within Nomad is ideal for small static secrets such as API keys, credentials, or configuration values that don't change frequently. However, because Nomad variables consume server resources and are not designed to handle large volumes of rapidly changing data, such as batch job results, organizations at scale or with dynamic secret needs should consider using Vault for more robust and scalable secrets management.

Offloading secrets management to Vault not only provides enhanced capabilities but also frees up compute resources on the Nomad server nodes. However, it's important to note that Nomad's secrets management capabilities are limited compared to dedicated secrets management solutions like HashiCorp Vault. Nomad does not support dynamic secrets generation or advanced features such as PKI (Public Key Infrastructure) management.

While Nomad's built-in secrets management is sufficient for many use cases and provides an excellent starting point, organizations with more complex requirements or growing Nomad footprint should consider implementing Vault alongside Nomad.

In short, storing Nomad variables as secrets is a great option for static secrets and ease of adoption, but for dynamic secrets, PKI, and more advanced secrets management features, Vault is the recommended solution.

How to Use Variables

Creating a variable

You can create and store a variable in three different ways: CLI, API, and UI.

You can define variables at different levels - job, group, or task. This hierarchical approach lets you control access to variables, limiting their visibility to specific scopes. For instance, a variable created at the task level is only accessible within that task, while a job-level variable can be used across groups and tasks within that job.
Some variables you may want to be accessible by all jobs which can be stored in a path of your choosing. If you do this, ensure the variable either is appropriate to be globally accessed or if not, it has the appropriate ACL policies attached.

CLI

Store variables tutorial covers how to create and store variables

API

[v1/var/](https://developer.hashicorp.com/nomad/api-docs/variables/variables#create-variable) endpoint creates, reads, and lists variables

UI

Go to the job for which you want to create a variable and click “Create a Variable” Input the k/v you’d like to add Click "Save Variables"

ACL's

You should always protect access to variables with Access Control Lists (ACLs).

The workload identity for each task automatically grants read and list access to variables found at Nomad-owned paths with the prefix nomad/jobs/, followed by the job ID, task group name, and task name.

It is recommended to create proper ACL policies for all variables stored outside the standard workload identity created ACL's. Visit Identity and Access Management portion of the Operating Guide, Task Access to Variables for more information on adding additional policies for a particular variable path or the Access Control tutorial.

Accessing from Tasks

Accessing variables and secrets from tasks is done by using template and nomadVarList function, however if you want a step by step tutorial visit the Access variables tutorial.

It’s recommended to render templates containing sensitive information in the secrets/ task working directory, which offers enhanced protection by being inaccessible outside the task, unreadable via API or CLI, and stored in-memory. For variables that are not sensitive, they can be stored under local or tmp directory which will be stored in plaintext.

For more information on the filesystem architecture of Nomad, visit the Filesystem concepts page.

Secret Environment Variable Example

This example shows accessing a variable from the "api-app" job path and assigning the value to the API_KEY environment variable and is rendered to the tasks secrets/ directory.

Groups and Tasks within the job can access the api-key variable and no other jobs or requests outside of the job can access it. The rendered template is only accessible to the task itself.

template {
  data        = <<EOH
{{ range nomadVarList "nomad/jobs/api-app" }}
  API_KEY = {{ .api-key }}
{{ end }}
EOH
    destination = "secrets/vars.env"
    env = true
}

Non-sensitive Environment Variable Example

This example shows a variable that has been created under the nomad/jobs path, which makes it accessible to all jobs within Nomad.

template {
  data        = <<EOH
{{ range nomadVarList "nomad/jobs" }}
  VAR = {{ .key }}
{{ end }}
EOH
    destination = "local/vars.env"
    env = true
}

Configuration File example

Nomad secrets and variables are not only for environment variables, but can also be used for configuration files.

template {
    data        = <<EOH
<!DOCTYPE html>
<html lang="en">
  <head><meta charset="utf-8"><title>Hello Variables</title></head>
  <body>
    <p>The task has access to the key variable! The key variable is:</p>
    <ul>
    
      {{- range nomadVarList "nomad/jobs/nginx" }}
      <li>{{ .key }}</li>
      {{- end }}
      
    </ul>
    <p><a href="/">View the index page.</a></p>
  </body>
</html>
EOH
        destination = "local/index.html"
      }

Locks

Nomad provides the ability to block a variable from being updated for a period of time by setting a lock on it.

Once a variable is locked, it can be read by everyone, but it can only be updated by the lock holder.

Variable locks are great for stateful applications that are clustered together that utilize leader election. View the variable locks concepts document for more details.