Using Sentinel policy as code for Nomad

Sentinel is HashiCorp's policy-as-code framework and language, and is integrated directly into Nomad Enterprise. It allows you to define and enforce fine-grained policies using business, security or compliance logic across your Nomad clusters.

By leveraging Sentinel, you can ensure compliance, security, and operational best practices are consistently applied to all workloads.

When to Use Sentinel and when to use access control lists

While both Sentinel and access control lists (ACLs) contribute to having a complete security and governance posture, they serve different purposes:

• Use Sentinel for complex, dynamic policies that require logic-based decision making, for example, enforcing resource quotas based on namespace or job name, allowlisting approved image registries, enforcing sensible defaults, or restricting when deployments can happen.

• Use ACLs for basic access control, such as determining which users or teams can submit jobs, or exec into them to specific namespaces.

Sentinel and ACLs are complementary, and should be used in tandem for a comprehensive security and compliance strategy. ACLs provide the foundational access control, while Sentinel adds an additional layer of fine-grained policy enforcement.

Best practices for authoring Sentinel policies

When crafting Sentinel policies for Nomad consider these recommended practices.

1. Modularity: Write focused policies that address specific concerns. This approach enhances readability and maintainability.

2. Performance: Be aware of the performance impact of Sentinel policies - all of them run, sequentially, on every job submission. Keep policies efficient, avoid unnecessary complexity and duplication of logic, and having too many small policies. As an example, having two separate policies to check that all tasks have both cpu and memory limits configured can easily be condensed into a single one checking for both.

3. Clear naming: Use descriptive names for your policies and functions. For example, enforce_resource_limits is more informative than check_resources.

4. Thorough testing: Always test your policies against various scenarios to ensure they behave as expected. The Sentinel CLI and the Sentinel playground are invaluable tools for this purpose. To get mock data for realistic testing, you can use the Nomad API to get the JSON definition an existing job, which needs to be converted to snake case (example with jq) and put into a job object.

5. Progressively Roll out policies: When introducing new policies, consider rolling them out progressively to avoid sudden workflow disruptions. Start with advisory mode to give users the time to prepare and adjust, upgrade to soft-mandatory, which requires an extra ACL and flag/parameter to bypass, to enforce the policy but allow users to override, and finally switch to hard-mandatory to enforce the policy without exceptions.

Note that as a Nomad operator, currently there is no way of tracking failed advisory-level policies. Soft-mandatory and hard-mandatory failures get reported in the audit logs.

1. Treat Sentinel policies as code: Version and track changes to your Sentinel policies in a git repository to facilitate collaboration, allowing rollbacks if needed. Implement a peer review process for policy changes to help catch potential issues and ensure policies align with organizational goals. These should be managed via the Terraform provider for Nomad's nomad_sentinel_policy resource.

The new Sentinel UI in Nomad

Since Nomad Enterprise 1.8, there is now a dedicated Sentinel UI in the Administration tab, which enables you to manage and visualize your Sentinel policies directly within the Nomad UI. It has a policy overview, a policy writing UI with syntax highlighting and validation, and a template library to get started quickly.

Example Sentinel policies

Practical example policies can be found among the templates in the Sentinel UI in the Administration tab and the nomad-guides GitHub repository.