Testing

Sentinel has a built-in test framework to validate a policy behaves as expected.

With an ever-increasing amount of automation surrounding technology, the guardrails provided by policies are a critical piece towards ensuring expected behavior. As a reliance on correct policy increases, it is important to test and verify policies.

Testing is a necessary step to fully realize policy as code. Just as good software is well tested, a good set of policies should be equally well tested.

Test Folder Structure

Policies are tested by asserting that rules are the expected values given a pre-configured input. Tests are run by executing the test command.

Sentinel is opinionated about the folder structure required for tests. This opinionated structure allows testing to be as simple as running sentinel test with no arguments. Additionally, it becomes simple to test in a CI or add new policies.

The structure Sentinel expects is test/<policy>/*.json where <policy> is the name of your policy file without the file extension. Within that folder is a list of JSON files. Each JSON file represents a single test case. Therefore, each policy can have multiple tests associated with it.

Test Case Format

Each JSON file within the test folder for a policy is a single test case.

The JSON file is the same configuration format as the CLI configuration file. The format lets you define mock data, imports to use, and more. This mock data is the key piece in being able to test policies: you craft a specific scenario and assert your policy behaves as you expect.

Test cases also use the special key test within the configuration file to assert the boolean value of rules. If the test key is omitted, the policy is expected to pass (the same as asserting that the main rule is true). If the test key is specified, only the rules specified in the map will be asserted. This means if you omit main, then the final policy result is not asserted.

Example with assertions:

{
  "param": {
    "day": "monday",
    "hour": 7
  },

  "test": {
    "main": false,
    "is_open_hours": false,
    "is_weekday": true
  }
}

The configuration above specifies some parameter data, and asserts the result of some rules. This is the same configuration used in the example section below.

Example

Lets use the following file as an example. Save this file to a directory and name it policy.sentinel. It can be named anything with the sentinel extension, but by naming it policy.sentinel your output should match the example output on this page.

// The day of the week.
param day

// The hour of the day.
param hour

is_weekday = rule { day not in ["saturday", "sunday"] }
is_open_hours = rule { hour > 8 and hour < 17 }
main = rule { is_open_hours and is_weekday }

A Passing Test

Next, let's define a single test case. Relative to where you saved the policy, create a file at the path test/policy/good.json.

{
  "param": {
    "day": "monday",
    "hour": 14
  }
}

Now run sentinel test:

$ sentinel test
PASS - policy.sentinel
  PASS - test/policy/good.json

This passed because the policy passed. We didn't assert any specific rules. By not specifying any assertions, test expects the policy itself to fully pass.

A Failing Test

Define another test case to fail. We want to verify our policy fails when expected, too.

Save the following as test/policy/7-am.json:

{
  "param": {
    "day": "monday",
    "hour": 7
  }
}

Now run sentinel test:

$ sentinel test
FAIL - policy.sentinel
  FAIL - test/policy/7-am.json
    expected "main" to be true, got: false
 
    trace:
      FALSE - policy.sentinel:9:1 - Rule "main"
        FALSE - policy.sentinel:9:15 - is_open_hours
          FALSE - policy.sentinel:8:24 - hour > 8 and hour < 17
            FALSE - policy.sentinel:8:24 - hour > 8
 
      FALSE - policy.sentinel:8:1 - Rule "is_open_hours"
        FALSE - policy.sentinel:8:24 - hour > 8
  PASS - test/policy/good.json

As you can see, the test fails because "main" is false. This is good because the policy should have failed since we specified an invalid hour. But, we expect main to be false and don't want our test to fail! Update 7-am.json to add test assertions:

{
  "param": {
    "day": "monday",
    "hour": 7
  },

  "test": {
    "main": false,
    "is_open_hours": false,
    "is_weekday": true
  }
}

And when we run the tests:

$ sentinel test
PASS - policy.sentinel
  PASS - test/policy/7-am.json
  PASS - test/policy/good.json

The test passes. We asserted that we expect the main rule to be false, the is_open_hours rule to be false, and the is_weekday rule to be true. By asserting some rules are true, we can verify that our policy is failing for reasons we expect.

Mocking

The above example demonstrates how to test by supplying different parameters. Parameters in a policy can be specifically useful when you want to control user-defined input values to a policy.

However, generally, when testing, you will need mimic the conditions you will see in production. Production implementations of Sentinel will supply data using one of two methods:

• Global data: Data is injected directly into the policy's scope and is accessible using normal identifiers, similar to variables.
• Imports: Data is stored behind an import and loaded on demand as needed by the policy author.

Proper testing of a policy requires that these values be able to be mocked - or, in other words, simulated in a way that allows the accurate testing of the scenarios that a policy could reasonably pass or fail under.

Mocking both globals and imports can be done by setting various parts of the configuration file.

Mocking Globals

Demonstrating the mocking of globals can be seen by making a few modifications to our example policy, removing the param declarations:

is_weekday = rule { day not in ["saturday", "sunday"] }
is_open_hours = rule { hour > 8 and hour < 17 }
main = rule { is_open_hours and is_weekday }

Then, change the param section in the configuration file to global.

{
  "global": {
    "day": "monday",
    "hour": 14
  }
}

This test should still pass, as if nothing had happened, although what we've done is shifted our parameters to globals, simulating an environment where day and hour are already defined for us.

Mocking Imports

To mock imports, we need to use the mock section of the configuration file.

Let's say the above example is behind an import named time.

The code now looks like this:

import "time"

is_weekday = rule { time.now.weekday_name not in ["Saturday", "Sunday"] }
is_open_hours = rule { time.now.hour > 8 and time.now.hour < 17 }
main = rule { is_open_hours and is_weekday }

To mock this import, we can mock it as static data. The configuration file now looks like, without assertions:

{
  "mock": {
    "time": {
      "now": {
        "weekday_name": "Monday",
        "hour": 14
      }
    }
  }
}

The policy will now pass, with the time import mocked.

Data can also be mocked as Sentinel code. In this case, the above configuration file would look like:

{
  "mock": {
    "time": "mock-time.sentinel"
  }
}

And a file named mock-time.sentinel would now hold your mock values:

day = "monday"
hour = 14

Mocking as Sentinel code allows more complex details to be mocked as well, such as functions. Say we wanted to mock the time.load() function. To mock this, just add it to the mock-time.sentinel file:

load = func(_) {
    return {
        "weekday_name":  "Monday",
        "hour": 14,
    }
}

Your code can now be written as:

import "time"

t = time.load("a_mock_timestamp")

is_weekday = rule { t.weekday_name not in ["Saturday", "Sunday"] }
is_open_hours = rule { t.hour > 8 and t.hour < 17 }
main = rule { is_open_hours and is_weekday }

To see more details, see the Mock Imports section in the configuration file.

Test Automation

The sentinel test command was designed to be automation friendly. We encourage you to enable a CI such as Travis CI on your policy repositories to continuously run tests. An example .travis.yml configuration file is shown below:

language: bash
script: sentinel test