Nomad
Authenticate users with PKCE and private key JWT
OpenID Connect (OIDC) authentication lets you provide Nomad access control list (ACL) tokens to users when they log in using a single sign-on (SSO) provider.
Nomad's OIDC SSO login feature includes these capabilities:
- Proof Key for Code Exchange, also known as PKCE, adds extra security for both traditional client secrets and client assertions.
- Private Key JWT, also known as client assertions, is a more secure alternative to client secrets.
These options offer extra security during the OIDC login flow. For a summary, refer to the Nomad OIDC concepts page.
To demonstrate how to use PKCE and Private Key JWT, this tutorial uses the open source Keycloak identity provider application running as a Nomad job. You will configure Nomad and Keycloak to automatically grant permissions in Nomad ACL, enable the Keycloak provider to require PKCE, and reconfigure the Keycloak provider to use client assertions.
Prerequisites
For this tutorial, you will need:
- The Nomad CLI version 1.10.0 or higher installed locally
- A Nomad v1.10 or higher cluster with ACLs enabled and bootstrapped, as well as:
- The ACL management token saved to the
NOMAD_TOKEN
environment variable - Docker installed on the client node
- The ACL management token saved to the
openssl
installed locally to generate client assertion certificates
Run Keycloak as a Nomad job
This job specification runs Keycloak in development mode with the username admin
and password admin
. Copy the contents to a file and save the file as keycloak.nomad.hcl
.
keycloak.nomad.hcl
variable "keycloak_tag" {
default = "26.0.8" # newer versions have a certificate upload bug at the moment
}
job "keycloak" {
group "kc" {
network {
port "http" {
static = 8080
}
}
service {
name = "keycloak"
port = "http"
provider = "nomad"
}
task "kc" {
driver = "docker"
config {
image = "quay.io/keycloak/keycloak:${var.keycloak_tag}"
args = ["start-dev", "--hostname", "http://${NOMAD_IP_http}:${NOMAD_PORT_http}"]
ports = ["http"]
# note: keycloak config will be lost unless you use persistent storage.
#volumes = [
# # /opt/keycloak-data on the host machine needs to be owned by user 1000
# "/opt/keycloak-data:/opt/keycloak/data",
#]
}
env {
KC_BOOTSTRAP_ADMIN_USERNAME = "admin"
KC_BOOTSTRAP_ADMIN_PASSWORD = "admin"
PROXY_ADDRESS_FORWARDING = true
KC_PROXY_HEADERS = "xforwarded"
}
resources {
memory = 800
}
}
}
}
Submit the job to Nomad.
$ nomad job run keycloak.nomad.hcl
Log in to Keycloak
Retrieve the Keycloak web address and open it in your web browser. In the following example, the address is 192.168.1.201:8080
.
$ nomad service info keycloak
Job ID Address Tags Node ID Alloc ID
keycloak 192.168.1.201:8080 [] 1885c25d c7f18cc3
At the Keycloak sign in page, enter the username admin
and the password admin
, and then click Sign In. Keycloak loads the landing page for the master realm with the Welcome to Keycloak message.
Configure Keycloak
You will now make the necessary configurations through the Keycloak web UI.
Create a realm
In Keycloak, a realm manages a set of users, credentials, roles, and groups. It is equivalent to a tenant.
- At the top left, click the dropdown list that displays Keycloak and master below it.
- Click Create realm.
- In Realm name, enter
Nomad
. - Click Create.
Keycloak creates a new realm, redirects you to the landing page for the Nomad realm, and displays the Welcome to Nomad message.
Create a group
A group in Keycloak is a set of attributes and role mappings that you can apply to a user. Users in this group will be given permissions in Nomad.
- In the left navigation, click Groups.
- Click Create group.
- In Name, enter
engineering
. - Click Create.
Create a user
In the left navigation, click Users.
Click Create new user.
Enter the following values for each labelled field:
Field name Value Username testuser
Email testuser@example.com
First Name Test
Last Name User
Click Join Groups.
Select the check box for the
engineering
group.Click Join.
Click Create to complete the process.
Next, create a password for the new user.
- From the User details page, at the top, click Credentials.
- Click Set password.
- Enter
password
in the Password and Password confirmation fields. - Turn off Temporary.
- Click Save.
- In the prompt that appears, click Save password.
Create a client scope
Nomad checks the ID token response from Keycloak for a group. Keycloak adds this group to the response with a client scope.
- In the left navigation, click Client scopes.
- Near the top of the page, click Create client scope.
- For Name enter
scope-for-nomad
. - Next to Type, click the dropdown and then select Default.
- Click Save.
Keycloak redirects you to the Client scope details page.
- In the top navigation, click Mappers.
- Click Configure a new mapper.
- In the Name column, click Group Membership.
- In the Add mapper form, for the Name, enter
nomad-mapper
. - For Token Claim Name, enter
kc-groups
. - Click Save.
Create the client
A client in Keycloak is an application or service that can request authentication of a user. Nomad is a client in Keycloak.
In the left navigation, click Clients.
Near the top of the page, click Create client.
For Client ID, enter
nomad-oidc
.Click Next to continue.
Turn on Client authentication.
Click Next to continue.
For Valid redirect URIs, enter
http://localhost:4649/oidc/callback
. Note that the port for this URI is4649
. This URI is for login attempts with the Nomad CLI.Click Add valid redirect URIs to add another URI.
Enter
https://localhost:4646/ui/settings/tokens
. Note that the port for this URI is4646
. This URI is for login attempts with the Nomad web UI.There are now two URI values in Valid redirect URIs. These values are the location that Keycloak redirects users to after they log in.
Click Save.
Verify scope for client
- From the Clients page, in the top navigation, click Client scopes.
- In the Search by name search field, enter
scope-for-nomad
. - Click the button with the arrow to search. Note that the Assigned type for the
scope-for-nomad
client scope is set to Default.
View credentials
Nomad requires a secret to verify that it has authorization to use this client in Keycloak.
- In the left navigation, click Clients.
- In the Client ID column, click nomad-oidc .
- At the top, click Credentials. The dropdown under Client Authenticator shows that Keycloak uses client ID and secret by default.
- Reveal the Client Secret.
- Copy this value and save it. You will use it in the next section.
Configure Nomad
Nomad requires the following configuration to complete the set up with Keycloak:
- An auth method using the Keycloak details
- A policy and role to specify user permissions in Nomad
- A binding rule to associate the Keycloak group to the Nomad role
Create an auth method
Copy the following example configuration to a file and then replace these placeholder values:
_KEYCLOAK_ADDRESS
: the address of the Keycloak service in Nomad, from the Log in to Keycloak section of this tutorial._KEYCLOAK_CLIENT_SECRET
: The client secret value from the View credentials section of this tutorial.
Save the file as auth-method-keycloak.json
.
Note that the ListClaimMappings
value maps the group kc-groups
in Keycloak to a value that you will define in the Nomad binding rule named keycloak_groups
.
auth-method-keycloak.json
{
"OIDCDiscoveryURL": "http://_KEYCLOAK_ADDRESS/realms/Nomad",
"OIDCClientSecret": "_KEYCLOAK_CLIENT_SECRET",
"OIDCClientID": "nomad-oidc",
"BoundAudiences": ["nomad-oidc"],
"AllowedRedirectURIs": [
"http://localhost:4649/oidc/callback",
"https://localhost:4646/ui/settings/tokens"
],
"OIDCScopes": [
"openid",
"scope-for-nomad"
],
"ListClaimMappings": {
"kc-groups": "keycloak_groups"
},
"VerboseLogging": true
}
Ensure that the NOMAD_TOKEN
environment variable is set to your Nomad management token, as only management tokens can create auth methods.
$ nomad acl token self | grep Type
Type = management
Create the auth method.
$ nomad acl auth-method create \
-type=OIDC \
-name=keycloak \
-default=true \
-max-token-ttl=5m \
-token-locality=global \
-config=@auth-method-keycloak.json
The command prints information about the auth method similar to the following example output.
Name = keycloak
Type = OIDC
Locality = global
Max Token TTL = 5m0s
Token Name Format = ${auth_method_type}-${auth_method_name}
Default = true
Create Index = 382
Modify Index = 382
Auth Method Config
JWT Validation Public Keys = <none>
JWKS URL = <none>
OIDC Discovery URL = http://192.168.1.201:8080/realms/Nomad
OIDC Client ID = nomad-oidc
OIDC Client Secret = redacted
OIDC Enable PKCE = false
OIDC Disable UserInfo = false
OIDC Scopes = openid,scope-for-nomad
Bound audiences = nomad-oidc
Bound issuer = <none>
Allowed redirects URIs = http://localhost:4649/oidc/callback,https://localhost:4646/ui/settings/tokens
Discovery CA pem = <none>
JWKS CA cert = <none>
Signing algorithms = <none>
Expiration Leeway = 0s
NotBefore Leeway = 0s
ClockSkew Leeway = 0s
Claim mappings = <none>
List claim mappings = {kc-groups: keycloak_groups}
Create policy and role
Create an ACL policy that grants access to the default
namespace and Nomad nodes.
First, create the policy file. Copy the following example policy to a file and save it as engineering-policy.hcl
.
namespace "default" {
capabilities = ["list-jobs"]
}
Create the policy.
$ nomad acl policy apply engineering-policy engineering-policy.hcl
Successfully wrote "engineering-policy" ACL policy!
Create a role and associate it to the policy.
$ nomad acl role create -name=engineering-role -policy=engineering-policy
ID = 6987c982-082d-35ac-aff3-57f24e5829a0
Name = engineering-role
Description = <none>
Policies = engineering-policy
Create Index = 448
Modify Index = 448
Create binding rule
Grant users in the /engineering
Keycloak group the Nomad ACL role named engineering-role
. Note the single quotes around the whole selector
value and the backticks around the /engineering
segment. These are important, as you must include /
in the selector expression.
$ nomad acl binding-rule create \
-auth-method=keycloak \
-bind-type=role \
-bind-name=engineering-role \
-selector='`/engineering` in list.keycloak_groups'
The command prints information about the binding-rule similar to the following example output.
ID = b5d6bc45-0667-868e-7054-a75b45b6676f
Description = <none>
Auth Method = keycloak
Selector = "`/engineering` in list.keycloak_groups"
Bind Type = role
Bind Name = engineering-role
Create Time = 2025-03-25 20:39:25.923021 +0000 UTC
Modify Time = 2025-03-25 20:39:25.923021 +0000 UTC
Create Index = 451
Modify Index = 451
Log in to Nomad
In your terminal session, log in to Nomad. You do not need additional arguments since the auth method included the -default=true
flag. This command opens a web browser window to Keycloak and prompts you to sign in.
$ nomad login
Enter testuser
in the Username or email field and password
in the Password field.
After a successful login, Keycloak redirects you to a page at localhost:4649/oidc/callback
that contains text with the message Signed in via your OIDC provider
.
Open your terminal and note the output from the login command. You are now signed into Nomad via authentication with Keycloak.
$ nomad login
Successfully logged in via OIDC and keycloak
Accessor ID = 3b45aa90-eeb7-7b55-11d6-fe5d597b1753
Secret ID = 83e70c65-54b4-e2a3-3158-6ab59b9bb989
Name = OIDC-keycloak
Type = client
Global = true
Create Time = 2025-03-26 14:02:27.365842 +0000 UTC
Expiry Time = 2025-03-26 14:07:27.365842 +0000 UTC
Create Index = 39
Modify Index = 39
Policies = []
Roles
ID Name
4d60501e-57c6-07ba-d06d-72f76ab1f650 engineering-role
Refer to the OIDC auth method page for information about OIDC configuration and troubleshooting.
Test the token
Copy the Secret ID from the output and use it to query Nomad for jobs status. In this example, the Secret ID is 83e70c65-54b4-e2a3-3158-6ab59b9bb989
.
$ NOMAD_TOKEN=83e70c65-54b4-e2a3-3158-6ab59b9bb989 nomad job status
ID Type Priority Status Submit Date
keycloak service 50 running 2025-03-26T09:51:59-04:00
Copy the secret ID again and use it to query the status of the keycloak
job. This command will fail because the OIDC token does not have the required permissions.
$ NOMAD_TOKEN=83e70c65-54b4-e2a3-3158-6ab59b9bb989 nomad job status keycloak
Error querying job: Unexpected response code: 403 (Permission denied)
The -max-token-ttl=5m
flag of the nomad acl auth-method create
command sets the token validity to expire after five minutes. After the TTL expires, you must run nomad login
again to get another valid token.
Enable PKCE
PKCE is an extension to the Authorization Code flow to prevent CSRF and authorization code injection attacks. Beginning with Nomad v1.10.0, Nomad supports PKCE.
To take advantage of the additional security of PKCE, you must enable it in Keycloak.
- Open the Keycloak web UI. To fetch the address, run
nomad service info keycloak
. - In the left navigation, click Clients.
- Under Client ID, click
nomad-oidc
. - At the top, click Advanced.
- Scroll to Advanced settings.
- Find Proof Key for Code Exchange Code Challenge Method.
- Click the dropdown and then select S256.
- Scroll to the bottom of the page and then click Save.
Keycloak now ensures that any authentication attempt uses PKCE. A login attempt with nomad login
now returns an error.
Error performing login: Unexpected response code: 400
(invalid OIDC complete-auth request: 1 error occurred:
* missing code)
Configure Nomad to enable PKCE
Update the auth method configuration file to enable PKCE in Nomad. Add the highlighted section for OIDCEnablePKCE
to the file and then save it.
auth-method-keycloak.json
{
"OIDCDiscoveryURL": "http://_KEYCLOAK_ADDRESS/realms/Nomad",
"OIDCClientSecret": "_KEYCLOAK_CLIENT_SECRET",
"OIDCEnablePKCE": true,
"OIDCClientID": "nomad-oidc",
"BoundAudiences": ["nomad-oidc"],
"AllowedRedirectURIs": [
"http://localhost:4649/oidc/callback",
"https://localhost:4646/ui/settings/tokens"
],
"OIDCScopes": [
"openid",
"scope-for-nomad"
],
"ListClaimMappings": {
"kc-groups": "keycloak_groups"
},
"VerboseLogging": true
}
Update the auth method. Note that the output shows that PKCE is enabled.
$ nomad acl auth-method update -config @auth-method-keycloak.json keycloak
Name = keycloak
Type = OIDC
# ...
OIDC Discovery URL = http://192.168.1.201:8080/realms/Nomad
OIDC Client ID = nomad-oidc
OIDC Client Secret = redacted
OIDC Enable PKCE = true
OIDC Disable UserInfo = false
# ...
Log in with Nomad to confirm that the functionality works.
$ nomad login
Successfully logged in via OIDC and keycloak
# ...
Enable client assertions
A client assertion is a token provided by a client application to confirm the proof of the client's identity.
Keycloak refers to client assertions as signed JWT or signed JWT with client secret. Nomad supports both methods. Client assertions provide additional security and function as an alternative to using client secrets.
Nomad builds a JWT and signs it with a private key that the OIDC provider can verify with Nomad's accompanying public key. Nomad asserts that it is a valid OIDC client without sending any secret information over the network.
- Navigate to the Keycloak web UI.
- In the left navigation, click Clients.
- Under Client ID, click
nomad-oidc
. - At the top, click Credentials.
- Next to Client Authenticator, click the dropdown.
Take note of the available options: Signed Jwt, Client Id and Secret, X509 Certificate, and Signed Jwt with Client Secret.
You have set up the Client Id and Secret authenticator. In the next section, you can choose how you want to set up a client assertion:
- Signed JWT with the client secret
- Signed JWT with Nomad's built-in private key pair
- Signed JWT with a new private key pair
Signed JWT with the client secret
This authenticator method uses the client secret as a Hash-Based Message Authentication Code (HMAC) key to sign the client assertion JWT. Nomad sends the signed JWT to Keycloak instead of sending the secret itself over the network.
Keycloak has the same client secret, which it uses to verify the JWT signature. Keycloak knows that Nomad must have the client secret, otherwise it would not have been able to use it to sign the JWT.
To enable the Signed Jwt with Client Secret authenticator, complete the following steps:
- Open the Keycloak web UI.
- In the left navigation, click Clients.
- Under Client ID, click
nomad-oidc
. - At the top, click Credentials.
- Next to Client Authenticator, click the dropdown and then select Signed Jwt with Client Secret.
- Next to Signature algorithm, click the dropdown and then select HS256.
- Click Save. In the dialog that appears, click Yes to confirm the change.
Attempting to log in with nomad login
will cause an error as the request from Nomad is missing the client_assertion_type
.
Error performing login: Unexpected response code: 500
(failed to exchange token with provider: Provider.Exchange:
unable to exchange auth code with provider: oauth2:
"invalid_client" "Parameter client_assertion_type is missing")
Update the auth method configuration file to enable the client assertion in Nomad. Add the highlighted section for OIDCClientAssertion
to the file and then save it.
The "KeySource" = "client_secret"
instructs Nomad to use the OIDCClientSecret
as a client assertion HMAC instead of a normal client secret.
auth-method-keycloak.json
{
"OIDCDiscoveryURL": "http://_KEYCLOAK_ADDRESS/realms/Nomad",
"OIDCClientSecret": "_KEYCLOAK_CLIENT_SECRET",
"OIDCClientAssertion": {
"KeySource": "client_secret",
"Algorithm": "HS256"
},
"OIDCEnablePKCE": true,
"OIDCClientID": "nomad-oidc",
"BoundAudiences": ["nomad-oidc"],
"AllowedRedirectURIs": [
"http://localhost:4649/oidc/callback",
"https://localhost:4646/ui/settings/tokens"
],
"OIDCScopes": [
"openid",
"scope-for-nomad"
],
"ListClaimMappings": {
"kc-groups": "keycloak_groups"
},
"VerboseLogging": true
}
Update the auth method. Note that the output now includes the client assertion attributes.
$ nomad acl auth-method update -config @auth-method-keycloak.json keycloak
Name = keycloak
Type = OIDC
# ...
Auth Method Config
JWT Validation Public Keys = <none>
JWKS URL = <none>
OIDC Discovery URL = http://192.168.1.201:8080/realms/Nomad
OIDC Client ID = nomad-oidc
OIDC Client Secret = redacted
OIDC Client Assertion KeySource = client_secret
OIDC Client Assertion Algorithm = HS256
OIDC Client Assertion Audience = http://192.168.1.201:8080/realms/Nomad
OIDC Enable PKCE = true
OIDC Disable UserInfo = false
# ...
Log in with Nomad to confirm that the functionality works.
$ nomad login
Successfully logged in via OIDC and keycloak
Accessor ID = 001b3ac1-f238-3003-fc94-539fa824b9a9
Secret ID = redacted
Name = OIDC-keycloak
Type = client
# ...
Roles
ID Name
19c19cbf-dd6a-6084-bca4-5505cb9028b8 engineering-role
Navigate back to the Configure Keycloak for client assertions section if you want to try out the other client assertion options.
Next Steps
In this tutorial you learned how to configure Nomad and Keycloak to automatically grant permissions with Nomad's ACL system. Then you enabled the Keycloak provider to require PKCE and reconfigured Keycloak to use client assertions.
To continue your learning, check out these resources:
- Learn more about the JWT Auth Method in Nomad.
- Learn more about the OIDC Auth method in Nomad.
- Learn how to use Vault as an OIDC provider with Nomad.