Bootstrap and explore Consul's access control system

Consul uses Access Control Lists (ACLs) to secure access to the UI, API, CLI, service communications, and agent communications. Review the following tutorials for securing gossip and RPC communication.

At the core, ACLs operate by grouping rules into policies, then associating one or more policies with a token. The diagram below illustrates this grouping process.

To complete this tutorial, you should have an operational Consul 1.4+ datacenter. You can use a local agent, Vagrant deployment, or Docker containers.

Bootstrapping the ACL system is a multi-step process, this tutorial will cover all the necessary steps.

• Enable ACLs on all the servers.
• Create the bootstrap token.
• Create the agent policy.
• Create the agent token.
• Apply the new token to the servers.
• Enable ACLs on the clients and apply the agent token

At the end of this tutorial, there are also several additional and optional steps.

Enable ACLs on all Consul servers

The first step for bootstrapping the ACL system is to enable ACLs on the Consul servers in the agent configuration. In this example, you are configuring the default policy of "deny", which means the datacenter is in a mode to only allow explicitly named resources, and a down policy of "extend-cache", which means that the agents will ignore token TTLs during an outage.

acl {
  enabled        = true
  default_policy = "deny"
  down_policy    = "extend-cache"
}

{
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "extend-cache"
  }
}

The servers will need to be restarted to load the new configuration. Take care to restart the servers one at a time and ensure each server has joined and is operating correctly before restarting another.

If ACLs are enabled correctly, the leader's logs will contain the following warning and info messages.

[INFO] acl: Created the anonymous token
[INFO] consul: ACL bootstrap enabled
[INFO] agent: Synced node info
[WARN] agent: Coordinate update blocked by ACLs
[INFO] acl: initializing acls
[INFO] consul: Created ACL 'global-management' policy

If you do not receive the ACL bootstrap enabled, the anonymous token creation, and the global-management policy creation message in the logs, ACLs have not been properly enabled.

Note, now that you have enabled ACLs, you will need a token to complete any operation. You can't do anything else to the datacenter until you bootstrap and generate the first bootstrap token. For simplicity, you will use the bootstrap token created during the ACL bootstrap for the remainder of the tutorial.

Create the initial management token

Once ACLs have been enabled, you can bootstrap your first token, called the bootstrap token. The bootstrap token is a management token with unrestricted privileges. It will be shared with all the servers in the quorum, since it will be added to the state store.

$ consul acl bootstrap

AccessorID: edcaacda-b6d0-1954-5939-b5aceaca7c9a
SecretID: 4411f091-a4c9-48e6-0884-1fcb092da1c8
Description: Bootstrap Token (Global Management)
Local: false
Create Time: 2018-12-06 18:03:23.742699239 +0000 UTC
Policies:
00000000-0000-0000-0000-000000000001 - global-management

On the server where the bootstrap command was issued, the logs should contain the following log message.

[INFO] consul.acl: ACL bootstrap completed
[DEBUG] http: Request PUT /v1/acl/bootstrap (2.347965ms) from=127.0.0.1:40566

Since ACLs have been enabled, you will need to use a token to complete any additional operations. For example, even checking the member list will require a token.

$ consul members -token "4411f091-a4c9-48e6-0884-1fcb092da1c8"

Node  Address            Status  Type    Build   Protocol  DC   Partition  Segment
fox   172.20.20.10:8301  alive   server  1.11.2  2         dc1  default    <all>
bear  172.20.20.11:8301  alive   server  1.11.2  2         dc1  default    <all>
wolf  172.20.20.12:8301  alive   server  1.11.2  2         dc1  default    <all>

Note using the token on the command line with the -token flag is not recommended, instead you can set it as an environment variable once.

$ export CONSUL_HTTP_TOKEN=4411f091-a4c9-48e6-0884-1fcb092da1c8

The bootstrap token can also be used in the server configuration file as the initial_management token.

Note, the bootstrap token can only be created once, bootstrapping will be disabled after the bootstrap token was created. Once the ACL system is bootstrapped, ACL tokens can be managed through the ACL API.

Create an agent token policy

Before creating a new token, you will need to create its associated policy. A policy is a set of rules that can be used to specify granular privileges. To learn more about rules, read the ACL rule specification documentation.

First create a policy file, agent-policy.hcl, and add the HCL stanzas below. Note, policy files should not be added to the Consul agent configuration directory.

node_prefix "" {
   policy = "write"
}
service_prefix "" {
   policy = "read"
}

This policy will allow all nodes to be registered and accessed and any service to be read. Note, this simple policy is not recommended in production. It is best practice to create separate node policies and tokens for each node in the datacenter with an exact-match node rule.

You only need to create one policy and can do this on any of the servers. If you have not set the CONSUL_HTTP_TOKEN environment variable to the bootstrap token, refer to the previous step.

$ consul acl policy create -name "agent-token" -description "Agent Token Policy" -rules @agent-policy.hcl

ID:           5102b76c-6058-9fe7-82a4-315c353eb7f7
Name:         agent-policy
Description:  Agent Token Policy
Datacenters:
Rules:
node_prefix "" {
   policy = "write"
}

service_prefix "" {
   policy = "read"
}

The returned value is the newly-created policy that you can now use when creating our agent token.

Create a Consul agent token

Using the newly created policy, you can create an agent token. Again you can complete this process on any of the servers. For this tutorial, all agents will share the same token. Note, the SecretID is the token used to authenticate API and CLI commands.

$ consul acl token create -description "Agent Token" -policy-name "agent-token"

AccessorID:   499ab022-27f2-acb8-4e05-5a01fff3b1d1
SecretID:     da666809-98ca-0e94-a99c-893c4bf5f9eb
Description:  Agent Token
Local:        false
Create Time:  2018-10-19 14:23:40.816899 -0400 EDT
Policies:
   fcd68580-c566-2bd2-891f-336eadc02357 - agent-token

Add the agent token to all Consul servers

Your final step for configuring the servers is to assign the token to all of the Consul servers via the configuration file and reload the Consul service one last time.

Add the agent token to the configuration file.

primary_datacenter = "dc1"

acl {
  enabled        = true
  default_policy = "deny"
  down_policy    = "extend-cache"
  tokens {
    agent = "da666809-98ca-0e94-a99c-893c4bf5f9eb"
  }
}

{
  "primary_datacenter": "dc1",
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "extend-cache",
    "tokens": {
      "agent": "da666809-98ca-0e94-a99c-893c4bf5f9eb"
    }
  }
}

At this point the coordinate warning should stop being written to the server's logs, however, you should be able to verify that the node information remains in sync.

[DEBUG] agent: Node info in sync

It is important to ensure the servers are configured properly, before enable ACLs on the clients. This will reduce any duplicate work and troubleshooting, if there is a misconfiguration.

Ensure the ACL system is configured properly

Before configuring the clients, check that the servers are healthy and have an assigned TaggedAddresses. You can use the consul catalog command and ACL Troubleshooting tutorial to check for a TaggedAddresses.

Finally, if you encounter issues that are unresolvable, or misplace the bootstrap token, you can reset the ACL system by updating the index. Follow the reset procedure in the ACL Troubleshooting tutorial.

After resetting the ACL system you can start again at Step 2.

Enable ACLs on Consul clients

Since ACL enforcement also occurs on the Consul clients, you need to also restart them with a configuration file that enables ACLs. You can use the same ACL agent token that you created for the servers. The same token can be used because you did not specify any node or service prefixes.

Create an agent configuration file on the clients in the configuration directory with the follow options.

acl {
  enabled        = true
  default_policy = "deny"
  down_policy    = "extend-cache"

  tokens {
    agent = "da666809-98ca-0e94-a99c-893c4bf5f9eb"
  }
}

{
  "acl": {
    "enabled": true,
    "default_policy": "deny",
    "down_policy": "extend-cache",
    "tokens": {
      "agent": "da666809-98ca-0e94-a99c-893c4bf5f9eb"
    }
  }
}

To ensure the agents are configured correctly, you can again use the /catalog endpoint.

Additional ACL configuration

Now that the nodes have been configured to use ACLs, you can configure the CLI, UI, and nodes to use specific tokens. All of the following steps are optional examples. In your own environment you will likely need to create more fine grained policies.

Configure the anonymous token (Optional)

The anonymous token is created during the bootstrap process, consul acl bootstrap. It is implicitly used if no token is supplied. In this section you will update the existing token with a newly created policy.

At this point ACLs are bootstrapped with ACL agent tokens configured, but there are no other policies set up. Even basic operations like consul members will be restricted by the ACL default policy of "deny":

$ consul members

You will not receive an error, since the ACL has filtered the list based on privileges. Since you did not present a token, you are using the anonymous token. This token, by default, does not provide access to any Consul objects.

If you supply the token you created above, you will receive a listing of nodes. That token has write privileges to an empty node prefix, meaning it has access to all nodes:

$ CONSUL_HTTP_TOKEN=4411f091-a4c9-48e6-0884-1fcb092da1c8 consul members

Node  Address            Status  Type    Build   Protocol  DC   Partition  Segment
fox   172.20.20.10:8301  alive   server  1.11.2  2         dc1  default    <all>
bear  172.20.20.11:8301  alive   server  1.11.2  2         dc1  default    <all>
wolf  172.20.20.12:8301  alive   server  1.11.2  2         dc1  default    <all>

It is common in many environments to allow listing of all nodes, even without a token. The policies associated with the special anonymous token can be updated to configure Consul's behavior when no token is supplied. The anonymous token is managed like any other ACL token, except that anonymous is used for the ID. In this example, you will give the anonymous token read privileges for all nodes:

$ consul acl policy create -name 'list-all-nodes' -rules 'node_prefix "" { policy = "read" }'

ID:           e96d0a33-28b4-d0dd-9b3f-08301700ac72
Name:         list-all-nodes
Description:
Datacenters:
Rules:
node_prefix "" { policy = "read" }

$ consul acl token update -id 00000000-0000-0000-0000-000000000002 -policy-name list-all-nodes -description "Anonymous Token - Can List Nodes"

Token updated successfully.
AccessorID:   00000000-0000-0000-0000-000000000002
SecretID:     anonymous
Description:  Anonymous Token - Can List Nodes
Local:        false
Create Time:  0001-01-01 00:00:00 +0000 UTC
Hash:         ee4638968d9061647ac8c3c99e9d37bfdd2af4d1eaa07a7b5f80af0389460948
Create Index: 5
Modify Index: 38
Policies:
   e96d0a33-28b4-d0dd-9b3f-08301700ac72 - list-all-nodes

The anonymous token is implicitly used if no token is supplied, so now you can run consul members without supplying a token and you will receive the list of nodes:

$ consul members

Node  Address            Status  Type    Build   Protocol  DC   Partition  Segment
fox   172.20.20.10:8301  alive   server  1.11.2  2         dc1  default    <all>
bear  172.20.20.11:8301  alive   server  1.11.2  2         dc1  default    <all>
wolf  172.20.20.12:8301  alive   server  1.11.2  2         dc1  default    <all>

The anonymous token is also used for DNS lookups since there is no way to pass a token as part of a DNS request. Here's an example lookup for the "consul" service:

$ dig @127.0.0.1 -p 8600 consul.service.consul

; <<>> DiG 9.8.3-P1 <<>> @127.0.0.1 -p 8600 consul.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 9648
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;consul.service.consul.         IN      A

;; AUTHORITY SECTION:
consul.                 0       IN      SOA     ns.consul. postmaster.consul. 1499584110 3600 600 86400 0

Now you get an NXDOMAIN error because the anonymous token doesn't have access to the "consul" service. Update the anonymous token's policy to allow for service reads of the "consul" service.

$ consul acl policy create -name 'service-consul-read' -rules 'service "consul" { policy = "read" }'

ID:           3c93f536-5748-2163-bb66-088d517273ba
Name:         service-consul-read
Description:
Datacenters:
Rules:
service "consul" { policy = "read" }

$ consul acl token update -id 00000000-0000-0000-0000-000000000002 --merge-policies -description "Anonymous Token - Can List Nodes" -policy-name service-consul-read

Token updated successfully.
AccessorID:   00000000-0000-0000-0000-000000000002
SecretID:     anonymous
Description:  Anonymous Token - Can List Nodes
Local:        false
Create Time:  0001-01-01 00:00:00 +0000 UTC
Hash:         2c641c4f73158ef6d62f6467c68d751fccd4db9df99b235373e25934f9bbd939
Create Index: 5
Modify Index: 43
Policies:
   e96d0a33-28b4-d0dd-9b3f-08301700ac72 - list-all-nodes
   3c93f536-5748-2163-bb66-088d517273ba - service-consul-read

With that new policy in place, the DNS lookup will succeed:

$ dig @127.0.0.1 -p 8600 consul.service.consul

; <<>> DiG 9.8.3-P1 <<>> @127.0.0.1 -p 8600 consul.service.consul
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46006
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;consul.service.consul.         IN      A

;; ANSWER SECTION:
consul.service.consul.  0       IN      A       127.0.0.1

The next section shows an alternative to the anonymous token.

Set agent-specific default tokens (optional)

An alternative to the anonymous token is the acl.tokens.default configuration item. When a request is made to a particular Consul agent and no token is supplied, the acl.tokens.default will be used for the token, instead of being left empty which would normally invoke the anonymous token.

This behaves very similarly to the anonymous token, but can be configured differently on each agent, if desired. For example, this allows more fine grained control of what DNS requests a given agent can service or can give the agent read access to some key-value store prefixes by default.

If using acl.tokens.default, then it's likely the anonymous token will have a more restrictive policy than shown in these examples.

Create tokens for the UI (Optional)

If you utilize the Consul UI with a restrictive ACL policy, as above, the UI will not function fully using the anonymous ACL token. It is recommended that a UI-specific ACL token is used, which can be set in the UI during the web browser session to authenticate the interface.

First create the new policy.

$ consul acl policy create -name "ui-policy" \
  -description "Necessary permissions for UI functionality" \
  -rules 'key_prefix"" { policy = "write" } node_prefix "" { policy = "read" } service_prefix "" { policy = "read" }'

ID:           9cb99b2b-3c20-81d4-a7c0-9ffdc2fbf08a
Name:         ui-policy
Description:  Necessary permissions for UI functionality
Datacenters:
Rules:
key_prefix "" { policy = "write" } node_prefix "" { policy = "read" } service_prefix "" { policy = "read" }

With the new policy, create a token.

$ consul acl token create -description "UI Token" -policy-name "ui-policy"

AccessorID:   56e605cf-a6f9-5f9d-5c08-a0e1323cf016
SecretID:     117842b6-6208-446a-0d1e-daf93854857d
Description:  UI Token
Local:        false
Create Time:  2018-10-19 14:55:44.254063 -0400 EDT
Policies:
   9cb99b2b-3c20-81d4-a7c0-9ffdc2fbf08a - ui-policy

The token can then be set on the "ACL" page of the UI.

Note, in this example, you have also given full write access to the KV through the UI.

Next steps

The ACL API can be used to create tokens for applications specific to their intended use and to create more specific ACL agent tokens for each agent's expected role. Now that you have bootstrapped ACLs, learn more about ACL rules

Notes on security

In this tutorial you configured a basic ACL environment with the ability to read all nodes by default, but with limited access to discover only the "consul" service.

Consider reading the ACL System documentation for information about tokens, policies, and more.

If your environment has stricter security requirements, note the following and consider these additional recommendations.

1. In this tutorial you added the agent token to the configuration file. This means the tokens are now saved on disk. If this is a security concern, tokens can be added to agents using the Consul CLI. However, this process is more complicated and takes additional care. To learn how to set tokens on with the Consul CLI refer to the Securing Consul with ACLs tutorial

2. It is recommended that each client get an ACL agent token with only node write privileges for its own node name, and service read privileges for the service prefixes expected to be registered on that client.

3. Anti-entropy syncing requires the ACL agent token to have service:write privileges for all services that may be registered with the agent. You should provide service:write for each separate service via a separate token that is used when registering via the API, or provided along with the registration in the configuration file. Note that service:write is the privilege required to assume the identity of a service and so Consul service mesh intentions are only enforceable to the extent that each service instance is unable to gain service:write on any other service name. For more details consult the Connect security documentation.