Troubleshoot the Access Control System
Consul provides a robust set of APIs that you can use to check the health of your datacenter. In this tutorial, you will learn about several Consul CLI commands that you can use to troubleshoot issues with tokens and policies. Additionally, you will learn about the ACL system reset procedure that can be used in case of an emergency.
Prerequisites
This tutorial assumes an existing datacenter running Consul 1.4 or newer. Access to a machine where you can execute Consul CLI commands, either a Consul agent or a local binary configured to control the remote datacenter.
All commands in this tutorial will need a valid token. For ease, we recommend using the initial management token since it has unlimited privileges.
Consul members
You can use the consul members
command
when configuring agents with tokens to check that they have the necessary
privileges to join the datacenter.
If an agent, server or client, is missing from the members list then ACLs have not been configured correctly on that agent or the token does not have the correct privileges. Only agents that have privileges to register themselves in the catalog will be included in the member list.
Use the consul acl
commands listed in the following sections to help
troubleshoot token privileges.
Consul catalog
The consul catalog nodes -detailed
command
will display node information, including "TaggedAddresses". If
"TaggedAddresses" is null for any of the agents, that agent’s ACLs are not
configured correctly. You can start debugging by reviewing the Consul logs on
all the servers. If ACLs are enabled correctly, you can investigate the agent's
token.
Use the consul acl
commands listed below to help troubleshoot troubleshoot
token privileges.
Consul ACL
Once you've confirmed the issue is not due to misconfiguration, you can use the following commands to help troubleshoot token or policy issues.
Consul ACL policy list
The consul acl policy list
command
will output all the available policies. When you're first setting up tokens you
should use it to ensure the list of policies and their rules are as expected. In
the example output below, there are two policies; the Consul created
global-management
policy and a user created policy named server-one-policy
.
You can use consul acl policy read -id <policy_id>
to investigate individual
policies. In the example output below, the server-one-policy
policy has node
write privileges for node "consul-server-one".
Consul ACL token list
The consul acl token list
command
will list all the tokens. Ensure this list only includes the tokens in use. It
is important for the security of your datacenter and you should check it often.
Since tokens do not expire, it is up to the operator to
delete tokens
that are not in use.
In the example output below, there are three tokens. The first token is created
by Consul during the bootstrap process, called bootstrap token. The second token
is user generated and was created with the server-one-policy
policy. The third
token is also created by Consul, but it has no privileges.
The consul acl token read
command
will provide information about the token specified. Ensure the privileges of the
token are expected. This is useful when checking a node or service has the
correct privileges to add itself to the catalog.
In the example output below includes the same information that is returned with
the consul acl token list
command, but is useful when you do not want to view
all tokens.
Use the consul acl token read
command first if the consul catalog
or
consul members
commands return unexpected results.
Reset the ACL system
If you encounter issues that are unresolvable, or misplace the bootstrap token,
you can reset the ACL system by updating the index. First, find the leader by
curl
'ing the /v1/status/leader
endpoint on any node. ACL reset must be
performed on the leader.
In this example, you can verify that the leader is at IP 172.17.0.3
. The
following commands need to be run on that server.
Re-run the bootstrap command to get the index number.
Then write the reset index into the bootstrap reset file: (here the reset index is 13)
After resetting the ACL system, you can initialize it again and recreate the bootstrap token.
Next steps
In this tutorial, you learned how to use the Consul CLI to troubleshoot the ACL system. Each command has a corresponding API endpoint. You also learned how to reset the the ACL system encase of emergency. Note, resetting the ACL system will invalidate all tokens generated before the reset.
To learn ACL policy management best practices, review Manage ACL Policies.