Application Leader Election with Sessions
For some applications, like HDFS, it is necessary to set one instance as a leader. This ensures the application data is current and stable.
This tutorial describes how to build client-side leader elections for service instances, using Consul. Consul's support for sessions allows you to build a system that can gracefully handle failures.
This tutorial is not related to Consul's leader election. If you are interested in the leader election used internally by Consul, please refer to the consensus protocol documentation instead.
Tip
The content of this tutorial also applies to Consul clusters hosted on HashiCorp Cloud (HCP).
Contending service instances
0 Imagine you have a set of service instances who are attempting to acquire leadership for a given service. All service instances that are participating should agree on a given key to coordinate. A good pattern is simply:
For this tutorial, our full key could be service/dbservice/leader
. However, for
clarity, our key will be lead
.
Create a session
The first step is to create a session using the Session HTTP API.
This will return a JSON object containing the session ID:
Acquire a session
The next step is to acquire a session for a given key from this instance
using the PUT method on a KV entry with the
?acquire=<session>
query parameter.
The <body>
of the PUT should be a
JSON object representing the local instance. This value is opaque to
Consul, but it should contain whatever information clients require to
communicate with your application (e.g., it could be a JSON object
that contains the node's name and the application's port).
This will either return true
or false
. If true
, the lock has been acquired and
the local service instance is now the leader. If false
is returned, some other node has acquired
the lock.
Watch the session
All instances now remain in an idle waiting state. In this state, they watch for changes
on key lead
. This is because the lock may be released or the instance could fail, etc.
The leader must also watch for changes since its lock may be released by an operator or automatically released due to a false positive in the failure detector.
By default, the session makes use of only the gossip failure detector. That is, the session is considered held by a node as long as the default Serf health check has not declared the node unhealthy. Additional checks can be specified if desired.
Watching for changes is done via a blocking query against <key>
. If they ever
notice that the Session
of the <key>
is blank, there is no leader, and then should
retry lock acquisition. Each attempt to acquire the key should be separated by a timed
wait. This is because Consul may be enforcing a lock-delay
.
Release the session
If the leader ever wishes to step down voluntarily, this should be done by simply releasing the lock:
Discover the leader
Another common practice regarding leader election is for non-leader instances may wish to identify the leader for the given set of service instances.
As with leader election, all instances that are participating should agree on the key
being used to coordinate. In this example, the name of the key is lead
.
Retrieve the key
Instances have a very simple role, they read the lead
key to discover the current leader. If the key has an associated Session
, then there is a leader.
If there is a leader then the value of the key will provide all the
application-dependent information required as a Base64 encoded blob in
the Value
field.
Retrieve session information
You can query the
/v1/session/info
endpoint to get details about the session
Next steps
In this tutorial you used a session to initiate manual leader election for a
set of service instances. To fully benefit from this process, instances should also watch the key using a blocking query for any
changes. If the leader steps down or fails, the Session
associated
with the key will be cleared. When a new leader is elected, the key
value will also be updated.
Using the acquire
parameter is optional. This means
that if you use leader election to update a key, you must not update the key
without the acquire parameter.