Gossip

This page provides conceptual information about the Serf gossip protocol and its implementation in Consul.

For more information about the underlying architecture, refer to control plane architecture.

Introduction

Serf is a gossip protocol that Consul implements in datacenter operations. Consul uses it to manage membership and broadcast messages to the cluster. There are two types of gossip pools available to Consul:

• The LAN gossip pool is enabled by default. It communications with all nodes in a single datacenter to share membership information. Consul uses the LAN gossip pool to automatically discover servers and distribute failure detection throughout the cluster. It also supports fast and reliable event broadcasts. The LAN gossip pool is required, and operates on port 8301 by default.
• The WAN gossip pool extends agent gossip to operate between a primary datacenter and one or more secondary datacenters in a WAN-federated environment. Membership information provided by the WAN pool allows servers to perform cross-datacenter requests. The WAN gossip pool operates on port 8302 by default.

Gossip encryption key

Communication between agents in the gossip pools is secured by a dedicated gossip encryption key. This encryption key is one layer of security in Consul's security model.

We recommend that you rotate the gossip encryption key periodically. Consul includes a built-in tool to automatically rotate gossip keys that is available through the consul keyring CLI command and the /operator/keyring API endpoint.

Lifeguard enhancements

The Serf protocol operates on the assumption that the local node is healthy, which enables packet processing in real time. If the local node experiences CPU or network exhaustion, the serfHealth check status can appear, resulting in false monitoring alarms, additional telemetry noise, and wasted resources on attempts to diagnose non-existent failures.

Lifeguard resolves this issue. For more details about Lifeguard, refer to the Making Gossip More Robust with Lifeguard blog post, which provides a high level overview of the HashiCorp Research paper Lifeguard : SWIM-ing with Situational Awareness. The Serf gossip protocol guide also provides details about the gossip protocol and Lifeguard.

Additional resources

For more information about the Serf protocol and related systems, refer to the following resources:

• Serf library on GitHub
• SWIM (Scalable Weakly-consistent Infection-style Process Group Membership)
• Serf documentation