Feature gates

Consul feature gates are datacenter-wide switches, similar to feature flags, that turn experimental Consul functionality on or off without restarting agents or editing configuration files. This topic describes how feature gates work, how to enable and disable them, and what to consider when you upgrade.

How feature gates work

Consul ships each gated feature in the binary but leaves it disabled until you turn it on. You can change a gate while the datacenter is running.

Feature gates provide the following benefits:

• Safe defaults. Every feature gate is disabled by default. Upgrading Consul does not change the behavior of your datacenter until you enable a feature.
• Consistent behavior. Consul stores gate settings in the Raft log, so all servers and agents in the datacenter agree on whether a feature is on.
• Protection during upgrades. A gate cannot take effect until every server in the datacenter runs a version that supports the feature.

Feature gates apply to a single datacenter. If you operate multiple datacenters or cluster peers, enable the feature in each datacenter that needs it.

Requirements

• Consul v2.1.0 or later.
• An access control list (ACL) token with operator:read permissions to read feature gates, or operator:write permissions to change them. Refer to Operator rules for more information.
• Client agent tokens with operator:read permissions. Client agents watch the gate state so that they can configure proxies consistently.

Available feature gates

Consul registers the following feature gates. All of them are experimental and disabled by default.

Run the consul operator feature list command to display the gates that your servers register.

Feature gate states

Consul tracks three values for each feature gate:

• Desired is the state that the most recent configuration request set. If no one configured the gate, the desired state is the gate's default, which is disabled for every gate.
• Eligible indicates whether every server in the datacenter runs at least the gate's minimum version.
• Effective is the state that Consul applies. A gate is effective only when it is desired and eligible.

The Source and Reason fields explain how Consul reached the effective state.

The Source field reports one of the following values:

The Reason field reports one of the following values:

Operator settings replace bootstrap settings, and both replace the gate's default.

The Consul leader re-evaluates every gate every 10 seconds and when an operator changes a setting. When you upgrade the last server in a datacenter, the affected gates become eligible without any further action.

Enable or disable a feature gate

You can manage feature gates with the CLI or the HTTP API.

View feature gates

Run the consul operator feature list command to view all feature gates.

$ consul operator feature list

To view a single gate, run the consul operator feature get command and specify the gate's name.

$ consul operator feature get peering-multiport-upstreams

The corresponding HTTP API endpoints are /v1/operator/features and /v1/operator/feature/:name.

Enable a feature gate

Run the consul operator feature set command to enable a feature.

$ consul operator feature set peering-multiport-upstreams enabled

After the command returns, check the Effective column in the output. If it shows false, check the Reason column. A reason of below-minimum-version means that at least one server still runs a version earlier than the gate's minimum version. Upgrade the remaining servers. The gate then takes effect without further action.

You can also use the PUT /v1/operator/feature/:name HTTP API endpoint.

Disable a feature gate

Run the consul operator feature set command with the disabled value.

$ consul operator feature set peering-multiport-upstreams disabled

Consul withdraws the feature's configuration from the proxies in the datacenter. To learn how disabling a gate affects traffic, refer to the gate's entry in Available feature gates.

Coordinate changes from multiple operators

If several operators or automation tools can change feature gates, use the -cas option so that an update applies only when no one else has changed the policy since you read it. For more information, refer to Prevent conflicting updates.

Configure Vault access on Kubernetes

When the Helm feature-gate job reads an ACL token from Vault, set global.secretsBackend.vault.featureGateSetRole to a Vault Kubernetes auth role that permits access to the token. Bind the role to the feature-gate job's ServiceAccount and the Helm release namespace. The token must have operator:write permissions to change feature gates.

You can reuse the role configured in global.secretsBackend.vault.manageSystemACLsRole by adding the feature-gate ServiceAccount to its bindings, or create a dedicated role. Setting manageSystemACLsRole alone does not configure the feature-gate job's Vault role. Refer to Configure Vault access for the Consul feature-gate job for both procedures.

This role applies to the Helm job's access to Vault. You do not need it to configure feature gates directly through the CLI, HTTP API, or server feature_gates.bootstrap configuration.

Enable a feature gate when you create a datacenter

You can set the initial state of feature gates in the server agent configuration file. Use the feature_gates block when you bootstrap a new datacenter.

The following example enables the peering-multiport-upstreams feature gate when the leader creates the datacenter's first policy:

feature_gates {
  bootstrap = {
    "peering-multiport-upstreams" = true
  }
}

Consul uses this block only once, when the leader creates the first feature gate policy for the datacenter. After that, the stored policy is authoritative and Consul ignores changes to the block. If the block differs from the stored policy, Consul logs the following warning:

feature_gates.bootstrap diverges from committed Raft policy; local config ignored

To change a gate in a running datacenter, use the CLI or the HTTP API.

The block is valid only on server agents. Consul does not start if the block appears on a client agent or names a gate that does not exist. In these cases, the configuration error contains feature_gates.bootstrap requires server = true or feature_gates.bootstrap contains unknown feature "<name>". For more information, refer to the feature_gates parameter.

Upgrade considerations

Follow these guidelines when you upgrade a datacenter that uses feature gates:

1. Upgrade all servers in the datacenter before you enable a feature. Consul keeps a gate disabled until every server meets its minimum version.
2. Upgrade client agents, mesh gateways, and proxies to the same version as the servers before you enable a feature that changes proxy behavior. Agents on earlier versions do not recognize the gate. For additional requirements, refer to the gate's entry in Available feature gates.
3. Enable the gate in each datacenter that needs it.

After every server in the datacenter runs a version that supports feature gates, the leader creates the feature gate policy. Until then, consul operator feature list returns no gates and the HTTP API returns an empty list with the X-Consul-Feature-Gates-Uninitialized header. A request for a single gate returns a 503 status code. Retry the request after the leader finishes its first reconciliation.

Next steps

• Refer to consul operator feature for the CLI reference.
• Refer to the feature gate HTTP API for the API reference.
• Refer to Multiport services across cluster peers for an example of a feature that uses a gate.