Set a global limit on traffic rates

This topic describes how to configure rate limits for RPC and gRPC traffic to the Consul server.

Introduction

Rate limits apply to each Consul server separately and are intended to limit the number of read requests or write requests to the server on the RPC and internal gRPC endpoints.

Because all requests coming to a Consul server eventually perform an RPC or an internal gRPC request, these limits also apply to other user interfaces, such as the HTTP API interface, the CLI, and the external gRPC endpoint for services in the Consul service mesh.

Refer to Initialize Rate Limit Settings for additional information about right-sizing your gRPC request configurations.

Set a global rate limit for a Consul server

Configure the following settings in your Consul server configuration to limit the RPC and gRPC traffic rates.

Set the rate limiter mode
Set the read_rate
Set the write_rate

In the following example, the Consul server is configured to prevent more than 500 read and 200 write RPC calls:

Consul server agent

limits = {
    rate_limit = {
        mode = "enforcing"
        read_rate = 500
        write_rate = 200
    }
}

Access rate limit logs

Consul prints a log line for each rate limit request. The log provides the information necessary for identifying the source of the request and the configured limit. Consul prints the log DEBUG log level and can drop the log to avoid affecting the server health. Dropping a log line increments the rpc.rate_limit.log_dropped metric.

The following example log shows that RPC request from 127.0.0.1:53562 to KVS.Apply exceeded the limit:

2023-02-17T10:01:15.565-0500 [DEBUG] agent.server.rpc-rate-limit: RPC 
exceeded allowed rate limit: rpc=KVS.Apply source_addr=127.0.0.1:53562 
limit_type=global/write limit_enforced=false

Review rate limit metrics

Consul captures the following metrics associated with rate limits:

Type of limit
Operation
Rate limit mode

Call the agent/metrics API endpoint to view the metrics associated with rate limits. Refer to View Metrics for API usage information. In the following example, Consul dropped a call to the consul service because it exceeded the limit by one call:

$ curl http://127.0.0.1:8500/v1/agent/metrics
{
  . . .  
  "Counters": [
    {
      "Name": "consul.rpc.rate_limit.exceeded",
      "Count": 1,
      "Sum": 1,
      "Min": 1,
      "Max": 1,
      "Mean": 1,
      "Stddev": 0,
      "Labels": {
        "service": "consul"
      }
    },
    {
      "Name": "consul.rpc.rate_limit.log_dropped",
      "Count": 1,
      "Sum": 1,
      "Min": 1,
      "Max": 1,
      "Mean": 1,
      "Stddev": 0,
      "Labels": {}
    }
  ],
  . . .

Refer to Telemetry for additional information.