Troubleshoot lease problems

Explanations, workarounds, and solutions for common lease problems in Vault.

429 - Too Many Requests

Problem

Vault returns a 429 - Too Many Requests response when users try to authenticate. For example:

Error making API request.

URL: PUT https://127.0.0.1:61555/v1/auth/userpass/login/foo
Code: 429. Errors:

* 1 error occurred:
    * request path "auth/userpass/login/foo": lease count quota exceeded

Cause

Vault returns a 429 - Too Many Requests response if a new lease request violates the configured lease quota limit.

To guard against lease explosions, Vault rejects authentication requests if completing the request would violate the configured lease quota limit.

Solution

1. Correct any client-side errors that may cause excessive lease creation.
2. Determine if your resource needs have changed and complete the Protecting Vault with Resource Quotas tutorial to determine new, appropriate defaults.
3. Use the vault lease CLI command or lease count quota endpoint to tune your lease count quota.

Lease explosion (degraded performance)

Problem

Your Vault nodes are out of memory and unresponsive to new lease requests.

Cause

Clients have caused a lease explosion with consistent, high-volume API requests.

Solution

To resolve a lease explosion, you need to mitigate the problem to stabilize Vault and provide space for cluster recovery then clean up your Vault environment.

1. Mitigate resource stress by adjusting TTL values for your Vault instance:

   Config level	Parameter	Precedence
   Database plugin	ttl or default_ttl	first
   Database plugin	max_ttl	first
   AuthN/secrets plugin	ttl or default_ttl	second
   AuthN/secrets plugin	max_ttl	second
   Vault	default_lease_ttl	last
   Vault	max_lease_ttl	last

   Granular TTLs on a role, group, or user level always override plugin and system-wide TTL values.

2. Use firewalls or load balancers to limit API calls to Vault from aberrant clients and reduce load on the struggling cluster .

3. Once the cluster stabilizes, check the active node to determine if you can wait for it to purge leases automatically or if you need to speed up the process by manually revoking leases.

4. If the cluster requires manual intervention, confirm you have a recent, valid snapshots of the cluster.

5. Once you confirm a valid snapshot of the cluster exists, use vault lease revoke to manually revoke the offending leases.

Related tutorials

• Troubleshoot irrevocable leases