Protecting Vault with Resource Quotas

23min
|
Vault
Interactive

Vault is an API driven system that all communication between the clients and Vault are done through Vault API.

As the number of client applications increases, rogue applications can degrade Vault performance. The issue is often caused by applications that are:

Generating an unbounded number of leases and/or loading too much data into Vault leading to exhausting Consul storage backend's available memory
Consuming an erroneously large amount of bandwidth leading to internal throttling of the Vault cluster as a whole

Solution

Set Vault resource quotas to protect your Vault environment's stability and network, as well as storage resource consumption from runaway application behavior and distributed denial of service (DDoS) attack.

The Vault operators can control how applications request resources from Vault, and Vault's storage and network infrastructure by setting the following:

Feature	Description	Vault OSS	Vault Enterprise
Rate Limit Quotas	Limit maximum amount of requests per second (RPS) to a system or mount to protect network bandwidth	✔️	✔️
Lease Count Quotas	Cap number of leases generated in a system or mount to protect system stability and storage performance at scale		✔️

Note

Lease Count Quotas requires Vault Enterprise Standard license.

To set the rate limit quotas and lease count quotas, use the sys/quotas/<type> endpoint. Each resource quota has a name to identify the quota rule. To manage an individual quota rule, the endpoint becomes sys/quotas/<type>/<name>.

The <type> can be:

rate-limit: rate limit quota
lease-count: lease count quota

Note

Batch tokens do not count towards the lease count quota. Creation of new batch tokens will, however, be blocked if the lease count quota is exceeded.

Note

Feature enhancements were introduced to both rate limit quota and lease count quota in Vault 1.12.0. Refer to each section for more details.

Prerequisites

To perform the tasks described in this tutorial, you need Vault v1.5 or later. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Lab setup

Open a terminal and start a Vault dev server with root as the root token.
```
$ vault server -dev -dev-root-token-id root
```
The Vault dev server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.
Insecure operation
Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.
Export an environment variable for the vault CLI to address the Vault server.
```
$ export VAULT_ADDR=http://127.0.0.1:8200
```
Export an environment variable for the vault CLI to authenticate with the Vault server.
```
$ export VAULT_TOKEN=root
```

The Vault server is ready.

Resource quota configuration

To configure the resource quota, use the sys/quotas/config endpoint.

Parameter	Type	Description
`enable_rate_limit_audit_logging`	boolean	Enable or disable audit logging when requests get rejected due to rate limit quota violations. By default, audit logging is disabled (`false`).

By default, the requests rejected due to rate limit quota violations are not written to the audit log. Therefore, if you wish to log the rejected requests for traceability, you must set the enable_rate_limit_audit_logging to true. The requests rejected due to reaching the lease count quotas are always logged that you do not need to set any parameter.

Note

Enabling the rate limit audit logging may have an impact on the Vault performance if the volume of rejected requests is large.

Enable a file audit device which outputs to /var/log/vault-audit.log (or your desired file location).
```
$ vault audit enable file file_path="/var/log/vault-audit.log"
```
To enable the audit logging for rate limit quotas, execute the following command.
```
$ vault write sys/quotas/config enable_rate_limit_audit_logging=true
```

Read the quota configuration to verify.

$ vault read sys/quotas/config

Key                                   Value
---                                   -----
enable_rate_limit_audit_logging       true
enable_rate_limit_response_headers    false
rate_limit_exempt_paths               []

Enable file audit device.

First, create the HTTP request payload specifying the audit log path to be /var/log/vault-audit.log (or your desired file location).

$ tee audit-payload.json <<EOF
{
  "type": "file",
  "options": {
      "file_path": "/var/log/vault-audit.log"
  }
}
EOF

Use the sys/audit endpoint to enable file audit log.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @audit-payload.json \
    $VAULT_ADDR/v1/sys/audit/file

Set the target file_path to your desired location.

Create the HTTP request payload to enable the rate limit quota audit logging.

$ tee payload.json <<EOF
{
   "enable_rate_limit_audit_logging": true
}
EOF

Invoke the sys/quotas/config endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/sys/quotas/config

Read the quota configuration to verify.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
    $VAULT_ADDR/v1/sys/quotas/config | jq -r ".data"

Example output:

{
  "enable_rate_limit_audit_logging": true,
  "enable_rate_limit_response_headers": false,
  "rate_limit_exempt_paths": []
}

Rate limit quotas

Rate limit quotas are designed to protect Vault against external distributed denial of service (DDoS) attacks and are fundamental to Vault's security model. Therefore, it is a part of Vault's core feature set available in both OSS and Enterprise.

To set rate limit quotas, use the sys/quotas/rate-limit/<name> endpoint.

Parameters

Parameter	Type	Description
`name`	string	Name of the quota rule
`path`	string	Target path or namespace to apply the quota rule. A blank path configures a global rate limit quota
`rate`	float	Rate for the number of allowed requests per second (RPS)
`role`	string	Vault 1.12 or later: Login role to apply this quota to. When this parameter is set, the path must be configured to a valid auth method with a concept of roles.
`interval`	second	The duration to enforce rate limiting for (default is 1 second)
`block_interval`	string	If set, when a client reaches a rate limit threshold, the client will be prohibited from any further requests until after the 'block_interval' has elapsed.

Note

If you are running Vault 1.12 or later, the path can be a fully qualified path, and it can end with * (e.g., auth/token/create*).

Create a rate limit quota named, "global-rate" which limits inbound workload to 500 requests per seconds.

$ vault write sys/quotas/rate-limit/global-rate rate=500
Success! Data written to: sys/quotas/rate-limit/global-rate

Read the global-rate rule to verify its configuration.

$ vault read sys/quotas/rate-limit/global-rate

Key               Value
---               -----
block_interval    0
interval          1
name              global-rate
path              n/a
rate              500
type              rate-limit

Note

In absence of path, this quota rule applies to the global level instead of a specific mount or namespace.

Create a rate limit quota named, "transit-limit" which limits the access to the Transit secrets engine to be 1000 requests per minute (60 seconds).

First, enable Transit secrets engine at transit.

$ vault secrets enable transit
Success! Enabled the pki secrets engine at: transit/

Now, create a rate limit quota.

$ vault write sys/quotas/rate-limit/transit-limit path="transit" rate=1000 interval=60
Success! Data written to: sys/quotas/rate-limit/transit-limit

Read the transit-limit rule to verify its configuration.

$ vault read sys/quotas/rate-limit/transit-limit

Key               Value
---               -----
block_interval    0
interval          60
name              transit-limit
path              transit/
rate              1000
role              n/a
type              rate-limit

Vault 1.12.0 or later

Previously, the path depth had to be the mount of an auth method or secrets engine. If you are running Vault 1.12 or later, you can set the path to be deeper than the mount point (in this example, transit/).

Create a rate limit quota named, "transit-order" to limit the data encryption requests using orders key to be 500 per second.

First, create an encryption key named, "orders".

$ vault write -f transit/keys/orders
Success! Data written to: transit/keys/orders

Now, create the "transit-order" rate limit quota.

$ vault write sys/quotas/rate-limit/transit-order path="transit/encrypt/orders" rate=500
Success! Data written to: sys/quotas/rate-limit/transit-order

Verify the rate limit quota configuration.

$ vault read sys/quotas/rate-limit/transit-order

Output:

Key               Value
---               -----
block_interval    0
interval          1
name              transit-order
path              transit/encrypt/orders
rate              500
role              n/a
type              rate-limit

Test

Be sure to visit the Test to understand the resource quota section to see how resource quotas enforcement works.

Vault Enterprise Example

Consider that you have K/V v2 secrets engine enabled at kv-v2 under us-west namespace. Create a rate limit quota named, "orders" which limits the incoming requests against kv-v2 to 20 requests per second maximum.

Create a namespace, "us-west".
```
$ vault namespace create us-west
```
Enable kv-v2 secrets engine in the us-west namespace.
```
$ vault secrets enable -ns="us-west" kv-v2
```

Now, create the orders quota rule.

$ vault write sys/quotas/rate-limit/orders \
     path="us-west/kv-v2" \
     rate=20

Verify the configuration.

$ vault read sys/quotas/rate-limit/orders

Key               Value
---               -----
block_interval    0
interval          1
name              orders
path              us-west/kv-v2/
rate              20
role              n/a
type              rate-limit

Create a rate limit quota named, "global-rate" which limits inbound workload to 500 requests per second.

Finally, invoke the sys/quotas/rate-limit endpoint.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{ "rate": 500 }' \
    $VAULT_ADDR/v1/sys/quotas/rate-limit/global-rate

Read the newly created global-rate quota rule.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
    $VAULT_ADDR/v1/sys/quotas/rate-limit/global-rate | jq -r ".data"

Output:

{
  "block_interval": 0,
  "interval": 1,
  "name": "global-rate",
  "path": "",
  "rate": 500,
  "type": "rate-limit"
}

Note

In absence of path, this quota rule applies to the root namespace instead of a specific mount or namespace.

Create a rate limit quota named, "transit-limit" which limits the access to the Transit secrets engine to be 1000 requests per minute (60 seconds).

First, enable Transit secrets engine at transit.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data '{"type":"transit"}' \
    $VAULT_ADDR/v1/sys/mounts/transit

Now, create a "transit-limit" rate limit quota.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data '{"path": "transit", "rate": 1000, "interval": 60 }' \
   $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-limit

Read the transit-limit rule to verify its configuration.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-limit | jq -r ".data"

Output:

{
   "block_interval": 0,
   "interval": 60,
   "name": "transit-limit",
   "path": "transit/",
   "rate": 1000,
   "role": "",
   "type": "rate-limit"
}

Vault 1.12.0 or later

Create a rate limit quota named, "encrypt-orders" to limit the data encryption requests using orders key to be 500 per second.

First, create an encryption key named, "orders".

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    $VAULT_ADDR/v1/transit/keys/orders

Now, create the "encrypt-orders" rate limit quota.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data '{ "path": "transit/encrypt/orders", "rate": 500 }' \
   $VAULT_ADDR/v1/sys/quotas/rate-limit/encrypt-orders

Verify the rate limit quota configuration.

$ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/sys/quotas/rate-limit/encrypt-orders | jq -r ".data"

Outpu:

{
   "block_interval": 0,
   "interval": 1,
   "name": "encrypt-orders",
   "path": "transit/encrypt/orders",
   "rate": 500,
   "role": "",
   "type": "rate-limit"
}

Test

Be sure to visit the Test to understand the resource quota section to see how resource quotas enforcement works.

Vault Enterprise Example

Create a namespace, "us-west".

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     $VAULT_ADDR/v1/sys/namespaces/us-west

Enable kv-v2 secrets engine in the us-west namespace.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --header "X-Vault-Namespace: us-west" \
    --request POST \
    --data '{"type": "kv-v2"}' \
    $VAULT_ADDR/v1/sys/mounts/kv-v2

Create the HTTP request payload defining parameters for "orders" quota rule.

$ tee payload.json <<EOF
{
   "path": "us-west/kv-v2",
   "rate": 20
}
EOF

Finally, invoke the sys/quota/rate-limit endpoint to create an "orders" quota rule.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
     --request POST \
     --data @payload.json \
     $VAULT_ADDR/v1/sys/quotas/rate-limit/orders

Lease count quotas

Note

Lease Count Quotas is a part of Vault Enterprise Platform features.

Lease count quota is designed to protect Vault from a large volume of leases and tokens persisted in Vault, which can pressure its storage backend. This acts as a guard rail for system stability in large-scale Vault Enterprise deployments where Vault is used as a service.

Parameters

Parameter	Type	Description
`name`	string	Name of the quota rule
`path`	string	Target path or namespace to apply the quota rule. A blank path configures a global lease count quota
`max_leases`	int	Maximum number of leases allowed by the quota rule
`role`	string	Vault 1.12 or later: Login role to apply this quota to. When this parameter is set, the path must be configured to a valid auth method with a concept of roles.

Note

If you are running Vault 1.12 or later, the path can be a fully qualified path, and it can end with * (e.g., auth/token/create*).

To demonstrate this feature, enable database secrets engine at "postgres" in the us-west namespace.
```
$ vault secrets enable -ns=us-west -path=postgres database
```
Create a lease count quota named, "db-creds" which limits the incoming requests for a new set of DB credentials to 100 concurrent, valid leases maximum.
```
$ vault write sys/quotas/lease-count/db-creds max_leases=100 \
   path="us-west/postgres"
```
Note
Similar to rate limit quotas, the quota rules apply globally in the absence of path. In the example, you created a quota rule against the us-west/postgres path. If a global quota rule exists on the root namespace, the quota rule defined on a specific path takes precedence.

Verify the configuration.

$ vault read sys/quotas/lease-count/db-creds

Key           Value
---           -----
counter       0
max_leases    100
name          db-creds
path          us-west/postgres/
role          n/a
type          lease-count

Vault 1.12.0 or later

Previously, you could only apply to the path of where the auth method was enabled. When you are running Vault 1.12 or later, you can specify the role if the target auth method has the concept of roles.

Create a role named "webapp" for approle auth method.

First, enable the approle auth method.

$ vault auth enable approle

Create "webapp" role.

$ vault write auth/approle/role/webapp \
   token_policies="webapp-policy" \
   token_ttl=4h \
   token_max_ttl=12h

Create a lease count quota named "webapp-tokens" which limits the creation of token for the webapp role to maximum of 100.
```
$ vault write sys/quotas/lease-count/webapp-tokens max_leases=100 \
   path="auth/approle" \
   role="webapp"
```
If you want to set it for the approle auth method enable in the us-west namespace, the path should be us-west/auth/approle.

Verify the configuration.

$ vault read sys/quotas/lease-count/webapp-tokens

Output:

Key           Value
---           -----
counter       0
max_leases    100
name          webapp-tokens
path          auth/approle/
role          webapp
type          lease-count

To demonstrate this feature, enable database secrets engine at "postgres" in the us-west namespace.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --header "X-Vault-Namespace: us-west" \
   --request POST \
   --data '{"type":"database"}' \
   $VAULT_ADDR/v1/sys/mounts/postgres

Create a lease count quota named, "db-creds" which limits the incoming requests for a new set of DB credentials to 100 concurrent, valid leases maximum.
First, create the HTTP request payload defining the parameters for db-creds quota rule.
```
$ tee payload.json <<EOF
{
   "max_leases": 100,
   "path": "us-west/postgres"
}
EOF
```
Now, invoke the sys/quotas/lease-count endpoint to create the quota rule.
```
$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/sys/quotas/lease-count/db-creds
```
Note
Similar to rate limit quotas, the quota rules apply globally in the absence of path. In the example, you created a quota rule against the us-west/postgres path. If a global quota rule exists on the root namespace, the quota rule defined on a specific path takes precedence.

Verify the configuration.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/sys/quotas/lease-count/db-creds | jq -r ".data"

Output:

{
   "counter": 0,
   "max_leases": 100,
   "name": "db-creds",
   "path": "us-west/postgres/",
   "role": "",
   "type": "lease-count"
}

Vault 1.12.0 or later

To demonstrate this feature, you are going to use the approle auth method. Enable the approle auth method.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data '{"type": "approle"}' \
   $VAULT_ADDR/v1/sys/auth/approle

Create a role named "webapp" for approle auth method.

First, create the API request payload.

$ tee payload.json <<EOF
{
   "token_policies": "webapp-policy",
   "token_ttl": "4h",
   "token_max_ttl": "12h"
}
EOF

Now, create a "webapp" role.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/auth/approle/role/webapp

Create a lease count quota named "webapp-tokens" which limits the creation of token for the webapp role to maximum of 100.
First, create the request payload.
```
$ tee payload_lcq.json <<EOF
{
   "max_leases": 100,
   "path": "auth/approle",
   "role": "webapp"
}
EOF
```
If you want to set it for the approle auth method enabled in the us-west namespace, the path should be us-west/auth/approle.
Now, invoke the sys/quotas/lease-count endpoint to create the quota rule.
```
$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data @payload_lcq.json \
   $VAULT_ADDR/v1/sys/quotas/lease-count/webapp-tokens
```

Verify the configuration.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/sys/quotas/lease-count/webapp-tokens | jq -r ".data"

Output:

{
   "counter": 0,
   "max_leases": 100,
   "name": "webapp-tokens",
   "path": "auth/approle/",
   "role": "webapp",
   "type": "lease-count"
}

Test

Be sure to visit the Test to understand the resource quota section to see how resource quotas enforcement works.

Test to understand the resource quotas

Now that you learned the basic commands, let's test to see how it works.

Rate limit quota test

Enable transit secrets engine if it is not enabled.
```
$ vault secrets enable transit
```
Create a "test" encryption key.
```
$ vault write -f transit/keys/test
```
For the purpose of demonstration, create a "test-transit" rate limit quota such that you can only make 1 requests every 10 seconds.
```
$ vault write sys/quotas/rate-limit/rate-test path=transit rate=1 interval=10
```

Create a shortcut script, test-encryption.sh which makes request to encrypt data using "test" key.

$ tee test-encryption.sh <<EOF
echo "Request 1"
vault write transit/encrypt/test plaintext=$(base64 <<< "4111 1111 1111 1111")
echo "\nRequest 2"
vault write transit/encrypt/test plaintext=$(base64 <<< "4222 2222 2222 2222")
echo "\nRequest 3"
vault write transit/encrypt/test plaintext=$(base64 <<< "4333 3333 3333 3333")
EOF

Ensure that the script is executable.
```
$ chmod +x test-encryption.sh
```

Run the script to see how the quota rule behaves.

$ ./test-encryption.sh

Output:

Request 1
Key            Value
---            -----
ciphertext     vault:v1:6zY2JVtuw3r11ST/JjMkmoNbbcCpQwGbJJ0BJfByk//GLQof8mcxAoerq5Y++BZK
key_version    1

Request 2
Error writing data to transit/encrypt/test: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/test
Code: 429. Errors:

* request path "transit/encrypt/test": rate limit quota exceeded

Request 3
Error writing data to transit/encrypt/test: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/test
Code: 429. Errors:

* request path "transit/encrypt/test": rate limit quota exceeded

The second and third requests failed because of the limit.

Earlier, you configured the resource quotas to enable audit logging of requests that were rejected due to rate limit quota rule violation. Inspect your audit log for its entry.

$ more /var/log/vault-audit.log | jq

If your audit log path is not /var/log/vault-audit.log, be sure to set it to the correct path.

...snip...
   "request": {
      "id": "4170a181-6a18-589b-f026-a62ad08ae644",
      "operation": "update",
      "namespace": {
         "id": "root"
      },
      "path": "transit/encrypt/test",
      "data": {
         "plaintext": "hmac-sha256:50978c7cb3f0f463491a26152353f485b579d1e8759ec7b7b7c28f3c5b9232cd"
      },
      "remote_address": "127.0.0.1",
      "remote_port": 61028
   },
   "error": "request path \"transit/encrypt/test\": rate limit quota exceeded"
}

You should find an error message indicating that rate limit quota was exceeded. You can trace the audit log to see how many requests were rejected due to the rate limit quota. It may be working as expected or you may find suspicious activities against a specific path.

If you are running Vault 1.12.0 or later, create another rate limit quota to specify the path to transit/encrypt/test where allowed rate is 2 with interval of 10 seconds.
```
$ vault write sys/quotas/rate-limit/encryption-limit \
   path="transit/encrypt/test" \
   rate=2 \
   interval=10
```

Run the script again to see how the quota rule behaves.

$ ./test-encryption.sh

Output:

Request 1
Key            Value
---            -----
ciphertext     vault:v1:44sOnzePg8c0XaxzGeGzwJ94UZ1uuif8p0nGuze8utbvWVhqNHDiIwo8VZmPrcli
key_version    1

Request 2
Key            Value
---            -----
ciphertext     vault:v1:w9tyXqoXH6uXuOWUktVWP/yrzOOZDv+TrRbbRU5GZEVappASIRnh3+6KwkJkjWKE
key_version    1

Request 3
Error writing data to transit/encrypt/test: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/test
Code: 429. Errors:

* request path "transit/encrypt/test": rate limit quota exceeded

Note

This time, only the last request failed. When a more granular path is set, the rate limit quota rule takes precedence against the path.

If you have another key or try to decript (transit/decrypt/test), the rate-test rate limit quota will be applied.

Lease count quota test

Similarly, create a very limited lease count quota named, "lease-test" which applies on the root level. It only allows 3 tokens and leases to be stored.
```
$ vault write sys/quotas/lease-count/lease-test path="auth/token" max_leases=3
```

Create a shortcut script, lease-count-test.sh which invokes the test path.

$ tee lease-count-test.sh <<EOF
echo "Request #1"
vault token create -policy=default
echo "\nRequest #2"
vault token create -policy=default
echo "\nRequest #3"
vault token create -policy=default
echo "\nRequest #4"
vault token create -policy=default
EOF

Make sure that the script is executable.
```
$ chmod +x lease-count-test.sh
```

Run the script to see how the quota rule behaves.

$ ./lease-count-test.sh

Three tokens were created successfully; however, the fourth request failed due to the lease count quota. Your output should look similar to follow.

Request #1
Key                  Value
---                  -----
token                hvs.CAESIAtHe_XP_2UmXVfRaR_zQRalVg_TWSzrsqEy6PmgInDqGh4KHGh2cy5COTNKQ2pzYVN2WlRncjF3akhPMVJ1ZHk
token_accessor       rhmyha9JjSxh9hkjT2i2i8An
token_duration       768h
token_renewable      true
token_policies       ["default"]
identity_policies    []
policies             ["default"]

Request #2
Key                  Value
---                  -----
token                hvs.CAESIE0L_nNkTLI3cLayb0NpyEw2gc3x6-pjVIpoYhfyyOBEGh4KHGh2cy5xSjhhNWRaVnp5bzF6V2FqUE44emZUZk8
token_accessor       oKKZViZet2tTl8TKqnbYMplX
token_duration       768h
token_renewable      true
token_policies       ["default"]
identity_policies    []
policies             ["default"]

Request #3
Key                  Value
---                  -----
token                hvs.CAESIP53xCUOcY0aJplpxagixzxZskdry6z1YUrYx5Urzk6eGh4KHGh2cy5UWGl0RTR2WDhLTUFFMGtLRnJJTjlBcHc
token_accessor       gtM8SoUEsaPkf9TvBLk5dswU
token_duration       768h
token_renewable      true
token_policies       ["default"]
identity_policies    []
policies             ["default"]

Request #4
Error creating token: Error making API request.

URL: POST http://127.0.0.1:8200/v1/auth/token/create
Code: 429. Errors:

* 1 error occurred:
   * request path "auth/token/create": lease count quota exceeded

Note

If your Vault server already has client tokens, the lease count quota may be exceeded sooner.

Also, you can find the lease count quota exceeded error in the audit log. (Be sure to set it to the correct audit log path for your environment.)

$ tail -f /var/log/vault-audit.log | jq

...snip...
  "response": {
    "mount_type": "token"
  },
  "error": "1 error occurred:\n\t* request path \"auth/token/create\": lease count quota exceeded\n\n"
}

If you revoke one of the tokens, you should be able to request a new one.
Example:
```
$ vault token revoke hvs.CAESIAtHe_XP_2UmXVfRaR_zQRalVg_TWSzrsqEy6PmgInDqGh4KHGh2cy5COTNKQ2pzYVN2WlRncjF3akhPMVJ1ZHk
Success! Revoked token (if it existed)
```
Now, request a new one.
```
$ vault token create -policy=default
```
The best practice is to set the tokens and leases' time-to-live (TTL) to be short and don't let them hang around longer than necessary. The lease count quotas allow you to set the upper limit to protect your Vault environment from running into an issue due to a lack of token and lease governance.

Delete the lease-test quota rule.

$ vault delete sys/quotas/lease-count/lease-test

Vault 1.12.0 or later

If you are running Vault 1.12.0 or later, you can set more granular lease count quota.

To demonstrate, enable approle auth method.
```
$ vault auth enable approle
```

Create a role named "test-role".

$  vault write auth/approle/role/test-role token_policies="default" \
      token_ttl=1h token_max_ttl=4h

Create a lease count quota to limit the max number of leases for the test-role to 2.

$ vault write sys/quotas/lease-count/test-role-limit \
   max_leases=2 \
   path="auth/approle" \
   role="test-role"

Retrieve the role ID of the test-role and store it in role_id.txt.

$ vault read -field=role_id auth/approle/role/test-role/role-id > role_id.txt

Generate a secret ID of the test-role and store it in secret_id.txt.

$ vault write -force -field=secret_id auth/approle/role/test-role/secret-id > secret_id.txt

Create a test script.

$ tee lease-count-test-2.sh <<EOF
echo "Login attempt #1"
vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)
echo "\nLogin attempt #2"
vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)
echo "\nLogin attempt #3"
vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)
EOF

Ensure that the script is executable.
```
$ chmod +x lease-count-test-2.sh
```

Run the script to see how the quota rule behaves.

$ ./lease-count-test-2.sh

Output:

Login attempt #1
Key                     Value
---                     -----
token                   hvs.CAESICjr1nqxXo4UDJyh9MkMwvecOyEFlE69Pgs7ftdmUvF4Gh4KHGh2cy4yN3Q1WXRWU0tpSlBsQWpkZGJOdXpMZHY
token_accessor          1P7KhrLmdrWRFIPR6IJnR6Ls
token_duration          1h
token_renewable         true
token_policies          ["default"]
identity_policies       []
policies                ["default"]
token_meta_role_name    test-role

Login attempt #2
Key                     Value
---                     -----
token                   hvs.CAESIGm6g-GUUPCJq3zvs_L1f1RIB4-7RD7a-yQxUaphFi0qGh4KHGh2cy5IVHluNVlEclZrSHNXb3lOZGVraWdmN20
token_accessor          ThLLvtE42QE0wMbAkTEvOA8N
token_duration          1h
token_renewable         true
token_policies          ["default"]
identity_policies       []
policies                ["default"]
token_meta_role_name    test-role

Login attempt #3
Error writing data to auth/approle/login: Error making API request.

URL: PUT http://127.0.0.1:8200/v1/auth/approle/login
Code: 429. Errors:

* 1 error occurred:
   * request path "auth/approle/login": lease count quota exceeded

The last attempt failed. The test-role-limit lease count quota should not affect other AppRole roles.

Clean up

If you wish to clean up your environment after completing the tutorial, follow the steps in this section.

Unset the VAULT_TOKEN environment variable.
```
$ unset VAULT_TOKEN
```
Unset the VAULT_ADDR environment variable.
```
$ unset VAULT_ADDR
```

Delete files created during the test.

$ rm test-encryption.sh lease-count-test.sh lease-count-test-2.sh role_id.txt secret_id.txt

If you are running Vault locally in -dev mode, you can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.
```
$ pgrep -f vault | xargs kill
```

Next steps

In this tutorial, you learned the basic commands to set resource quotas to protect your Vault environment. To leverage this feature, you need Vault 1.5 or later.

Rate limit quotas allow Vault operators to set inbound request rate limits which can be set on the root level or a specific path. This is available in both Vault OSS and Vault Enterprise.

Lease count quotas require Vault Enterprise Platform and allow operators to set the maximum number of tokens and leases to be persisted at any given time. This can prevent Vault from exhausting the resource on the storage backend.

You also learned that audit logging can be enabled to trace the number of requests that were rejected due to the rate limit quota.

Help and Reference

25 tutorials

Vault Enterprise
Learn features that are only available to Vault Enterprise.
- Vault

Solution

Prerequisites

Lab setup

Resource quota configuration

Rate limit quotas

Parameters

Vault 1.12.0 or later

Vault Enterprise Example

Lease count quotas

Parameters

Vault 1.12.0 or later

Test to understand the resource quotas

Rate limit quota test

Lease count quota test

Vault 1.12.0 or later

Clean up

Next steps

Help and Reference

This tutorial also appears in: