• HashiCorp Developer

  • HashiCorp Cloud Platform
  • Terraform
  • Packer
  • Consul
  • Vault
  • Boundary
  • Nomad
  • Waypoint
  • Vagrant
Vault
  • Install
  • Tutorials
  • Documentation
  • API
  • Try Cloud(opens in new tab)
  • Sign up
Fundamentals

Skip to main content
13 tutorials
  • Configure Vault
  • Production Hardening
  • Generate Root Tokens Using Unseal Keys
  • Rekeying & Rotating Vault
  • Protecting Vault with Resource Quotas
  • Performance Tuning
  • Identity: Entities and Groups
  • Codify Management of Vault Using Terraform
  • Codify Management of Vault Enterprise Using Terraform
  • Emergency Break-Glass Features
  • Create Customized HTTP Headers for your Vault Requests
  • Apply Codified OSS Vault Configuration to HCP Vault with Terraform
  • Audit Device Logs and Incident Response with Elasticsearch

  • Resources

  • Tutorial Library
  • Certifications
  • Community Forum
    (opens in new tab)
  • Support
    (opens in new tab)
  • GitHub
    (opens in new tab)
  1. Developer
  2. Vault
  3. Tutorials
  4. Fundamentals
  5. Protecting Vault with Resource Quotas

Protecting Vault with Resource Quotas

  • 23min

  • VaultVault
  • InteractiveInteractive

Vault is an API driven system that all communication between the clients and Vault are done through Vault API.

As the number of client applications increases, rogue applications can degrade Vault performance. The issue is often caused by applications that are:

  • Generating an unbounded number of leases and/or loading too much data into Vault leading to exhausting Consul storage backend's available memory
  • Consuming an erroneously large amount of bandwidth leading to internal throttling of the Vault cluster as a whole

Solution

Set Vault resource quotas to protect your Vault environment's stability and network, as well as storage resource consumption from runaway application behavior and distributed denial of service (DDoS) attack.

The Vault operators can control how applications request resources from Vault, and Vault's storage and network infrastructure by setting the following:

FeatureDescriptionVault OSSVault Enterprise
Rate Limit QuotasLimit maximum amount of requests per second (RPS) to a system or mount to protect network bandwidth✔️✔️
Lease Count QuotasCap number of leases generated in a system or mount to protect system stability and storage performance at scale✔️

NOTE: Lease Count Quotas requires Vault Enterprise Standard license.

To set the rate limit quotas and lease count quotas, use the sys/quotas/<type> endpoint. Each resource quota has a name to identify the quota rule. To manage an individual quota rule, the endpoint becomes sys/quotas/<type>/<name>.

The <type> can be:

  • rate-limit: rate limit quota
  • lease-count: lease count quota

NOTE: Batch tokens do not count towards the lease count quota. Creation of new batch tokens will, however, be blocked if the lease count quota is exceeded.

NOTE: Feature enhancements were introduced to both rate limit quota and lease count quota in Vault 1.12.0. Refer to each section for more details.

Prerequisites

To perform the tasks described in this tutorial, you need Vault v1.5 or later. Refer to the Getting Started tutorial to install Vault. Make sure that your Vault server has been initialized and unsealed.

Launch Terminal

This tutorial includes a free interactive command-line lab that lets you follow along on actual cloud infrastructure.

Lab setup

  1. Open a terminal and start a Vault dev server with root as the root token.

    $ vault server -dev -dev-root-token-id root
    

    The Vault dev server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.

    Insecure operation: Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.

  2. Export an environment variable for the vault CLI to address the Vault server.

    $ export VAULT_ADDR=http://127.0.0.1:8200
    
  3. Export an environment variable for the vault CLI to authenticate with the Vault server.

    $ export VAULT_TOKEN=root
    

The Vault server is ready.

Resource quota configuration

To configure the resource quota, use the sys/quotas/config endpoint.

ParameterTypeDescription
enable_rate_limit_audit_loggingbooleanEnable or disable audit logging when requests get rejected due to rate limit quota violations. By default, audit logging is disabled (false).

By default, the requests rejected due to rate limit quota violations are not written to the audit log. Therefore, if you wish to log the rejected requests for traceability, you must set the enable_rate_limit_audit_logging to true. The requests rejected due to reaching the lease count quotas are always logged that you do not need to set any parameter.

CAUTION: Enabling the rate limit audit logging may have an impact on the Vault performance if the volume of rejected requests is large.

  1. Enable a file audit device which outputs to /var/log/vault-audit.log (or your desired file location).

    $ vault audit enable file file_path="/var/log/vault-audit.log"
    
  2. To enable the audit logging for rate limit quotas, execute the following command.

    $ vault write sys/quotas/config enable_rate_limit_audit_logging=true
    
  3. Read the quota configuration to verify.

    $ vault read sys/quotas/config
    
    Key                                   Value
    ---                                   -----
    enable_rate_limit_audit_logging       true
    enable_rate_limit_response_headers    false
    rate_limit_exempt_paths               []
    
  1. Enable file audit device.

    First, create the HTTP request payload specifying the audit log path to be /var/log/vault-audit.log (or your desired file location).

    $ tee audit-payload.json <<EOF
    {
      "type": "file",
      "options": {
          "file_path": "/var/log/vault-audit.log"
      }
    }
    EOF
    

    Use the sys/audit endpoint to enable file audit log.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
        --request POST \
        --data @audit-payload.json \
        $VAULT_ADDR/v1/sys/audit/file
    

    Set the target file_path to your desired location.

  2. Create the HTTP request payload to enable the rate limit quota audit logging.

    $ tee payload.json <<EOF
    {
       "enable_rate_limit_audit_logging": true
    }
    EOF
    
  3. Invoke the sys/quotas/config endpoint.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
        --request POST \
        --data @payload.json \
        $VAULT_ADDR/v1/sys/quotas/config
    
  4. Read the quota configuration to verify.

    $ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
        $VAULT_ADDR/v1/sys/quotas/config | jq -r ".data"
    

    Example output:

    {
      "enable_rate_limit_audit_logging": true,
      "enable_rate_limit_response_headers": false,
      "rate_limit_exempt_paths": []
    }
    

Rate limit quotas

Rate limit quotas are designed to protect Vault against external distributed denial of service (DDoS) attacks and are fundamental to Vault's security model. Therefore, it is a part of Vault's core feature set available in both OSS and Enterprise.

To set rate limit quotas, use the sys/quotas/rate-limit/<name> endpoint.

Parameters

ParameterTypeDescription
namestringName of the quota rule
pathstringTarget path or namespace to apply the quota rule. A blank path configures a global rate limit quota
ratefloatRate for the number of allowed requests per second (RPS)
rolestringVault 1.12 or later: Login role to apply this quota to. When this parameter is set, the path must be configured to a valid auth method with a concept of roles.
intervalsecondThe duration to enforce rate limiting for (default is 1 second)
block_intervalstringIf set, when a client reaches a rate limit threshold, the client will be prohibited from any further requests until after the 'block_interval' has elapsed.

NOTE: If you are running Vault 1.12 or later, the path can be a fully qualified path, and it can end with * (e.g., auth/token/create*).

  1. Create a rate limit quota named, "global-rate" which limits inbound workload to 500 requests per seconds.

    $ vault write sys/quotas/rate-limit/global-rate rate=500
    Success! Data written to: sys/quotas/rate-limit/global-rate
    
  2. Read the global-rate rule to verify its configuration.

    $ vault read sys/quotas/rate-limit/global-rate
    
    Key               Value
    ---               -----
    block_interval    0
    interval          1
    name              global-rate
    path              n/a
    rate              500
    type              rate-limit
    

    NOTE: In absence of path, this quota rule applies to the global level instead of a specific mount or namespace.

  3. Create a rate limit quota named, "transit-limit" which limits the access to the Transit secrets engine to be 1000 requests per minute (60 seconds).

    First, enable Transit secrets engine at transit.

    $ vault secrets enable transit
    Success! Enabled the pki secrets engine at: transit/
    

    Now, create a rate limit quota.

    $ vault write sys/quotas/rate-limit/transit-limit path="transit" rate=1000 interval=60
    Success! Data written to: sys/quotas/rate-limit/transit-limit
    
  4. Read the transit-limit rule to verify its configuration.

    $ vault read sys/quotas/rate-limit/transit-limit
    
    Key               Value
    ---               -----
    block_interval    0
    interval          60
    name              transit-limit
    path              transit/
    rate              1000
    role              n/a
    type              rate-limit
    

Vault 1.12.0 or later

Previously, the path depth had to be the mount of an auth method or secrets engine. If you are running Vault 1.12 or later, you can set the path to be deeper than the mount point (in this example, transit/).

  1. Create a rate limit quota named, "encrypt-orders" to limit the data encryption requests using orders key to be 500 per second.

    First, create an encryption key named, "orders".

    $ vault write -f transit/keys/orders
    Success! Data written to: transit/keys/orders
    

    Now, create the "encrypt-orders" rate limit quota.

    $ vault write sys/quotas/rate-limit/transit-order path="transit/encrypt/orders" rate=500
    Success! Data written to: sys/quotas/rate-limit/transit-order
    
  2. Verify the rate limit quota configuration.

    $ vault read sys/quotas/rate-limit/transit-order
    

    Output:

    Key               Value
    ---               -----
    block_interval    0
    interval          1
    name              transit-order
    path              transit/encrypt/orders
    rate              500
    role              n/a
    type              rate-limit
    

Test: Be sure to visit the Test to understand the resource quota section to see how resource quotas enforcement works.

Vault Enterprise Example

Consider that you have K/V v2 secrets engine enabled at kv-v2 under us-west namespace. Create a rate limit quota named, "orders" which limits the incoming requests against kv-v2 to 20 requests per second maximum.

  1. Create a namespace, "us-west".

    $ vault namespace create us-west
    
  2. Enable kv-v2 secrets engine in the us-west namespace.

    $ vault secrets enable -ns="us-west" kv-v2
    
  3. Now, create the orders quota rule.

    $ vault write sys/quotas/rate-limit/orders \
         path="us-west/kv-v2" \
         rate=20
    
  4. Verify the configuration.

    $ vault read sys/quotas/rate-limit/orders
    
    Key               Value
    ---               -----
    block_interval    0
    interval          1
    name              orders
    path              us-west/kv-v2/
    rate              20
    role              n/a
    type              rate-limit
    

Create a rate limit quota named, "global-rate" which limits inbound workload to 500 requests per second.

  1. Finally, invoke the sys/quotas/rate-limit endpoint.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
        --request POST \
        --data '{ "rate": 500 }' \
        $VAULT_ADDR/v1/sys/quotas/rate-limit/global-rate
    
  2. Read the newly created global-rate quota rule.

    $ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
        $VAULT_ADDR/v1/sys/quotas/rate-limit/global-rate | jq -r ".data"
    

    Output:

    {
      "block_interval": 0,
      "interval": 1,
      "name": "global-rate",
      "path": "",
      "rate": 500,
      "type": "rate-limit"
    }
    

    NOTE: In absence of path, this quota rule applies to the root namespace instead of a specific mount or namespace.

  3. Create a rate limit quota named, "transit-limit" which limits the access to the Transit secrets engine to be 1000 requests per minute (60 seconds).

    First, enable Transit secrets engine at transit.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
        --request POST \
        --data '{"type":"transit"}' \
        $VAULT_ADDR/v1/sys/mounts/transit
    

    Now, create a "transit-limit" rate limit quota.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data '{"path": "transit", "rate": 1000, "interval": 60 }' \
       $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-limit
    
  4. Read the transit-limit rule to verify its configuration.

    $ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
       $VAULT_ADDR/v1/sys/quotas/rate-limit/transit-limit | jq -r ".data"
    

    Output:

    {
       "block_interval": 0,
       "interval": 60,
       "name": "transit-limit",
       "path": "transit/",
       "rate": 1000,
       "role": "",
       "type": "rate-limit"
    }
    

Vault 1.12.0 or later

Previously, the path depth had to be the mount of an auth method or secrets engine. If you are running Vault 1.12 or later, you can set the path to be deeper than the mount point (in this example, transit/).

  1. Create a rate limit quota named, "encrypt-orders" to limit the data encryption requests using orders key to be 500 per second.

    First, create an encryption key named, "orders".

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
        --request POST \
        $VAULT_ADDR/v1/transit/keys/orders
    

    Now, create the "encrypt-orders" rate limit quota.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data '{ "path": "transit/encrypt/orders", "rate": 500 }' \
       $VAULT_ADDR/v1/sys/quotas/rate-limit/encrypt-orders
    
  2. Verify the rate limit quota configuration.

    $ curl -s --header "X-Vault-Token: $VAULT_TOKEN" \
       $VAULT_ADDR/v1/sys/quotas/rate-limit/encrypt-orders | jq -r ".data"
    

    Outpu:

    {
       "block_interval": 0,
       "interval": 1,
       "name": "encrypt-orders",
       "path": "transit/encrypt/orders",
       "rate": 500,
       "role": "",
       "type": "rate-limit"
    }
    

Test: Be sure to visit the Test to understand the resource quota section to see how resource quotas enforcement works.

Vault Enterprise Example

Consider that you have K/V v2 secrets engine enabled at kv-v2 under us-west namespace. Create a rate limit quota named, "orders" which limits the incoming requests against kv-v2 to 20 requests per second maximum.

  1. Create a namespace, "us-west".

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
         --request POST \
         $VAULT_ADDR/v1/sys/namespaces/us-west
    
  2. Enable kv-v2 secrets engine in the us-west namespace.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
        --header "X-Vault-Namespace: us-west" \
        --request POST \
        --data '{"type": "kv-v2"}' \
        $VAULT_ADDR/v1/sys/mounts/kv-v2
    
  3. Create the HTTP request payload defining parameters for "orders" quota rule.

    $ tee payload.json <<EOF
    {
       "path": "us-west/kv-v2",
       "rate": 20
    }
    EOF
    
  4. Finally, invoke the sys/quota/rate-limit endpoint to create an "orders" quota rule.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
         --request POST \
         --data @payload.json \
         $VAULT_ADDR/v1/sys/quotas/rate-limit/orders
    

Lease count quotas

NOTE: Lease Count Quotas is a part of Vault Enterprise Platform features.

Lease count quota is designed to protect Vault from a large volume of leases and tokens persisted in Vault, which can pressure its storage backend. This acts as a guard rail for system stability in large-scale Vault Enterprise deployments where Vault is used as a service.

Parameters

ParameterTypeDescription
namestringName of the quota rule
pathstringTarget path or namespace to apply the quota rule. A blank path configures a global lease count quota
max_leasesintMaximum number of leases allowed by the quota rule
rolestringVault 1.12 or later: Login role to apply this quota to. When this parameter is set, the path must be configured to a valid auth method with a concept of roles.

NOTE: If you are running Vault 1.12 or later, the path can be a fully qualified path, and it can end with * (e.g., auth/token/create*).

  1. To demonstrate this feature, enable database secrets engine at "postgres" in the us-west namespace.

    $ vault secrets enable -ns=us-west -path=postgres database
    
  2. Create a lease count quota named, "db-creds" which limits the incoming requests for a new set of DB credentials to 100 concurrent, valid leases maximum.

    $ vault write sys/quotas/lease-count/db-creds max_leases=100 \
       path="us-west/postgres"
    

    NOTE: Similar to rate limit quotas, the quota rules apply globally in the absence of path. In the example, you created a quota rule against the us-west/postgres path. If a global quota rule exists on the root namespace, the quota rule defined on a specific path takes precedence.

  3. Verify the configuration.

    $ vault read sys/quotas/lease-count/db-creds
    
    Key           Value
    ---           -----
    counter       0
    max_leases    100
    name          db-creds
    path          us-west/postgres/
    role          n/a
    type          lease-count
    

Vault 1.12.0 or later

Previously, you could only apply to the path of where the auth method was enabled. When you are running Vault 1.12 or later, you can specify the role if the target auth method has the concept of roles.

  1. Create a role named "webapp" for approle auth method.

    First, enable the approle auth method.

    $ vault auth enable approle
    

    Create "webapp" role.

    $ vault write auth/approle/role/webapp \
       token_policies="webapp-policy" \
       token_ttl=4h \
       token_max_ttl=12h
    
  2. Create a lease count quota named "webapp-tokens" which limits the creation of token for the webapp role to maximum of 100.

    $ vault write sys/quotas/lease-count/webapp-tokens max_leases=100 \
       path="auth/approle" \
       role="webapp"
    

    If you want to set it for the approle auth method enable in the us-west namespace, the path should be us-west/auth/approle.

  3. Verify the configuration.

    $ vault read sys/quotas/lease-count/webapp-tokens
    

    Output:

    Key           Value
    ---           -----
    counter       0
    max_leases    100
    name          webapp-tokens
    path          auth/approle/
    role          webapp
    type          lease-count
    
  1. To demonstrate this feature, enable database secrets engine at "postgres" in the us-west namespace.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --header "X-Vault-Namespace: us-west" \
       --request POST \
       --data '{"type":"database"}' \
       $VAULT_ADDR/v1/sys/mounts/postgres
    
  2. Create a lease count quota named, "db-creds" which limits the incoming requests for a new set of DB credentials to 100 concurrent, valid leases maximum.

    First, create the HTTP request payload defining the parameters for db-creds quota rule.

    $ tee payload.json <<EOF
    {
       "max_leases": 100,
       "path": "us-west/postgres"
    }
    EOF
    

    Now, invoke the sys/quotas/lease-count endpoint to create the quota rule.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data @payload.json \
       $VAULT_ADDR/v1/sys/quotas/lease-count/db-creds
    

    NOTE: Similar to rate limit quotas, the quota rules apply globally in the absence of path. In the example, you created a quota rule against the us-west/postgres path. If a global quota rule exists on the root namespace, the quota rule defined on a specific path takes precedence.

  3. Verify the configuration.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       $VAULT_ADDR/v1/sys/quotas/lease-count/db-creds | jq -r ".data"
    

    Output:

    {
       "counter": 0,
       "max_leases": 100,
       "name": "db-creds",
       "path": "us-west/postgres/",
       "role": "",
       "type": "lease-count"
    }
    

Vault 1.12.0 or later

Previously, you could only apply to the path of where the auth method was enabled. When you are running Vault 1.12 or later, you can specify the role if the target auth method has the concept of roles.

  1. To demonstrate this feature, you are going to use the approle auth method. Enable the approle auth method.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data '{"type": "approle"}' \
       $VAULT_ADDR/v1/sys/auth/approle
    
  2. Create a role named "webapp" for approle auth method.

    First, create the API request payload.

    $ tee payload.json <<EOF
    {
       "token_policies": "webapp-policy",
       "token_ttl": "4h",
       "token_max_ttl": "12h"
    }
    EOF
    

    Now, create a "webapp" role.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data @payload.json \
       $VAULT_ADDR/v1/auth/approle/role/webapp
    
  3. Create a lease count quota named "webapp-tokens" which limits the creation of token for the webapp role to maximum of 100.

    First, create the request payload.

    $ tee payload_lcq.json <<EOF
    {
       "max_leases": 100,
       "path": "auth/approle",
       "role": "webapp"
    }
    EOF
    

    If you want to set it for the approle auth method enabled in the us-west namespace, the path should be us-west/auth/approle.

    Now, invoke the sys/quotas/lease-count endpoint to create the quota rule.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       --request POST \
       --data @payload_lcq.json \
       $VAULT_ADDR/v1/sys/quotas/lease-count/webapp-tokens
    
  4. Verify the configuration.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
       $VAULT_ADDR/v1/sys/quotas/lease-count/webapp-tokens | jq -r ".data"
    

    Output:

    {
       "counter": 0,
       "max_leases": 100,
       "name": "webapp-tokens",
       "path": "auth/approle/",
       "role": "webapp",
       "type": "lease-count"
    }
    

Test: Be sure to visit the Test to understand the resource quota section to see how resource quotas enforcement works.

Test to understand the resource quotas

Now that you learned the basic commands, let's test to see how it works.

Rate limit quota test

  1. Enable transit secrets engine if it is not enabled.

    $ vault secrets enable transit
    
  2. Create a "test" encryption key.

    $ vault write -f transit/keys/test
    
  3. For the purpose of demonstration, create a "test-transit" rate limit quota such that you can only make 1 requests every 10 seconds.

    $ vault write sys/quotas/rate-limit/rate-test path=transit rate=1 interval=10
    
  4. Create a shortcut script, test-encryption.sh which makes request to encrypt data using "test" key.

    $ tee test-encryption.sh <<EOF
    echo "Request 1"
    vault write transit/encrypt/test plaintext=$(base64 <<< "4111 1111 1111 1111")
    echo "\nRequest 2"
    vault write transit/encrypt/test plaintext=$(base64 <<< "4222 2222 2222 2222")
    echo "\nRequest 3"
    vault write transit/encrypt/test plaintext=$(base64 <<< "4333 3333 3333 3333")
    EOF
    
  5. Ensure that the script is executable.

    $ chmod +x test-encryption.sh
    
  6. Run the script to see how the quota rule behaves.

    $ ./test-encryption.sh
    

    Output:

    Request 1
    Key            Value
    ---            -----
    ciphertext     vault:v1:6zY2JVtuw3r11ST/JjMkmoNbbcCpQwGbJJ0BJfByk//GLQof8mcxAoerq5Y++BZK
    key_version    1
    
    Request 2
    Error writing data to transit/encrypt/test: Error making API request.
    
    URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/test
    Code: 429. Errors:
    
    * request path "transit/encrypt/test": rate limit quota exceeded
    
    Request 3
    Error writing data to transit/encrypt/test: Error making API request.
    
    URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/test
    Code: 429. Errors:
    
    * request path "transit/encrypt/test": rate limit quota exceeded
    

    The second and third requests failed because of the limit.

  7. Earlier, you configured the resource quotas to enable audit logging of requests that were rejected due to rate limit quota rule violation. Inspect your audit log for its entry.

    $ more /var/log/vault-audit.log | jq
    

    If your audit log path is not /var/log/vault-audit.log, be sure to set it to the correct path.

    ...snip...
       "request": {
          "id": "4170a181-6a18-589b-f026-a62ad08ae644",
          "operation": "update",
          "namespace": {
             "id": "root"
          },
          "path": "transit/encrypt/test",
          "data": {
             "plaintext": "hmac-sha256:50978c7cb3f0f463491a26152353f485b579d1e8759ec7b7b7c28f3c5b9232cd"
          },
          "remote_address": "127.0.0.1",
          "remote_port": 61028
       },
       "error": "request path \"transit/encrypt/test\": rate limit quota exceeded"
    }
    

    You should find an error message indicating that rate limit quota was exceeded. You can trace the audit log to see how many requests were rejected due to the rate limit quota. It may be working as expected or you may find suspicious activities against a specific path.

  8. If you are running Vault 1.12.0 or later, create another rate limit quota to specify the path to transit/encrypt/test where allowed rate is 2 with interval of 10 seconds.

    $ vault write sys/quotas/rate-limit/encryption-limit \
       path="transit/encrypt/test" \
       rate=2 \
       interval=10
    
  9. Run the script again to see how the quota rule behaves.

    $ ./test-encryption.sh
    

    Output:

    Request 1
    Key            Value
    ---            -----
    ciphertext     vault:v1:44sOnzePg8c0XaxzGeGzwJ94UZ1uuif8p0nGuze8utbvWVhqNHDiIwo8VZmPrcli
    key_version    1
    
    Request 2
    Key            Value
    ---            -----
    ciphertext     vault:v1:w9tyXqoXH6uXuOWUktVWP/yrzOOZDv+TrRbbRU5GZEVappASIRnh3+6KwkJkjWKE
    key_version    1
    
    Request 3
    Error writing data to transit/encrypt/test: Error making API request.
    
    URL: PUT http://127.0.0.1:8200/v1/transit/encrypt/test
    Code: 429. Errors:
    
    * request path "transit/encrypt/test": rate limit quota exceeded
    

    NOTE: This time, only the last request failed. When a more granular path is set, the rate limit quota rule takes precedence against the path.

    If you have another key or try to decript (transit/decrypt/test), the rate-test rate limit quota will be applied.

Lease count quota test

  1. Similarly, create a very limited lease count quota named, "lease-test" which applies on the root level. It only allows 3 tokens and leases to be stored.

    $ vault write sys/quotas/lease-count/lease-test path="auth/token" max_leases=3
    
  2. Create a shortcut script, lease-count-test.sh which invokes the test path.

    $ tee lease-count-test.sh <<EOF
    echo "Request #1"
    vault token create -policy=default
    echo "\nRequest #2"
    vault token create -policy=default
    echo "\nRequest #3"
    vault token create -policy=default
    echo "\nRequest #4"
    vault token create -policy=default
    EOF
    
  3. Make sure that the script is executable.

    $ chmod +x lease-count-test.sh
    
  4. Run the script to see how the quota rule behaves.

    $ ./lease-count-test.sh
    

    Three tokens were created successfully; however, the fourth request failed due to the lease count quota. Your output should look similar to follow.

    Request #1
    Key                  Value
    ---                  -----
    token                hvs.CAESIAtHe_XP_2UmXVfRaR_zQRalVg_TWSzrsqEy6PmgInDqGh4KHGh2cy5COTNKQ2pzYVN2WlRncjF3akhPMVJ1ZHk
    token_accessor       rhmyha9JjSxh9hkjT2i2i8An
    token_duration       768h
    token_renewable      true
    token_policies       ["default"]
    identity_policies    []
    policies             ["default"]
    
    Request #2
    Key                  Value
    ---                  -----
    token                hvs.CAESIE0L_nNkTLI3cLayb0NpyEw2gc3x6-pjVIpoYhfyyOBEGh4KHGh2cy5xSjhhNWRaVnp5bzF6V2FqUE44emZUZk8
    token_accessor       oKKZViZet2tTl8TKqnbYMplX
    token_duration       768h
    token_renewable      true
    token_policies       ["default"]
    identity_policies    []
    policies             ["default"]
    
    Request #3
    Key                  Value
    ---                  -----
    token                hvs.CAESIP53xCUOcY0aJplpxagixzxZskdry6z1YUrYx5Urzk6eGh4KHGh2cy5UWGl0RTR2WDhLTUFFMGtLRnJJTjlBcHc
    token_accessor       gtM8SoUEsaPkf9TvBLk5dswU
    token_duration       768h
    token_renewable      true
    token_policies       ["default"]
    identity_policies    []
    policies             ["default"]
    
    Request #4
    Error creating token: Error making API request.
    
    URL: POST http://127.0.0.1:8200/v1/auth/token/create
    Code: 429. Errors:
    
    * 1 error occurred:
       * request path "auth/token/create": lease count quota exceeded
    

    NOTE: If your Vault server already has client tokens, the lease count quota may be exceeded sooner.

    Also, you can find the lease count quota exceeded error in the audit log. (Be sure to set it to the correct audit log path for your environment.)

    $ tail -f /var/log/vault-audit.log | jq
    
    ...snip...
      "response": {
        "mount_type": "token"
      },
      "error": "1 error occurred:\n\t* request path \"auth/token/create\": lease count quota exceeded\n\n"
    }
    
  5. If you revoke one of the tokens, you should be able to request a new one.

    Example:

    $ vault token revoke hvs.CAESIAtHe_XP_2UmXVfRaR_zQRalVg_TWSzrsqEy6PmgInDqGh4KHGh2cy5COTNKQ2pzYVN2WlRncjF3akhPMVJ1ZHk
    Success! Revoked token (if it existed)
    

    Now, request a new one.

    $ vault token create -policy=default
    

    The best practice is to set the tokens and leases' time-to-live (TTL) to be short and don't let them hang around longer than necessary. The lease count quotas allow you to set the upper limit to protect your Vault environment from running into an issue due to a lack of token and lease governance.

  6. Delete the lease-test quota rule.

    $ vault delete sys/quotas/lease-count/lease-test
    

Vault 1.12.0 or later

If you are running Vault 1.12.0 or later, you can set more granular lease count quota.

  1. To demonstrate, enable approle auth method.

    $ vault auth enable approle
    
  2. Create a role named "test-role".

    $  vault write auth/approle/role/test-role token_policies="default" \
          token_ttl=1h token_max_ttl=4h
    
  3. Create a lease count quota to limit the max number of leases for the test-role to 2.

    $ vault write sys/quotas/lease-count/test-role-limit \
       max_leases=2 \
       path="auth/approle" \
       role="test-role"
    
  4. Retrieve the role ID of the test-role and store it in role_id.txt.

    $ vault read -field=role_id auth/approle/role/test-role/role-id > role_id.txt
    
  5. Generate a secret ID of the test-role and store it in secret_id.txt.

    $ vault write -force -field=secret_id auth/approle/role/test-role/secret-id > secret_id.txt
    
  6. Create a test script.

    $ tee lease-count-test-2.sh <<EOF
    echo "Login attempt #1"
    vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)
    echo "\nLogin attempt #2"
    vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)
    echo "\nLogin attempt #3"
    vault write auth/approle/login role_id=$(cat role_id.txt) secret_id=$(cat secret_id.txt)
    EOF
    
  7. Ensure that the script is executable.

    $ chmod +x lease-count-test-2.sh
    
  8. Run the script to see how the quota rule behaves.

    $ ./lease-count-test-2.sh
    

    Output:

    Login attempt #1
    Key                     Value
    ---                     -----
    token                   hvs.CAESICjr1nqxXo4UDJyh9MkMwvecOyEFlE69Pgs7ftdmUvF4Gh4KHGh2cy4yN3Q1WXRWU0tpSlBsQWpkZGJOdXpMZHY
    token_accessor          1P7KhrLmdrWRFIPR6IJnR6Ls
    token_duration          1h
    token_renewable         true
    token_policies          ["default"]
    identity_policies       []
    policies                ["default"]
    token_meta_role_name    test-role
    
    Login attempt #2
    Key                     Value
    ---                     -----
    token                   hvs.CAESIGm6g-GUUPCJq3zvs_L1f1RIB4-7RD7a-yQxUaphFi0qGh4KHGh2cy5IVHluNVlEclZrSHNXb3lOZGVraWdmN20
    token_accessor          ThLLvtE42QE0wMbAkTEvOA8N
    token_duration          1h
    token_renewable         true
    token_policies          ["default"]
    identity_policies       []
    policies                ["default"]
    token_meta_role_name    test-role
    
    Login attempt #3
    Error writing data to auth/approle/login: Error making API request.
    
    URL: PUT http://127.0.0.1:8200/v1/auth/approle/login
    Code: 429. Errors:
    
    * 1 error occurred:
       * request path "auth/approle/login": lease count quota exceeded
    

    The last attempt failed. The test-role-limit lease count quota should not affect other AppRole roles.

Clean up

If you wish to clean up your environment after completing the tutorial, follow the steps in this section.

  1. Unset the VAULT_TOKEN environment variable.

    $ unset VAULT_TOKEN
    
  2. Unset the VAULT_ADDR environment variable.

    $ unset VAULT_ADDR
    
  3. Delete files created during the test.

    $ rm test-encryption.sh lease-count-test.sh lease-count-test-2.sh role_id.txt secret_id.txt 
    
  4. If you are running Vault locally in -dev mode, you can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.

    $ pgrep -f vault | xargs kill
    

Next steps

In this tutorial, you learned the basic commands to set resource quotas to protect your Vault environment. To leverage this feature, you need Vault 1.5 or later.

Rate limit quotas allow Vault operators to set inbound request rate limits which can be set on the root level or a specific path. This is available in both Vault OSS and Vault Enterprise.

Lease count quotas require Vault Enterprise Platform and allow operators to set the maximum number of tokens and leases to be persisted at any given time. This can prevent Vault from exhausting the resource on the storage backend.

You also learned that audit logging can be enabled to trace the number of requests that were rejected due to the rate limit quota.

Help and Reference

  • Resource Quotas Concepts doc
  • Rate limit quota API doc
  • Lease count quota API doc
 Previous
 Next

This tutorial also appears in:

  •  
    7 tutorials
    Vault 1.12 Release Highlights
    The listed tutorials were updated to showcase the new enhancements introduced in Vault 1.12.
    • Vault
  •  
    23 tutorials
    Vault Enterprise
    Learn features that are only available to Vault Enterprise.
    • Vault

On this page

  1. Protecting Vault with Resource Quotas
  2. Solution
  3. Prerequisites
  4. Lab setup
  5. Resource quota configuration
  6. Rate limit quotas
  7. Lease count quotas
  8. Test to understand the resource quotas
  9. Clean up
  10. Next steps
Give Feedback(opens in new tab)
  • Certifications
  • System Status
  • Terms of Use
  • Security
  • Privacy
  • Trademark Policy
  • Trade Controls
  • Give Feedback(opens in new tab)