Vault
Vault usage metrics
Client calculation and sizing can be complex to compute when measuring Vault Enterprise usage. Vault provides telemetry metrics that can be monitored with solutions such as Prometheus and Grafana or Splunk; however, it requires additional configuration and tools.
Vault v1.6 introduced Vault Metrics dashboard in Web UI. This enhancements help to find active client count and historical client counts within Vault.
Policy requirements
To follow the steps in this tutorial, the following policies are required.
# To retrieve the usage metrics
path "sys/internal/counters/activity" {
capabilities = ["read"]
}
# To read and update the usage metrics configuration
path "sys/internal/counters/config" {
capabilities = ["read", "update"]
}
#---------------------------
# Vault Enterprise only
#---------------------------
# To view existing namespaces if any
path "sys/namespaces" {
capabilities = ["list", "read", "update"]
}
# UI to show the namespace selector
path "sys/internal/ui/namespaces" {
capabilities = ["read", "list", "update", "sudo"]
}
# UI to list existing mounts
path "sys/internal/ui/mounts" {
capabilities = ["read", "sudo"]
}
# To read and update the usage metrics configuration for any namespace
path "+/sys/internal/counters/config" {
capabilities = ["read", "update"]
}
# To retrieve the usage metrics for any namespace
path "+/sys/internal/counters/activity" {
capabilities = ["read"]
}
Enable usage metrics
Usage metrics are a feature that is enabled by default for Vault Enterprise but disabled for Vault Community Edition.
To enable usage tracking for Vault Community Edition, use the
sys/internal/counters/config
endpoint. (This step is NOT necessary for Vault
Enterprise.)
$ vault write sys/internal/counters/config enabled=enable
Valid values for enabled
parameter are: default
, enable
, and disable
.
Retention period
By default, Vault maintains activity logs to track
active clients for 24 months. If you wish to change the retention period, use
the retention_months
parameter.
Example:
$ vault write sys/internal/counters/config enabled=enable retention_months=12
Usage metrics dashboard
Sign into the Vault UI, and select Client Count.
The default view for usage metrics is for the current billing period.
Part of what contributes to Vault pricing is client usage. The Current month and History tabs display three client usage metrics: Total clients, Entity clients, and Non-entity clients. Here is an example of the metrics shown for the history.
The data are grouped into 3 distinct categories:
Running client total are the primary metric on which pricing is based. It is the sum of entity clients (or distinct entities) and non-entity clients.
Entity clients (distinct entities) are representations of a particular user, client, or application that belongs to a defined Vault entity. If you are unfamiliar with Vault entities, refer to the Identity: Entities and Groups tutorial.
Non-entity clients are clients without an entity attached. This is because some customers or workflows might avoid using entity-creating authentication methods and instead depend on token creation through the Vault API. Refer to understanding non-entity tokens to learn more.
Note
The non-entity client count excludes
root
tokens. Refer to the FAQs documentation which describes the difference between the entity clients and non-entity clients.Secrets sync clients are the number of external destinations Vault connects to sync the secrets. Refer to the documentation for more details.
ACME clients are the ACME connections that authenticate under the same certificate identifier (domain) as the same certificate entity for client count calculations. Refer to the documentation for more details.
Vault Enterprise Namespaces
If you have namespaces, the entity clients and non-entity clients are also shown as graphs per namespace. The Attribution section also displays the top namespace where you can expect to find your most used namespaces with respect to client usage (Vault 1.9 or later). This allows you to detect which namespace had the most number of total clients in the given billing period.
You can export the metrics data by clicking on the Export attribution data button.
This downloads the usage metrics data on your local drive in comma separated
values format (.csv
) containing all client count usage metrics by namespace.
Auth mount attribution
The clients are also shown as graphs per auth mount. The Attribution section also displays the top auth mounts where you can expect to find your most used auth mounts with respect to client usage (Vault 1.10 or later). This allows you to detect which auth mount had the most number of total clients in the given billing period. You can filter for auth mounts within a namespace, or view auth mounts across namespaces. The auth mount attribution functionality can also be used if you are not using namespaces at all. You can export the metrics data by clicking on the Export attribution data button.
Querying usage metrics
Under the Filters, select the time period, and select custom end month. This brings up a dialog where you can specify the month and year to retrieve data for a different billing period.
It is important to keep in mind that Vault begins collecting data when the feature is enabled. For example, if the feature is enabled in March of 2021, a query cannot be made in the UI for data in January of 2021.
Vault will return metrics from March of 2020 through most recent full month. To specify a different ending month, select the down-arrow next to the current billing period.
You can select Current billing period or Custom end month to edit the period.
Filter by namespaces (Vault Enterprise)
Vault Enterprise namespaces provide multi-tenancy to isolate Vault usage. You may want to view the client usage metrics per namespace.
Under Filters, you can enter the target namespace in the text field next to the billing period.
When you put focus on the text field, it displays existing namespaces.
Querying usage metrics via CLI or API
You can query the usage count metrics using Vault CLI or API.
Retrieve the usage metrics.
$ vault read -format=json sys/internal/counters/activity | jq -r ".data"
Example output:
{
"by_namespace": [
{
"counts": {
"clients": 2666,
"distinct_entities": 1564,
"entity_clients": 1564,
"non_entity_clients": 1102,
"non_entity_tokens": 1102
},
"mounts": [],
"namespace_id": "ai0nK",
"namespace_path": "education/certification/"
},
{
"counts": {
"clients": 2565,
"distinct_entities": 1297,
"entity_clients": 1297,
"non_entity_clients": 1268,
"non_entity_tokens": 1268
},
"mounts": [],
"namespace_id": "QbHPY",
"namespace_path": "education/training/security-edu/"
},
{
"counts": {
"clients": 2192,
"distinct_entities": 1257,
"entity_clients": 1257,
"non_entity_clients": 935,
"non_entity_tokens": 935
},
"mounts": [],
"namespace_id": "RFdWg",
"namespace_path": "education/training/security-edu/vault-edu/"
},
...snip...
],
"end_time": "2022-02-28T23:59:59Z",
"months": [],
"start_time": "2021-03-01T00:00:00Z",
"total": {
"clients": 13346,
"distinct_entities": 6675,
"entity_clients": 6675,
"non_entity_clients": 6671,
"non_entity_tokens": 6671
}
}
The output shows client usage metrics for each namespace.
Query with a time frame
To query the client usage metrics for the month of September, 2021. The start
time is September 1, 2021 (2021-09-01T00:00:00Z
) and the end time is September
30, 2021 (2021-09-30T23:59:59Z
).
The start_time
and end_time
should be an RFC3339 timestamp or Unix epoch
time.
$ vault read -format=json sys/internal/counters/activity \
start_time=2021-09-01T00:00:00Z \
end_time=2021-09-30T23:59:59Z | jq -r ".data"
Partial month client counts
Note
This feature requires Vault 1.10 or later.
Use the sys/internal/counters/activity/monthly
endpoint to get the number of
client counts per namespace for the current month.
$ vault read -format=json sys/internal/counters/activity/monthly | jq -r ".data"
Example output:
{
"by_namespace":[
{
"namespace_id":"0lHBL",
"namespace_path":"finance/",
"counts":{
"distinct_entities":0,
"non_entity_tokens":100,
"clients":100
}
},
{
"namespace_id":"RxD81",
"namespace_path":"marketing/",
"counts":{
"distinct_entities":5,
"non_entity_tokens":15,
"clients":20
}
},
{
"namespace_id":"root",
"namespace_path":"",
"counts":{
"distinct_entities":85,
"non_entity_tokens":15,
"clients":100
}
}
],
"distinct_entities": 90,
"non_entity_tokens": 130,
"clients": 220
}
Considerations for querying usage metrics
Data is collected at the end of each month. In order to display the data for a month in particular, the entire month has to pass.
In the heading, the UI will show the data that has been returned. For example, if you searched data from January 2021 but data is only available from March 2021, the heading will reflect that.
When running Vault 1.10.x or earlier and if there is missing data somewhere in your search (e.g. data is available from January 2020 through October 2020, but April is missing), Vault will return metrics from May through October 2020, because that is the most recent contiguous set. It does not mean the other data is unavailable.
Export activity log using API
Version requirement
This section introduces the Activity Export API introduced in Vault 1.11.0.
Vault's activity log is an internal system for tracking monthly client usage containing entity IDs or synthetic client IDs from non-entity tokens. Each client ID is only stored once per month. Previously, the activity log was only used internally to compute the count of clients.
As of Vault 1.11.0, the Activity Export API allows users to download the activity log based on the user-defined start time and end time.
A single client appears once in the output, and API uses the earliest activity in the time range for the client.
Example usage
Create your request payload specifying the start time and end time. Use RFC3339
timestamp or Unix epoch time to specify the start and end time. You can also
specify the output format. Currently available format is CSV or JSON (csv
or
json
).
The following request payload specifies the start time to January 1, 2022 and the end time to be June 1, 2022. The output should be in CSV format.
$ tee payload.json <<EOF
{
"start_time": "1641024000",
"end_time": "1654066800",
"format": "csv"
}
EOF
Invoke the /sys/internal/counters/activity/export
endpoint and store the
output in a file, /tmp/activity-export.csv
.
$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
$VAULT_ADDR/v1/sys/internal/counters/activity/export > /tmp/activity-export.csv
Refer to the Activity Export API documentation as well.
Version-specific client count metrics details
You should be aware that there are significant nuances in how client count metrics are computed between versions of Vault at or beyond version 1.9.0, and versions between 1.6.0 and 1.9.0.
Important details on these nuances are noted by version.
Vault 1.6 | Vault 1.8 | Vault 1.9 |
---|---|---|
Client count metrics features introduced. | Entity tracking is done on access instead of on creation. | Change in non-entity client computation logic. For non-entity tokens (where there is no entity to which tokens map) Vault uses the contents of the token to generate a unique client identifier, based on the namespace ID and and policies. The clientID will prevent the same token from being duplicated in the overall client count. Non-entity token tracking is done on access instead of creation. |
Non entity tokens counted at create time. | Wrapped tokens and control groups are not counted as clients. | root tokens are not counted towards client count aggregate. |
Wrapped tokens and control groups are counted as clients. | Local auth mount accessors count towards clients and as non-entity tokens | Local auth mount accessors do count towards clients but not as non-entity tokens. |
Change in non-entity client computation logic. For non-entity tokens (where there is no entity to which tokens map) Vault uses the contents of the token to generate a unique client identifier, based on the namespace ID and and policies. The clientID will prevent the same token from being duplicated in the overall client count. | Local auth mount accessors count towards clients and as non-entity tokens. | Display of clients per namespace (top 10) in the UI and the ability to export data for all namespaces. |
For additional details about how clients are computed, refer to What is a Client?.
Vault auditor tool
If you are running at least Vault version 1.3, you can use the
vault-auditor
tool to extract metrics from the server audit device logs. These
metrics are typically useful for metering and billing, but also helpful for
getting critical entity and token counts for troubleshooting scenarios.
Download vault-auditor binaries for macOS and Linux, and you can use the tool wherever you have access to the audit device log files.
Note
vault-auditor
currently supports only audit device logs from Vault versions 1.3 through 1.5.
Here is an example interaction with vault-auditor
, beginning with help output and concluding with an example where the tool is used to parse a hypothetical directory of audit devices logs.
Get help for vault-auditor
with the --help
flag.
$ vault-auditor --help
Usage: vault-auditor [--version] [--help] <command> [<args>]
Available commands are:
parse Parse audit logs to find entity and token counts
version Prints the vault-auditor version
Currently, the tool features just two sub-commands, parse
and version
; check the help for the parse
sub-command to learn more about how it works.
$ vault-auditor parse --help
Usage: vault-auditor parse [options] <directory>
Parse one or more audit logs inside the given directory
to count unique tokens.
$ vault-auditor parse /tmp/audit-logs
Options:
-prefix=<string>
String that every line in the audit logs is prefixed with.
This should only be necessary if the audit prefix contains a { character.
-start-date=<string>
Only count log lines on/after this date.
RFC3339, YYYY-MM-DD, YYYY-MM, and Unix epoch time formats are supported.
-end-date=<string>
Only count log lines on/before this date.
RFC3339, YYYY-MM-DD, YYYY-MM, and Unix epoch time formats are supported.
Note in the help output, that vault-auditor
expects to be executed against a directory containing Vault audit device log files. It supports three optional flags for handling prefixed log entries and controlling the start and end ranges to operate on.
The most basic use of vault-auditor
is to execute it and pass in the full path to a directory containing one or more Vault audit device log files. In this example, the logs are contained in an audit-device-logs
directory
$ vault-auditor parse audit-logs
The parsing of logs will emit progress indicators and when successful, the output resembles this example.
vault-zus1-0 [=======================================] 100 % 0s
vault-zus1-1 [=======================================] 100 % 0s
vault-zus1-2 [=======================================] 100 % 0s
vault-zus1-3 [=======================================] 100 % 0s
vault-zus1-4 [=======================================] 100 % 0s
Distinct Entities: 7139
Non-Entity Tokens: 273
Total Clients: 7412
Total files processed: 5
Date range: 2020-10-02T08:36:29Z - 2020-10-07T15:57:50Z
There were a total of 5 logs processed from the audit-logs
directory, with a total of 273 non-entity tokens identified, meaning tokens which were created directly from the Token auth method (such as root tokens). The tool also found 7139 distinct entities for which tokens were issued through a Vault auth method. The total clients count is a combination of distinct entities and non-entity tokens.
Lastly, the date range represented by the parsed logs is shown.
Limits of the auditor tool
The vault auditor tool was released along with Vault v1.6.0.
Since that time, some improvements were made to the Vault client count computation logic in newer versions of Vault. These changes are reflected only in the usage metrics API and web UI, but are not available in the auditor tool.
The following specific improvements are not reflected in the auditor output.
Vault v1.8
- Eliminated wrapped tokens and control groups from client count thereby reducing the non-entity token count. Prior to 1.8, the creation and usage of control groups and wrapping tokens incremented the client count via non-entity tokens every time the response was read in the case of a wrapping token, and every time a control group was created.
- Changed logic of counting of non entity tokens on usage instead of at create time therefore resulting in more accurate client counts.
Vault 1.9
- Changed the non-entity token computation logic to de-duplicate non-entity tokens as described in the Version Specific Client Count Metrics Details section.
- Changed the local auth mount computation logic such that local auth mounts count towards clients but not as non-entity tokens. Prior to v1.9, local auth mounts counted towards non-entity tokens.
- New functionality to display clients per namespace (top 10) in the UI and the ability to export data for all namespaces. Prior to 1.9, there was no ability to view the split of clients per namespace on the UI, and there was no ability to export this data via the UI.
Learn more from the FAQ
Which version of Vault reflects the most accurate client counts?.
KMIP client metrics
To perform the tasks described in this section, you need to have a Vault Enterprise v1.7 or later with Advanced Data Protection module.
Vault Enterprise 1.7 introduced improvements to the audit device metrics to help users understand usage and scalability of the KMIP secrets engine for different workflows.
When you parse the Vault Enterprise audit logs using the vault-auditor
tool, pass the -kmip-info
flag to get the metrics specific to the KMIP secrets engine. The output will include the number of KMIP certificates, active KMIP certificates, and managed objects.
$ vault-auditor parse -kmip-info audit_log
Distinct Entities: 14
Non-Entity Tokens: 1
Total Clients: 15
Total files processed: 1
Date range: 2021-03-20T20:11:41Z - 2021-03-23T22:56:17Z
Distinct KMIP Certificates: 24
Distinct Active KMIP Certificates: 20
Distinct KMIP Managed Objects: 20
From the example output, you can observe that there are 24 distinct KMIP certificates issued by Vault, with 20 of those active during the date range of the audit device log, along with a total of 20 KMIP managed objects.
Recent changes
Vault 1.17
When you are running PKI secrets engine to issue and renew leaf server certificate with Automatic Certificate Management Environment (ACME) protocol, usage metrics dashboard reports the ACME clients separately for license utilization reporting.
Vault 1.16
Usage metric dashboard tracks the secrets sync client counts. When you use Vault's secret sync feature to automatically update secrets in external destination, Vault considers the destination as a secret sync client. Refer to the documentation for more details.
Vault 1.11
Vault 1.11 introduced Activity Export API allowing you to download the client count aggregated for the user-defined billing period. Refer to the Export activity log using API section for more detail.
Vault 1.10
Starting Vault 1.10, usage metrics dashboard refers to the non-entity tokens as non-entity clients, and unique entities as entity clients.
Old term (v1.9 and earlier) | New term (v1.10 and later) |
---|---|
Total active clients | Total clients |
Unique entities | Entity clients |
Non-entity tokens | Non-entity clients |
Previously, if there is missing data within your billing period (e.g. data is available from January 2021 through October 2021, but April is missing), Vault returned metrics from May through October 2021, because that is the most recent contiguous set. As of Vault 1.11.0, Vault returns all available data within the specified billing period.
Vault 1.9
Starting in Vault 1.9, Vault changed the non-entity token computation logic to
deduplicate non-entity tokens. For non-entity tokens (where there is no entity
to which tokens map) Vault uses the contents of the token to generate a unique
client identifier, based on the namespace ID and policies. The clientID
will
prevent the same token from being duplicated in the overall client count.
Non-entity token tracking is done on access instead of creation.
Help and reference
If you are running Vault v1.9 or later, refer to the version-specific
client count metrics details
section to learn about key differences in the computation of client usage
metrics between versions. For Vault v1.3.x through v1.5.x, refer to the
Vault auditor tool section on how to use the
vault-auditor
tool to extract the usage metrics.
In addition, refer to the following documentation: