Overview

The Vault 1.18.x upgrade guide contains information on deprecations, important or breaking changes, and remediation recommendations for anyone upgrading from Vault 1.17. Please read carefully.

Important changes

Strict validation for Azure auth login requests

Azure auth plugin requires resource_group_name, vm_name, and vmss_name to match the JWT claims on login.

Vault versions before 1.18.7 did not strictly validate the resource_group_name, vm_name, and vmss_name parameters against their token claims for clients logging in with Azure authentication.

Recommendation

Review the Token validation section of the Azure authN plugin guide for more information on the new validation requirements.

Activity Log Changes

Default Activity Log Querying Period

The field default_report_months can no longer be configured or read. Any previously set values will be ignored by the system.

Attempts to modify default_report_months through the /sys/internal/counters/config endpoint, will result in the following warning from Vault:

WARNING! The following warnings were returned from Vault:

  * default_report_months is deprecated: defaulting to billing start time

The current_billing_period toggle for /sys/internal/counters/activity is also deprecated, as this will be set true by default.

Attempts to set current_billing_period will result in the following warning from Vault:

WARNING! The following warnings were returned from Vault:

  * current_billing_period is deprecated; unless otherwise specified, all requests will default to the current billing period

Docker image no longer contains curl

The curl binary is no longer included in the published Docker container images for Vault and Vault Enterprise. If your workflow depends on curl being available in the container, consider one of the following strategies:

Create a wrapper container image

Use the HashiCorp image as a base image to create a new container image with curl installed.

FROM hashicorp/vault-enterprise
RUN apk add curl

NOTE: While this is the preferred option it will require managing your own registry and rebuilding new images.

Install it at runtime dynamically

When running the image as root (not recommended), you can install it at runtime dynamically by using the apk package manager:

docker exec <CONTAINER-ID> apk add curl

kubectl exec -ti <NAME> -- apk add curl

When running the image as non-root without privilege escalation (recommended) you can use existing tools to install a static binary of curl into the vault users home directory:

docker exec <CONTAINER-ID> wget https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64 -O /home/vault/curl && chmod +x /home/vault/curl

kubectl exec -ti <NAME> -- wget https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64 -O /home/vault/curl && chmod +x /home/vault/curl

NOTE: When using this option you'll want to verify that the static binary comes from a trusted source.

Request limiter configuration removal

Vault 1.16.0 included an experimental request limiter. The limiter was disabled by default with an opt-in request_limiter configuration.

Further testing indicated that an alternative approach improves performance and reduces risk for many workloads. Vault 1.17.0 included a new adaptive overload protection feature that prevents outages when Vault is overwhelmed by write requests.

Adaptive overload protection was a beta feature in 1.17.0.

As of Vault 1.18.0, the adaptive overload protection feature for writes is now GA and enabled by default for the integrated storage backend.

The beta request_limiter configuration stanza is officially removed in Vault 1.18.0.

Vault will output two types of warnings if the request_limiter stanza is detected in your Vault config.

1. A UI warning message printed to stderr:

WARNING: Request Limiter configuration is no longer supported; overriding server configuration to disable

1. A log line with level WARN, appearing in Vault's logs:

... [WARN]  unknown or unsupported field request_limiter found in configuration at config.hcl:22:1

Product usage reporting

As of 1.18.2, Vault will collect anonymous product usage metrics for HashiCorp. This information will be collected alongside client activity data, and will be sent automatically if automated reporting is configured, or added to manual reports if manual reporting is preferred.

See the main page for Vault product usage metrics reporting for more details, and information about opt-out.

Known issues and workarounds

Seal/Seal Wrapped - Duplicate HSM Keys

Affected Versions

• All versions that support migration from Shamir to HSM-backed unseal/seal wrap in HSM-HA configurations.

Issue

During a migration from Shamir to an HSM-backed unseal configuration with HSM - High Availability (HA), duplicate HSM keys may be created. These issues can occur even after a seal migration to HSM that initially appeared successful. The root cause is under investigation, with potential links to key handling during HA configuration or migration processes.

• Unseal failures: Nodes may fail to unseal after a restart, with errors such as CKR_DATA_INVALID.
• Duplicate HSM keys: These may be created, resulting in intermittent read failures with errors such as CKR_SIGNATURE_INVALID and CKR_KEY_HANDLE_INVALID for any seal wrapped value - see /vault/docs/enterprise/sealwrap#wrapped-parameters.

Workaround

As a workaround, always run Vault with generate_key = false, creating the required keys within the HSM manually during the setup process.

Database and LDAP Secrets Engine - Unwanted secret rotation on backend restart

Affected Versions

• All versions that support skipping automatic import rotation of static roles.

Issue

The Database secrets engine and the LDAP secrets engine both support skipping automatic import rotation of static roles with the config-level field skip_static_role_import_rotation and the static role-level field skip_import_rotation.

Any static roles configured with either of these fields set to true will have their passwords rotated when Vault is restarted. The password rotation will not occur on restart if the static role has already had its password rotated.

Workaround

Upgrade to 1.18.9+, 1.19.3+. For previous versions, the only workaround is to avoid restarting the backend.

Authorization failures using Azure federated identity credentials

A regression was introduced when attempting to fix automated unsealing using managed identity based credentials. The regression occurs when using Azure federated identity credentials supplying the client ID via an environment variable or through configuration while not supplying the tenant id and client secret.

The regression causes authorization failures within auto-unseal, managed keys and seal wrapping.

Workaround

None at this time.

Impacted versions

Affects 1.19.0-rc1, 1.18.5, 1.17.12, 1.16.16

LDAP static role rotations on upgrade

Affected Versions

• 1.19.0, 1.19.1
• 1.18.5, 1.18.6, 1.18.7
• 1.17.12, 1.17.13, 1.17.14
• 1.16.16, 1.16.17, 1.16.18

Fixed Versions

• 1.19.3+
• 1.18.9+
• 1.17.16+
• 1.16.20+

Issue

Vault automatically rotates existing static roles tied to LDAP credentials once when upgrading to an affected version. After the one-time rotation, the static roles behave as expected.

Workaround

If you rely on LDAP static roles, avoid upgrading to the affected versions.

Database static role rotations on upgrade

Affected Roles

Any database static role that was created prior to Vault 1.15.0 will be affected upon upgrading to the following Vault versions:

Affected Versions:

• 1.19.0, 1.19.1, 1.19.2
• 1.18.5, 1.18.6, 1.18.7, 1.18.8
• 1.17.12, 1.17.13, 1.17.14, 1.17.15
• 1.16.16, 1.16.17, 1.16.18, 1.16.19

Issue

Vault will automatically rotate static database credentials once, for all roles created prior to 1.15.0, when upgrading to affected versions. After the one-time rotation, the static roles behave as expected.

Fixed Versions:

• 1.19.3+
• 1.18.9+
• 1.17.16+
• 1.16.20+

Vault log file missing subsystem logs

Fixed Versions

• 1.16.18, 1.17.14, 1.18.7, 1.19.1

Issue

Vault configured with log_file is not capturing all logs. Some entries, including plugin logs, are missing from the output file; however, they appear as expected in standard error and standard output.

Workaround

Upgrade to fixed versions.

Azure Auth fails to authenticate Uniform VMSS instances

Affected versions

• 1.19.1+
• 1.18.7+
• 1.17.14+
• 1.16.18+

Issue

Azure Authentication from a uniform VMSS instance with a user assigned managed identity on the VMSS incorrectly results in an error. There was a regression introduced from adding validations on the JWT claims against the provided VM, VMSS, and resource group names without accounting for the uniform VMSS format.

Workaround

None at this time.

External Enterprise plugins cannot run on a standby node when it becomes active

External Enterprise plugins can't run on a standby node when it becomes active because standby nodes don't extract the artifact when the plugin is registered.

Recommendation

As a workaround, add the plugin .zip artifact on every node and register the plugin on the active node. Then, extract the contents of the zip file on the follower nodes similar to the following folder structure for vault-plugin-secrets-keymgmt_0.16.0+ent_darwin_arm64.zip.

<plugin-directory>/vault-plugin-secrets-keymgmt_0.16.0+ent_darwin_arm64
├── metadata.json
├── metadata.json.sig
└── vault-plugin-secrets-keymgmt

Alternatively, upgrade to one of the following Vault versions: 1.16.21+, 1.17.17+, 1.18.10+, 1.19.4+. See Register external plugins for more details.