Vault
Manage secrets across namespaces
Enterprise Feature
Vault namespaces feature is a part of Vault Enterprise.
This tutorial demonstrates the capability introduced in Vault Enterprise 1.13.0. If you are not familiar with Vault Namespaces, complete the Secure Multi-Tenancy with Namespaces tutorial first. In addition, the Vault Namespace and Mount Structuring Guide provides some guidance on designing the namespace hierarchy.
Challenge
The purpose of Vault namespaces is to create an isolated Vault environment within a cluster so that each organization, team, or application can manage secrets independently. Therefore, Vault clients must authenticate into a specific target namespace where the secrets live.

In some use cases, this imposes a burden on the Vault clients especially if the client must read secrets from dozens of independent namespaces (namespaces with no hierarchical relationship). This operational overhead becomes more prominent when you are using Vault Agent to automate secrets management.
Tip
When namespaces have hierarchical relationship, you can write namespace-aware policies to allow access to secrets in the child namespaces.
Solution
Vault Enterprise 1.13.0 introduced the group_policy_application_mode
flag
which enables secrets sharing across multiple independent namespaces. With this
change, a single instance of the Vault Agent can fetch secrets across multiple
namespaces. (See Cross namespace secret sharing
for an Agent injector example.)

Tutorial example use case
This tutorial demonstrates the secrets management across namespaces
funcationality using the userpass
auth method, however, all auth methods are
supported. Refer to the configure cross namespace access
documentation
for more information.
Prerequisites
- Familiarity with Vault namespaces or completed the Secure Multi-Tenancy with Namespaces tutorial
- Vault Enterprise binary v1.13.0 or later
Lab setup
Note
If you do not have access to an HCP Vault Dedicated cluster, visit the Create a Vault Cluster on HCP tutorial.
Launch the HCP Portal and login.
Click Vault in the left navigation pane.
In the Vault clusters pane, click vault-cluster.
Under Cluster URLs, click Public Cluster URL.
In a terminal, set the
VAULT_ADDR
environment variable to the copied address.$ export VAULT_ADDR=<Public_Cluster_URL>
Return to the Overview page and click Generate token.
Copy the Admin Token.
Return to the terminal and set the
VAULT_TOKEN
environment variable.$ export VAULT_TOKEN=<token>
Set the
VAULT_NAMESPACE
environment variable toadmin
.$ export VAULT_NAMESPACE=admin
The
admin
namespace is the top-level namespace automatically created by HCP Vault. All CLI operations default to use the namespace defined in this environment variable.Type
vault status
to verify your connectivity to the Vault cluster.$ vault status Key Value --- ----- Recovery Seal Type shamir Initialized true Sealed false Total Recovery Shares 1 Threshold 1 Version 1.9.2+ent Storage Type raft ...snipped...
The Vault Dedicated server is ready.
Scenario setup
To demonstrate the secrets sharing across namespaces, you need the following:
- New namespaces:
us-west-org
andus-east-org
- In the
us-west-org
namespace:- Enabled userpass auth method and created a "tam-user" with
customer-info-read-only
policy attached. The password is "my-long-password" - Enabled KV v2 secrets engine at
kv-customer-info
with some test data - Create a
customer-info-read-only
policy allowing to read secrets atkv-customer-info/data/*
- Created an identity entity, "TAM" and added "tam-user" as a member (an entity alias)
- Enabled userpass auth method and created a "tam-user" with
- In the
us-east-org
namespace:- Enabled KV v2 secrets engine at
kv-marketing
with some test data - Created a marketing-read-only policy allowing read permission on the
kv-marketing/data/campaign
path - Create an entity group named "campaign-admin" with the marketing-read-only policy attached
- Enabled KV v2 secrets engine at
Tip
If you are not familiar with entities and groups, read the Identity: Entities and Groups tutorial.
Use your preferred text editor, paste in the following to create a script to setup the Vault elements needed for demonstration.
lab-setup.sh
# Create new namespaces - they are peer vault namespace create us-west-org vault namespace create us-east-org #-------------------------- # us-west-org namespace #-------------------------- VAULT_NAMESPACE=admin/us-west-org vault secrets enable -path="kv-customer-info" kv-v2 VAULT_NAMESPACE=admin/us-west-org vault kv put kv-customer-info/customer-001 name="Example LLC" contact_email="admin@example.com" # Create a policy to allow read access to kv-marketing VAULT_NAMESPACE=admin/us-west-org vault policy write customer-info-read-only -<<EOF path "kv-customer-info/data/*" { capabilities = ["read"] } EOF VAULT_NAMESPACE=admin/us-west-org vault auth enable userpass VAULT_NAMESPACE=admin/us-west-org vault write auth/userpass/users/tam-user password="my-long-password" policies=customer-info-read-only # Create an entity VAULT_NAMESPACE=admin/us-west-org vault auth list -format=json | jq -r '.["userpass/"].accessor' > accessor.txt VAULT_NAMESPACE=admin/us-west-org vault write -format=json identity/entity name="TAM" | jq -r ".data.id" > entity_id.txt VAULT_NAMESPACE=admin/us-west-org vault write identity/entity-alias name="tam-user" canonical_id=$(cat entity_id.txt) mount_accessor=$(cat accessor.txt) #-------------------------- # us-east-org namespace #-------------------------- VAULT_NAMESPACE=admin/us-east-org vault secrets enable -path="kv-marketing" kv-v2 VAULT_NAMESPACE=admin/us-east-org vault kv put kv-marketing/campaign start_date="March 1, 2023" end_date="March 31, 2023" prize="Certification voucher" quantity="100" # Create a policy to allow read access to kv-marketing VAULT_NAMESPACE=admin/us-east-org vault policy write marketing-read-only -<<EOF path "kv-marketing/data/campaign" { capabilities = ["read"] } EOF # Create a group VAULT_NAMESPACE=admin/us-east-org vault write -format=json identity/group name="campaign-admin" policies="marketing-read-only" member_entity_ids=$(cat entity_id.txt)
Change the file permission to make it executable.
$ chmod +x lab-setup.sh
Run the script.
$ ./lab-setup.sh
Enable secrets sharing
Vault 1.13.0 introduced a new API endpoint,
sys/config/group-policy-application
which sets the
group_policy_application_mode
parameter to allow Vault clients to manage
secrets across multiple namespaces.
Look up the current setting.
$ vault read sys/config/group-policy-application Key Value --- ----- group_policy_application_mode within_namespace_hierarchy
The default setting is
within_namespace_hierarchy
.Set the parameter value to
any
.$ vault write sys/config/group-policy-application \ group_policy_application_mode="any"
Output:
Success! Data written to: sys/config/group-policy-application
Read it back to verify.
$ vault read sys/config/group-policy-application Key Value --- ----- group_policy_application_mode any
Verification: Read secrets
Log in as "tam-user" and store the returned token in token.txt. Remember that auth methods are tied to the namespace they are configured. The "tam-user" is a user created in the userpass auth method in the us-west-org namespace.
$ VAULT_NAMESPACE=admin/us-west-org vault login -no-print -field=token \ -method=userpass username=tam-user > token.txt Password (will be hidden):
Enter the password
my-long-password
when prompted, and pressreturn
.Tip
You can ignore the returned warning about the VAULT_TOKEN environment variable to take precedence.
Check to see if you can read the
kv-marketing/data/campaign
in the us-east-org namespace using the generated token.$ VAULT_NAMESPACE=admin/us-east-org VAULT_TOKEN=$(cat token.txt) \ vault kv get kv-marketing/campaign
Output:
======= Secret Path ======= kv-marketing/data/campaign ======= Metadata ======= Key Value --- ----- created_time 2023-02-17T04:46:03.22039Z custom_metadata <nil> deletion_time n/a destroyed false version 1 ======= Data ======= Key Value --- ----- end_date March 31, 2023 prize Certification voucher quantity 100 start_date March 1, 2023
You can read the secrets in the
us-east-org
namespace.(Optional) The "tam-user" token has the
customer-info-read-only
policy attached. Therefore, you can read the secrets atkv-customer-info/
in the us-west-org namespace.$ VAULT_NAMESPACE=admin/us-west-org VAULT_TOKEN=$(cat token.txt) \ vault kv get kv-customer-info/customer-001
Example output:
=========== Secret Path =========== kv-customer-info/data/customer-001 ======= Metadata ======= Key Value --- ----- created_time 2023-02-17T17:47:06.147303Z custom_metadata <nil> deletion_time n/a destroyed false version 1 ======== Data ======== Key Value --- ----- contact_email admin@example.com name Example LLC
Revert back the parameter setting
Set the
group_policy_application_mode
parameter value back towithin_namespace_hierarchy
.$ vault write sys/config/group-policy-application \ group_policy_application_mode="within_namespace_hierarchy"
Try to read the secrets again.
$ VAULT_NAMESPACE=admin/us-east-org VAULT_TOKEN=$(cat token.txt) \ vault kv get kv-marketing/campaign
Output:
Error making API request. Namespace: us-east-org/ URL: GET http://127.0.0.1:8200/v1/sys/internal/ui/mounts/kv-marketing/campaign Code: 403. Errors: * preflight capability check returned 403, please ensure client's policies grant access to path "kv-marketing/campaign/"
When the
group_policy_application_mode
parameter is set towithin_namespace_hierarchy
, you cannot access the secrets in the namespaces that has no hierarchical relationship.
Clean up
If you used the Bash script to set up the scenario, manually delete the regional namespaces to keep the cluster, or delete the cluster if it was only used for this tutorial.
If you used Terraform to set up the scenario, follow the steps below to keep the cluster, or delete the cluster if it was only used for this tutorial.
Make sure that
VAULT_TOKEN
,VAULT_NAMESPACE
, andVAULT_ADDR
environment variables are set.$ echo $VAULT_ADDR $VAULT_TOKEN $VAULT_NAMESPACE
Destroy the Vault resources created by Terraform.
$ terraform destroy -auto-approve ...snip... Destroy complete! Resources: 14 destroyed.
Remove the Terraform state files.
$ rm *.tfstate.*
Unset the
VAULT_TOKEN
andVAULT_ADDR
environment variables.$ unset VAULT_TOKEN VAULT_ADDR VAULT_NAMESPACE
Remove the entity / token / accessor files.
$ rm accessor.txt entity_id.txt token.txt
Summary
This tutorial demonstrated the new API endpoint,
sys/config/group-policy-application
and its group_policy_application_mode
parameter introduced in Vault 1.13.0 to allow Vault clients to manage secrets
across multiple independent namespaces.
Although secrets engines, auth methods, policies, and tokens are tied to each namespaces, the entity group can pull entities and groups from other namespaces. If you are not familiar with entities and groups, read the Identity: Entities and Groups tutorial.