• HashiCorp Developer

  • HashiCorp Cloud Platform
  • Terraform
  • Packer
  • Consul
  • Vault
  • Boundary
  • Nomad
  • Waypoint
  • Vagrant
Vault
  • Install
  • Tutorials
  • Documentation
  • API
  • Try Cloud(opens in new tab)
  • Sign up
Enterprise

Skip to main content
23 tutorials
  • Install a HashiCorp Enterprise License
  • Secure Multi-Tenancy with Namespaces
  • Vault Namespace and Mount Structuring Guide
  • Move Secrets Engines and Auth Methods Across Namespaces
  • Disaster Recovery Replication Setup
  • Disaster Recovery Replication Failover and Failback
  • Performance Standby Nodes
  • Setting up Performance Replication
  • Performance Replication with Paths Filter
  • Monitoring Vault Replication
  • Protecting Vault with Resource Quotas
  • Codify Management of Vault Enterprise Using Terraform
  • PKI Secrets Engine with Managed Keys
  • Sentinel Policies
  • Sentinel HTTP Import
  • Control Groups
  • Transform Secrets Engine
  • Tokenize Data with Transform Secrets Engine
  • KMIP Secrets Engine
  • Key Management Secrets Engine with Azure Key Vault
  • Key Management Secrets Engine with GCP Cloud KMS
  • HSM Integration - Seal Wrap
  • HSM Integration - Entropy Augmentation

  • Resources

  • Tutorial Library
  • Certifications
  • Community Forum
    (opens in new tab)
  • Support
    (opens in new tab)
  • GitHub
    (opens in new tab)
  1. Developer
  2. Vault
  3. Tutorials
  4. Enterprise
  5. Secure Multi-Tenancy with Namespaces

Secure Multi-Tenancy with Namespaces

  • 27min

  • EnterpriseEnterprise
  • VaultVault

Enterprise Only: The namespaces feature requires Vault Enterprise Standard license, or run HCP Vault.

Everything in Vault is path-based, and often uses the terms path and namespace interchangeably. The application namespace pattern is a useful construct for providing Vault as a service to internal customers, giving them the ability to implement secure multi-tenancy within Vault in order to provide isolation and ensure teams can self-manage their own environments.

Personas

The scenario described in this tutorial introduces the following personas:

  • operations is the cluster-level administrator with privileged policies
  • org-admin is the organization-level administrator
  • team-admin is the team-level administrator

Challenge

When Vault is primarily used as a central location to manage secrets, multiple organizations within a company may need to be able to manage their secrets in a self-serving manner. This means that a company needs to implement a Vault as a Service model allowing each organization (tenant) to manage their own secrets and policies. The most importantly, tenants should be restricted to work only within their tenant scope.

Multi-Tenant

Solution

Create a namespace dedicated to each team, organization, or app where they can perform all necessary tasks within their tenant namespace.

Each namespace can have its own:

  • Policies
  • Auth Methods
  • Secrets Engines
  • Tokens
  • Identity entities and groups

Tokens are locked to a namespace. Identity groups can pull in entities and groups from other namespaces.

Prerequisites

To perform the tasks described in this tutorial, you need to have a Vault Enterprise environment.

NOTE: The creation of namespaces should be performed by a user with a highly privileged token such as root to set up isolated environments for each organization, team, or application.

Lab setup

Open a terminal and start a Vault dev server with root as the root token.

$ vault server -dev -dev-root-token-id root

The Vault dev server defaults to running at 127.0.0.1:8200. The server is also initialized and unsealed.

Insecure operation: Do not run a Vault dev server in production. This approach is only used here to simplify the unsealing process for this demonstration.

Export an environment variable for the vault CLI to address the Vault server.

$ export VAULT_ADDR=http://127.0.0.1:8200

Export an environment variable for the vault CLI to authenticate with the Vault server.

$ export VAULT_TOKEN=root

The Vault server is ready.

Scenario introduction

In this tutorial, you are going to create a namespace dedicated to the Education organization which has Training and Certification teams. Delegate operational tasks to the team admins so that the Vault cluster operators won't have to be involved.

Scenario

The following steps are demonstrated:

  1. Create namespaces
  2. Write policies
  3. Set up entities and groups
  4. Test the organization admin user
  5. Test the team admin user
  6. Audit ambient credentials

Create namespaces

(Persona: operations)

Important Notes: This tutorial focuses on the use of Vault namespaces. Refer to Vault Limits and Maximums to understand the known upper limits on namespaces. Also, read Vault Namespace and Mount Structuring Guide for additional guidance on structuring Vault namespaces.

To create a new namespace, run: vault namespace create <namespace_name>

  1. Create a namespace dedicated to the education organizations.

    $ vault namespace create education
    
    Key                Value
    ---                -----
    custom_metadata    map[]
    id                 72cdg
    path               education/
    

    In this example, the education namespace ID is 72cdg.

  2. Create child namespaces called training under the education namespace.

    $ vault namespace create -namespace=education training
    
    Key                Value
    ---                -----
    custom_metadata    map[]
    id                 PjzDJ
    path               education/training/
    
  3. Create child namespaces called certification under the education namespace.

    $ vault namespace create -namespace=education certification
    
    Key                Value
    ---                -----
    custom_metadata    map[]
    id                 PdrIi
    path               education/certification/
    
  4. List the existing namespaces on the root.

    $ vault namespace list
    
    Keys
    ----
    education/
    
  5. List the existing namespace on the education namespace.

    $ vault namespace list -namespace=education
    
    Keys
    ----
    certification/
    training/
    

Custom metadata

NOTE: This section is informational. You can proceed to the Write policies section without performing the command below.

If you are running Vault 1.12.0 or later, you can add custom metadata to each namespace using -custom-metadata flag. Custom metadata can be useful when you have machine-generated namespace names. You can set custom metadata to provide human-friendly tags or names to the namespace.

$ vault namespace create -custom-metadata=team="Marketing (US)" ns-mktg-usa

Eaxmple output:

Key                Value
---                -----
custom_metadata    map[team:Marketing (US)]
id                 vZ5YB
path               ns-mktg-usa/

You can add custom metadata to an existing namespaces using the patch sub-command.

$ vault namespace patch -custom-metadata=language=English ns-mktg-usa

Key                Value
---                -----
custom_metadata    map[language:English team:Marketing (US)]
id                 vZ5YB
path               ns-mktg-usa/

You can pass multiple -custom-metadata=<key>=<value> to set more than one custom metadata with vault namespace create or vault namespace patch commands.

For example:

$ vault namespace create -custom-metadata=team="Marketing (US)" \
   -custom-metadata=language=English \
   -custom-metadata=alias="team-marketing" \
   ns-mktg-usa

NOTE: This section uses jq to process the JSON output for readability.

  1. Create a namespace for the education organization.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
          --request POST \
          $VAULT_ADDR/v1/sys/namespaces/education | jq -r ".data"
    

    Example output:

    {
       "custom_metadata": {},
       "id": "e0bcu",
       "path": "education/"
    }
    
  2. Now, create a namespace called training under education. To do so, pass the top-level namespace name in the X-Vault-Namespace header.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
          --header "X-Vault-Namespace: education" \
          --request POST \
          $VAULT_ADDR/v1/sys/namespaces/training | jq -r ".data"
    

    Example output:

    {
       "custom_metadata": {},
       "id": "SYIaz",
       "path": "education/training/"
    }
    
  3. Similarly, create a certification namespace under education namespace.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
          --header "X-Vault-Namespace: education" \
          --request POST \
          $VAULT_ADDR/v1/sys/namespaces/certification | jq -r ".data"
    

    Example output:

    {
       "custom_metadata": {},
       "id": "mLUfb",
       "path": "education/certification/"
    }
    
  4. List the existing namespaces on the root level.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
          --request LIST \
          $VAULT_ADDR/v1/sys/namespaces | jq -r ".data"
    

    Example output:

    {
       "key_info": {
          "education/": {
             "custom_metadata": {},
             "id": "e0bcu",
             "path": "education/"
          }
       },
       "keys": [
          "education/"
       ]
    }
    
  5. List the existing namespaces under the education namespace.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
          --request LIST \
          $VAULT_ADDR/v1/education/sys/namespaces | jq -r ".data"
    

    Example output:

    {
       "key_info": {
          "certification/": {
             "custom_metadata": {},
             "id": "mLUfb",
             "path": "education/certification/"
          },
          "training/": {
             "custom_metadata": {},
             "id": "SYIaz",
             "path": "education/training/"
          }
       },
       "keys": [
          "certification/",
          "training/"
       ]
    }
    

Custom metadata

NOTE: This section is informational. You can proceed to the Write policies section without performing the command below.

If you are running Vault 1.12.0 or later, you can add custom metadata to each namespace. Custom metadata can be useful when you have machine-generated namespace names. You can set custom metadata to provide human-friendly tags or names to the namespace.

Use custom_metadata in the API request payload.

$ tee payload.json <<EOF
{
   "custom_metadata": {
      "team": "Marketing (US)"
   }
}
EOF

Create a ns-mktg-usa namespace.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload.json \
      $VAULT_ADDR/v1/sys/namespaces/ns-mktg-usa | jq -r ".data"

Eaxmple output:

{
  "custom_metadata": {
    "team": "Marketing (US)"
  },
  "id": "9qb1y",
  "path": "ns-mktg-usa/"
}

You can add custom metadata to an existing namespaces using the PATCH verb.

Modify the request payload.

$ tee payload.json <<EOF
{
   "custom_metadata": {
      "language": "English"
   }
}
EOF

Now, invoke the API.

$ curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --header "Content-Type: application/merge-patch+json" \
      --request PATCH \
      --data @payload.json \
      $VAULT_ADDR/v1/sys/namespaces/ns-mktg-usa | jq -r ".data"

Eaxmple output:

{
  "custom_metadata": {
    "language": "English",
    "team": "Marketing (US)"
  },
  "id": "9qb1y",
  "path": "ns-mktg-usa/"
}

You can pass multiple -custom-metadata=<key>=<value> to set more than one custom metadata with vault namespace create or vault namespace patch commands.

For example:

$ tee payload.json <<EOF
{
   "custom_metadata": {
      "team": "Marketing (US)",
      "language": "English"
   }
}
EOF
  1. Open a web browser to launch the Vault UI (e.g. http://127.0.0.1:8200/ui) and then login.

  2. Select Access.

  3. Select Namespaces and then select Create Namespace.

  4. Enter education in the Path field.

  5. Click Save.

  6. To create child namespaces, select the down-arrow on the upper left corner of the UI, and select education. NS Selection

  7. Under the Access tab, select Namespaces and then click Create a namespace.

  8. Enter training in the Path field, and click Save.

  9. Select Create a namespace again, and then enter certification in the Path field, and click Save. Child Namespaces

Write policies

(Persona: operations)

In this scenario, there is an organization-level administrator who is a superuser within the scope of the education namespace. Also, there is a team-level administrator for training and certification.

Policy for education admin

Requirements:

  • Create and manage namespaces
  • Create and manage policies
  • Enable and manage secrets engines
  • Create and manage entities and groups
  • Manage tokens
edu-admin.hcl
# Manage namespaces
path "sys/namespaces/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage policies
path "sys/policies/acl/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# List policies
path "sys/policies/acl" {
   capabilities = ["list"]
}

# Enable and manage secrets engines
path "sys/mounts/*" {
   capabilities = ["create", "read", "update", "delete", "list"]
}

# List available secrets engines
path "sys/mounts" {
  capabilities = [ "read" ]
}

# Create and manage entities and groups
path "identity/*" {
   capabilities = ["create", "read", "update", "delete", "list"]
}

# Manage tokens
path "auth/token/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage secrets at 'edu-secret'
path "edu-secret/*" {
   capabilities = ["create", "read", "update", "delete", "list"]
}

Policy for training admin

Requirements:

  • Create and manage child-namespaces
  • Create and manage policies
  • Enable and manage secrets engines
training-admin.hcl
# Manage namespaces
path "sys/namespaces/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage policies
path "sys/policies/acl/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# List policies
path "sys/policies/acl" {
  capabilities = ["list"]
}

# Enable and manage secrets engines
path "sys/mounts/*" {
   capabilities = ["create", "read", "update", "delete", "list"]
}

# List available secrets engines
path "sys/mounts" {
  capabilities = [ "read" ]
}

# Manage secrets at 'team-secret'
path "team-secret/*" {
   capabilities = ["create", "read", "update", "delete", "list"]
}

NOTE: Refer to the Additional Discussion section to learn more about policy authoring with namespaces.

To target a specific namespace, you can do one of the following:

Option 1: Set VAULT_NAMESPACE so that all subsequent CLI commands will be executed against that particular namespace.

$ export VAULT_NAMESPACE=<namespace_name>
$ vault policy write <policy_name> <policy_file>

Option 2: Specify the target namespace with -namespace flag

$ vault policy write -namespace=<namespace_name> <policy_name> <policy_file>

Since you have to deploy policies onto education and education/training namespaces, use -namespace flag instead of environment variable.

  1. Create edu-admin policy under education namespace.

    $ vault policy write -namespace=education edu-admin edu-admin.hcl
    
  2. Create training-admin policy under education/training namespace.

    $ vault policy write -namespace=education/training training-admin \
          training-admin.hcl
    

To target a specific namespace, you can do one of the following.

Option 1: Pass the target namespace in the X-Vault-Namespace header

Option 2: Prepend the API endpoint with namespace name (e.g. <namespace_name>/sys/policies/acl)

  1. First, create an API request payload containing the stringified edu-admin policy.

    $ tee edu-payload.json <<EOF
    {
      "policy": "# Manage namespaces\npath \"sys/namespaces/*\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\", \"sudo\"]\n}\n\n# Manage policies\npath \"sys/policies/acl/*\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\", \"sudo\"]\n}\n\n# List policies\npath \"sys/policies/acl\" {\n   capabilities = [\"list\"]\n}\n\n# Enable and manage secrets engines\npath \"sys/mounts/*\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]\n}\n\n# List available secrets engines\npath \"sys/mounts\" {\n  capabilities = [ \"read\" ]\n}\n\n# Create and manage entities and groups\npath \"identity/*\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]\n}\n\n# Manage tokens\npath \"auth/token/*\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\", \"sudo\"]\n}\n\n\n# Manage tokens\npath \"edu-secret/*\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]\n}\n"
    }
    EOF
    
  2. Create edu-admin policy under education namespace.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
          --header "X-Vault-Namespace: education" \
          --request PUT \
          --data @edu-payload.json \
          https://127.0.0.1:8200/v1/sys/policies/acl/edu-admin
    
  3. Create an API request payload containing the stringified /training-admin policy.

    $ tee training-payload.json <<EOF
    {
    "policy": "# Manage namespaces\npath \"sys/namespaces/*\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\", \"sudo\"]\n}\n\n# Manage policies\npath \"sys/policies/acl/*\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\", \"sudo\"]\n}\n\n# List policies\npath \"sys/policies/acl\" {\n  capabilities = [\"list\"]\n}\n\n# Enable and manage secrets engines\npath \"sys/mounts/*\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]\n}\n\n# List available secrets engines\npath \"sys/mounts\" {\n  capabilities = [ \"read\" ]\n}\n\n# Manage secrets at 'team-secret'\npath \"team-secret/*\" {\n   capabilities = [\"create\", \"read\", \"update\", \"delete\", \"list\"]\n}  \n"
    }
    EOF
    
  4. Create training-admin policy under education/training namespace.

    $ curl --header "X-Vault-Token: $VAULT_TOKEN" \
          --request PUT \
          --data @training-payload.json \
          https://127.0.0.1:8200/v1/education/training/sys/policies/acl/training-admin
    
  1. In the Web UI, make sure that the CURRENT NAMESPACE is set to education in the upper left menu.

  2. Click the Policies tab, and then select Create ACL policy.

  3. Toggle Upload file sliding switch, and click Choose a file to select your edu-admin.hcl file you authored. This loads the policy and sets the Name to be edu-admin.

  4. Click Create Policy to complete.

  5. Set the CURRENT NAMESPACE to be education/training in the upper left menu.

    Namespace

  6. In the Policies tab, select Create ACL policy.

  7. Toggle Upload file, and click Choose a file to select your training-admin.hcl file you authored.

  8. Click Create Policy.

To codify the creation of namespaces and policies under each namespaces, read the Codify Management of Vault Enterprise Using Terraform tutorial.

Set up entities and groups

(Persona: operations)

In the education namespace, create an entity, Bob Smith with edu-admin policy attached. Add a userpass user, bob as an entity alias. By default, Bob Smith has no visibility into the education/training namespace since the bob user was defined in the education namespace.

You are going to create an internal group named, Training Admin in the education/training namespace with training-admin policy attached. To grant the training-admin policy for bob, add the Bob Smith entity to the Training Admin group as a member entity.

Entities and Groups

This step only demonstrates CLI commands and Web UI to create entities and groups. Refer to the Identity - Entities and Groups tutorial if you need the full details. Also, read the Additional Discussion section for an example of setting up external groups.

  1. Enable the userpass auth method.

    $ vault auth enable -namespace=education userpass
    
  2. Create a user bob under the education namespace.

    $ vault write -namespace=education \
            auth/userpass/users/bob password="training"
    
  3. Create an entity for Bob Smith with edu-admin policy attached. Save the generated entity ID in a file named entity_id.txt.

    $ vault write -namespace=education -format=json identity/entity name="Bob Smith" \
            policies="edu-admin" | jq -r ".data.id" > entity_id.txt
    
  4. Get the mount accessor for userpass auth method and save it in accessor.txt.

    $ vault auth list -namespace=education -format=json \
            | jq -r '.["userpass/"].accessor' > accessor.txt
    
  5. Create an entity alias for Bob Smith to attach bob.

    $ vault write -namespace=education identity/entity-alias name="bob" \
            canonical_id=$(cat entity_id.txt) mount_accessor=$(cat accessor.txt)
    
  6. Create a group, "Training Admin" in education/training namespace with Bob Smith entity as its member.

    $ vault write -namespace=education/training identity/group \
            name="Training Admin" policies="training-admin" \
            member_entity_ids=$(cat entity_id.txt)
    
  1. In the Web UI, make sure that the CURRENT NAMESPACE is set to education in the upper left menu.

  2. Click the Access tab, and select Enable new method.

  3. Select the Username & Password radio button, and click Next.

  4. Click Enable Method and then select View Method.

  5. Select Create user. Userpass Config

  6. Enter bob in the Username field and training in the Password field. Userpass Config

  7. Click Save.

  8. From the Access tab, select Entities and then Create entity.

  9. Enter Bob Smith in the Name field, and edu-admin in the Policies field.

  10. Click Create.

  11. Select Add alias. Enter bob in the Name field and select userpass/ (userpass) from the Auth Backend drop-down list.

  12. Click Create.

  13. Click the Access tab and select Entities.

  14. Select the Bob Smith entity and copy its ID displayed under the Details tab.

  15. Now, set the CURRENT NAMESPACE to education/training.

    Namespace

  16. In the Access tab, select Groups, and select Create group.

  17. Paste in the entity ID in the Member Entity IDs field you copied.

  18. Enter Training Admin in the Name field, training-admin in the Policies field, and click Create.

Test the Bob Smith entity

(Persona: org-admin)

  1. Log in as bob into the education namespace.

    $ vault login -namespace=education -method=userpass \
          username="bob" password="training"
    

    Example output:

    Key                    Value
    ---                    -----
    token                  hvs.CAESIExs8fiVHOAcwMZH6OPvtrL2a813UyknmPNXTtdVGXCWGiQKImh2cy5XTTlsemtQVm5IYUttcHJiZHRUR3VuRXguZTBiY3U
    token_accessor         M4MQ06Zr1pr2NA2g2vSwMNTn.e0bcu
    token_duration         768h
    token_renewable        true
    token_policies         ["default"]
    identity_policies      ["edu-admin"]
    policies               ["default" "edu-admin"]
    token_meta_username    bob
    

    User bob only has default policy attached to his token (token_policies); however, he inherited the edu-admin policy from the Bob Smith entity (identity_policies).

  2. Store the generated token value in BOB_TOKEN environment variable.

    Example:

    $ export BOB_TOKEN="hvs.CAESIDloyzuzE5SjJwWiZ0inIpKGKSuqusk4nagD4wPXBY0-GiQKImh2cy5xeGFuRzNZS1dYMWhvTW5OWW9UdnJNRnMuZTBiY3U"
    
  3. Test to make sure that bob can create a namespace, enable secrets engine, and whatever else that you want to verify against the education namespace.

    $ export VAULT_NAMESPACE="education"
    
  4. Verify that you can create a new namespace called web-app.

    $ VAULT_TOKEN=$BOB_TOKEN vault namespace create web-app
    
    Key                Value
    ---                -----
    custom_metadata    map[]
    id                 wWIAh
    path               education/web-app/
    
  5. Verify that you can enable key/value v2 secrets engine at edu-secret.

    $ VAULT_TOKEN=$BOB_TOKEN vault secrets enable -path=edu-secret kv-v2
    Success! Enabled the kv-v2 secrets engine at: edu-secret/
    

    Optionally, you can create new policies to test that bob can perform the operations as expected.

  6. Unset the VAULT_NAMESPACE environment variable.

    $ unset VAULT_NAMESPACE
    
  1. Log in as bob into the education namespace.

    $ curl --header "X-Vault-Namespace: education" \
          --request POST \
          --data '{"password": "training"}' \
          $VAULT_ADDR/v1/auth/userpass/login/bob | jq -r ".auth"
    

    Example output:

    {
    "client_token": "hvs.CAESIDloyzuzE5SjJwWiZ0inIpKGKSuqusk4nagD4wPXBY0-GiQKImh2cy5xeGFuRzNZS1dYMWhvTW5OWW9UdnJNRnMuZTBiY3U",
    "accessor": "4DHq7lIlqgElZkqifCyr4kFv.e0bcu",
    "policies": [
       "default",
       "edu-admin"
    ],
    "token_policies": [
       "default"
    ],
    "identity_policies": [
       "edu-admin"
    ],
    "metadata": {
       "username": "bob"
    },
    "lease_duration": 2764800,
    "renewable": true,
    "entity_id": "d646746e-a884-8c0a-ae72-1026f0478b32",
    "token_type": "service",
    "orphan": true,
    "mfa_requirement": null,
    "num_uses": 0
    }
    

    Use bob only has default policy attached to his token (token_policies); however, he inherited the edu-admin policy from the Bob Smith entity (identity_policies). Also, training-admin policy is listed under external_namespace_policies due to its membership to the Training Admin group in education/training namespace.

  2. Store the generated token value in BOB_TOKEN environment variable.

    Example:

    $ export BOB_TOKEN="hvs.CAESIDloyzuzE5SjJwWiZ0inIpKGKSuqusk4nagD4wPXBY0-GiQKImh2cy5xeGFuRzNZS1dYMWhvTW5OWW9UdnJNRnMuZTBiY3U"
    
  3. Verify that you can create a new namespace called web-app. Be sure to use generated bob's client token.

    $ curl --header "X-Vault-Token: $BOB_TOKEN" \
          --request POST \
          $VAULT_ADDR/v1/education/sys/namespaces/web-app
    
  4. Verify that you can enable key/value v2 secrets engine at edu-secret.

    $ curl --header "X-Vault-Token: $BOB_TOKEN" \
          --request POST \
          --data '{"type": "kv-v2"}' \
          $VAULT_ADDR/v1/education/sys/mounts/edu-secret
    
  1. Open a web browser and launch the Vault UI (e.g. http://127.0.0.1:8200/ui). If you are already logged in, sign out.

  2. At the Sign in to Vault, set the Namespace to education.

  3. Select Username from the Method drop-down, enter bob in the Username field, and password in the Password field.

    Login

  4. Click Sign in. Notice that the CURRENT NAMESPACE is set to education in the upper left corner of the UI.

  5. To add a new namespace, select Access.

  6. Select Namespaces and then click Create a namespace.

  7. Enter web-app in the Path field, and then click Save.

  8. Select Secrets, and then Enable new engine.

  9. Select the KV radio button and click Next.

  10. Enter edu-secret in the Path field.

  11. Click Enable Engine to finish.

Test the training admin group

(Persona: team-admin)

  1. Look up the token details.

    $ VAULT_TOKEN=$BOB_TOKEN vault token lookup
    

    Example output:

    Key                            Value
    ---                            -----
    accessor                       4DHq7lIlqgElZkqifCyr4kFv.e0bcu
    creation_time                  1664329195
    creation_ttl                   768h
    display_name                   education-auth-userpass-bob
    entity_id                      d646746e-a884-8c0a-ae72-1026f0478b32
    expire_time                    2022-10-29T18:39:55.055562-07:00
    explicit_max_ttl               0s
    external_namespace_policies    map[SYIaz:[training-admin]]
    id                             hvs.CAESIDloyzuzE5SjJwWiZ0inIpKGKSuqusk4nagD4wPXBY0-GiQKImh2cy5xeGFuRzNZS1dYMWhvTW5OWW9UdnJNRnMuZTBiY3U
    identity_policies              [edu-admin]
    issue_time                     2022-09-27T18:39:55.055566-07:00
    meta                           map[username:bob]
    namespace_path                 education/
    num_uses                       0
    orphan                         true
    path                           auth/userpass/login/bob
    policies                       [default]
    renewable                      true
    ttl                            767h45m2s
    type                           service
    

    Notice that the external_namespace_policies parameter lists training-admin policy. The user bob inherited this policy from the Training Admin group defined in the education/training namespace although bob user was created in the education namespace.

    Verify that bob can perform the operations permitted by the training-admin policy.

  2. Set the target namespace as an environment variable.

    $ export VAULT_NAMESPACE="education/training"
    
  3. Create a new namespace called vault-training.

    $ VAULT_TOKEN=$BOB_TOKEN vault namespace create vault-training
    
    Key                Value
    ---                -----
    custom_metadata    map[]
    id                 MCDB7
    path               education/training/vault-training/
    
  4. Enable key/value v1 secrets engine at team-secret.

    $ VAULT_TOKEN=$BOB_TOKEN vault secrets enable -path=team-secret -version=1 kv
    Success! Enabled the kv secrets engine at: team-secret/
    
  5. Unset the VAULT_NAMESPACE environment variable.

    $ unset VAULT_NAMESPACE
    
  1. Look up the token details.

    $ curl --header "X-Vault-Token: $BOB_TOKEN" \
          --header "X-Vault-Namespace: education/training" \
          $VAULT_ADDR/v1/auth/token/lookup-self | jq -r ".data"
    

    Example output:

    {
       "accessor": "4DHq7lIlqgElZkqifCyr4kFv.e0bcu",
       "creation_time": 1664329195,
       "creation_ttl": 2764800,
       "display_name": "education-auth-userpass-bob",
       "entity_id": "d646746e-a884-8c0a-ae72-1026f0478b32",
       "expire_time": "2022-10-29T18:39:55.055562-07:00",
       "explicit_max_ttl": 0,
       "external_namespace_policies": {
          "SYIaz": [
             "training-admin"
          ]
       },
       "id": "hvs.CAESIDloyzuzE5SjJwWiZ0inIpKGKSuqusk4nagD4wPXBY0-GiQKImh2cy5xeGFuRzNZS1dYMWhvTW5OWW9UdnJNRnMuZTBiY3U",
       "identity_policies": [
          "edu-admin"
       ],
       "issue_time": "2022-09-27T18:39:55.055566-07:00",
       "meta": {
          "username": "bob"
       },
       "namespace_path": "education/",
       "num_uses": 0,
       "orphan": true,
       "path": "auth/userpass/login/bob",
       "policies": [
          "default"
       ],
       "renewable": true,
       "ttl": 2763207,
       "type": "service"
    }
    

    Notice that the external_namespace_policies parameter lists training-admin policy. The user bob inherited this policy from the Training Admin group defined in the education/training namespace although bob user was created in the education namespace.

  2. Verify that bob can perform the operations permitted by the training-admin policy. Create a new namespace called vault-training.

    $ curl --header "X-Vault-Token: $BOB_TOKEN" \
          --request POST \
          $VAULT_ADDR/v1/education/training/sys/namespaces/vault-training | jq -r ".data"
    

    Example output:

    {
       "custom_metadata": {},
       "id": "iTCLI",
       "path": "education/training/vault-training/"
    }
    
  3. Enable key/value v1 secrets engine at team-secret.

    $ curl --header "X-Vault-Token: $BOB_TOKEN" \
          --request POST \
          --data '{"type": "kv"}' \
          $VAULT_ADDR/v1/education/training/sys/mounts/team-secret
    
  1. Stay signed in as bob.

  2. Set the CURRENT NAMESPACE to be education/training in the upper left menu.

  3. To add a new namespace, select Access.

  4. Select Namespaces and then click Create a namespace.

  5. Enter vault-training in the Path field, and then click Save.

  6. Select Secrets, and then Enable new engine.

  7. Select the KV radio button and click Next.

  8. Enter team-secret in the Path field.

  9. Click Enable Engine to finish.

Summary

As this tutorial demonstrated, each namespace you created behaves as an isolated Vault environment. By default, there is no visibility into other namespaces regardless of its hierarchical relationship. In order for Bob to operate in education/training namespace, you can enable an auth method in the education/training namespace so that he can log in. Or, as demonstrated in this tutorial, you can use Vault identity to associate entities and groups defined in different namespaces.

NOTE: Bob still needs to log into the education namespace since his token is tied to the education namespace and it is invalid to log into the education/training namespace.

Audit ambient credentials

(Persona: operator)

Many auth and secrets providers, such as AWS, Azure, GCP, and AliCloud, use ambient credentials to authenticate API calls. For example, AWS may:

  1. Use an access key and secret key configured in Vault.
  2. If not present, check for environment variables such as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
  3. If not present, load credentials configured in ~/.aws/credentials.
  4. If not present, use instance metadata.

This becomes a problem if these ambient credentials are not intended to be used within a particular namespace.

For example, suppose that your Vault server is running on an AWS EC2 instance. You give the owner of a namespace a particular set of permissions to use for AWS. However, that owner does not configure them. So, Vault falls back to using the credentials available in instance metadata, leading to a privilege escalation.

To handle this:

  • Ensure no environment variables are available that could grant a privilege escalation.
  • Ensure that any privileges granted through instance metadata (in this example) or other ambient identity info represent a loss of privilege.
  • Directly configure the correct credentials in namespaces, and restrict access to that endpoint so credentials can't later be edited to use ambient credentials.

Summary: As this tutorial demonstrated, each namespace you created behaves as an isolated Vault environment. Once you sign into a namespace, there is no visibility into other namespaces regardless of its hierarchical relationship. Tokens, policies, and secrets engines are tied to its namespace; therefore, each client must acquire a valid token for each namespace to access their secrets.

Additional discussion

Leveraging identity for auth methods with external groups

For simplicity, this tutorial used the username and password (userpass) auth method which was enabled in the education namespace as well as the education/training namespace.

However, most likely, your organization uses an auth method such as ldap or okta and map appropriate policies to those externally defined groups. For these types of auth methods, you have two options:

  1. Enable the auth method in each namespace the same way you performed in the set up entities and groups step with userpass.
  2. Enable the auth method in the root namespace and use Identity Groups to pull in external groups and map policies in each namespace.

NOTE: As mentioned in the Solution section, identity groups can pull in entities and groups from other namespaces.

The second option enables an auth method in the root namespace rather than enable in multiple namespaces. The following steps demonstrate the second option to create the "Training Admin" group as described in this tutorial.

First, enable and configure the desired auth method (e.g. LDAP) in the root namespace.

$ vault auth enable ldap

Configure the LDAP auth method.

$ vault write auth/ldap/config \
        url="ldap://ldap.example.com" \
        userdn="ou=Users,dc=example,dc=com" \
        groupdn="ou=Groups,dc=example,dc=com" \
        groupfilter="(&(objectClass=group)(member:1.2.840.113556.1.4.1941:={{.UserDN}}))" \
        groupattr="cn" \
        upndomain="example.com" \
        certificate=@ldap_ca_cert.pem \
        insecure_tls=false \
        starttls=true

For each LDAP group to which you want to map policies, create an external group in the root namespace with an alias whose name exactly matches the LDAP group name.

Get the mount accessor for ldap auth method and save it in accessor.txt.

$ vault auth list -format=json \
        | jq -r '.["ldap/"].accessor' > accessor.txt

Create an external group and save the generated group ID in group_id.txt.

$ vault write -format=json identity/group name="training_admin_root" \
        type="external" \
        | jq -r ".data.id" > group_id.txt

Create a group alias whose name exactly matches the LDAP group name. The following example uses a group name in LDAP is ops_training.

$ vault write -format=json identity/group-alias name="ops_training" \
        mount_accessor=$(cat accessor.txt) \
        canonical_id=$(cat group_id.txt)

In the education/training namespace, create an internal group which has the external group (training_admin_root) as its member. Attach the training-admin policy to this internal group. Remember that you created this policy inside of the education/training namespace in the write policies step.

$ vault write -namespace=education/training identity/group \
        name="Training Admin" \
        policies="training-admin" \
        member_group_ids=$(cat group_id.txt)

NOTE: Remember that the auth methods are enabled in an explicit namespace. Therefore, the users must authenticate into the namespace where the auth method is configured (in this case, the ldap auth method is configured in the root namespace).

Let's assume the user, "bob_smith" belongs to the LDAP ops_training group.

$ vault login -method=ldap username=bob_smith
Password (will be hidden):

Success! You are now authenticated.
# ...snip...

When you check the returned token's properties, the output should display training-admin as its external_namespace_policies.

$ vault token Lookup

Key                            Value
---                            -----
...snip...
external_namespace_policies    map[9dKXw:[training-admin]]
id                             s.10LCp6O5xnweGVm1eyJbp127
identity_policies              <nil>
...snip...
policies                       [default]
renewable                      true
ttl                            767h59m11s
type                           service

Since you did not assign any policy when you created the training_admin_root external group, the identity_policies shows nil. Therefore, bob_smith's token only has the default policy in the root namespace. However, it has training-admin policy in the external namespace of ID, 9dKXw (in this example, it is the ID of education/training namespace).

Set the target namespace as an environment variable for convenience, and bob_smith is ready to operate against the education/training namespace.

$ export VAULT_NAMESPACE="education/training"

The namespaces feature is designed to create an isolation around each namespace so that Vault tenants can operate independently by having a mini-Vault environment of their own. However, the second option leaves some dependency on the root namespace. The root-level admin is responsible for creating the mapping between the LDAP groups and the Vault's identity groups. The users log into the root namespace instead of logging in the target namespace.

Policy with namespaces

In this tutorial, you created policies in each namespace (education and education/training). Therefore, you did not have to specify the target namespace in the policy paths.

If you want to create policies in the root namespace to control education and education/training namespaces, prepend the namespace in the paths.

For example:

# Manage policies in the 'education' namespace
path "education/sys/policies/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage tokens in the 'education' namespace
path "education/auth/token/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage policies under 'education/training' namespace
path "education/training/sys/policies/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage tokens in the 'education/training' namespace
path "education/training/auth/token/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
...

In the write policies step, you deployed the training-admin policy in the education/training namespace. The path is relative to the working namespace. So, if you want to create the training-admin policy in the education namespace instead, the paths starts with training/ rather than education/training/.

# Manage namespaces
path "training/sys/namespaces/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}

# Manage policies
path "training/sys/policies/*" {
   capabilities = ["create", "read", "update", "delete", "list", "sudo"]
}
...

NOTE: Policies are tied to its namespace. Therefore, if you created the training_admin policy in the education namespace, it is available only to those clients authenticated in the education namespace.

Clean up

  1. Unset the VAULT_TOKEN environment variable.

    $ unset VAULT_TOKEN
    
  2. Unset the VAULT_ADDR environment variable.

    $ unset VAULT_ADDR
    
  3. You can stop the Vault dev server by pressing Ctrl+C where the server is running. Or, execute the following command.

    $ pgrep -f vault | xargs kill
    

Help and reference

  • Namespaces
  • Vault Namespace and Mount Structuring Guide
  • Streamline Secrets Management with Vault Agent and Vault 0.11
 Previous
 Next

This tutorial also appears in:

  •  
    7 tutorials
    Vault 1.12 Release Highlights
    The listed tutorials were updated to showcase the new enhancements introduced in Vault 1.12.
    • Vault

On this page

  1. Secure Multi-Tenancy with Namespaces
  2. Personas
  3. Challenge
  4. Solution
  5. Prerequisites
  6. Lab setup
  7. Scenario introduction
  8. Create namespaces
  9. Write policies
  10. Set up entities and groups
  11. Test the Bob Smith entity
  12. Test the training admin group
  13. Audit ambient credentials
  14. Additional discussion
  15. Clean up
  16. Help and reference
Give Feedback(opens in new tab)
  • Certifications
  • System Status
  • Terms of Use
  • Security
  • Privacy
  • Trademark Policy
  • Trade Controls
  • Give Feedback(opens in new tab)