Kubernetes with HCP Vault

HCP Vault with AWS EKS and JWT Auth Method

  • 12min

  • HCP
  • Vault

HashiCorp Cloud Platform (HCP) is a fully managed platform offering HashiCorp Products as a Service (HPaaS) to automate infrastructure on any cloud.

In this tutorial, you will learn the process required to authenticate an AWS EKS cluster to HCP Vault with the JWT auth method.

Prerequisites

The following prerequisites are required:

TIP: Ensure that you have authenticated with the AWS CLI, and that the CLI is targeting the region where you created your EKS cluster. Review the AWS documentation for instructions on how to configure the AWS CLI.

Complete the steps detailed in the manual deployment tutorial to enable communication between your HCP Vault cluster servers and the Vault Agent running in your EKS cluster.

Your EKS cluster security group must also allow traffic from the HCP Vault CIDR range. If your EKS cluster endpoint uses port 443, create a security group rule to allow ingress traffic from HCP Vault to the primary EKS cluster security group.

The HashiCorp Virtual Network (HVN) section of the HCP UI can assist you in composing the correct eksctl commands to do this when you establish the peering connection between your HVN and VPC.

You will configure Vault from your development host. As a result, the HCP Vault cluster needs to be publicly available. You should instead configure HCP Vault over a bastion host using the Vault private interface instead when working in production.

Scenario environments

The scenario consists of 3 distinct environments:

  • Your local host: This is where you will use the terminal along with eksctl, vault, and kubectl CLI commands to perform the tasks which make up the scenario.

  • Your HCP account: This is where you have deployed your HCP Vault cluster. You can use Terraform to deploy the HCP Vault cluster, or do so manually with the HCP web UI.

  • Your AWS account: This is where you have deployed your EKS cluster and its VPC that is peered to your HVN with appropriate subnet, route table, and security group configuration.

Prepare local scenario environment

On your local host, prepare the scenario environment.

For ease of clean up and simplicity, create a temporary directory that contains all required configurations for the scenario, and assign that directory name value to the environment variable HC_LEARN_LAB.

$ export HC_LEARN_LAB="$(mktemp -d -t hc-learn-XXXX)"

$ export HC_LEARN_LAB="$(mktemp -d -t hc-learn-XXXX)"

Now change into this directory; you will execute all scenario commands on the local host from this directory or one of its descendants.

$ cd ${HC_LEARN_LAB?}

$ cd ${HC_LEARN_LAB?}

Kubernetes stores cluster connection information in a special file called kubeconfig. You can retrieve the Kubernetes configuration settings for your EKS cluster, and merge them into your local kubeconfig file.

  1. Use the AWS CLI to retrieve the kubeconfig.

    $ aws eks --region <your-region> update-kubeconfig --name <your-cluster-name>

    $ aws eks --region <your-region> update-kubeconfig --name <your-cluster-name>

  2. You can use the HCP Portal to retrieve the client configuration information you need to connect the Vault Agents in your EKS cluster to HCP Vault. Navigate to the Vault resource page in the HCP portal, and then select the Vault cluster.

  3. Click "Generate Token". Copy the administrator token and set it in your terminal to an environment variable named VAULT_TOKEN.

    $ export VAULT_TOKEN=hvs.********************************************************************************************************

    $ export VAULT_TOKEN=hvs.********************************************************************************************************

  4. Click the clipboard next to "Public". This will copy the public Vault address to your clipboard. Set it as the VAULT_ADDR environment variable.

    $ export VAULT_ADDR=https://learn.vault.**************.aws.hashicorp.cloud:8200

    $ export VAULT_ADDR=https://learn.vault.**************.aws.hashicorp.cloud:8200

  5. Click the clipboard next to "Private". This will copy the private Vault address to your clipboard. Set the Vault private address as the VAULT_PRIVATE_ADDR environment variable. You will use this later in the tutorial to enable the EKS cluster to access HCP Vault over the HVN peering connection.

    $ export VAULT_PRIVATE_ADDR=https://learn.private.vault.**************.aws.hashicorp.cloud:8200

    $ export VAULT_PRIVATE_ADDR=https://learn.private.vault.**************.aws.hashicorp.cloud:8200

  6. Since HCP Vault uses the Enterprise Namespace feature with a default namespace named admin, you need to set the VAULT_NAMESPACE environment variable value to admin.

    $ export VAULT_NAMESPACE=admin

    $ export VAULT_NAMESPACE=admin

Install Vault Agents on EKS

Use the official vault-helm chart to install Vault Agents in your EKS cluster.

  1. Retrieve the Helm chart from the HashiCorp Helm repository.

    $ helm repo add hashicorp https://helm.releases.hashicorp.com && \
    helm repo update

    $ helm repo add hashicorp https://helm.releases.hashicorp.com && \
    helm repo update

    Example output:

    "hashicorp" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈ Happy Helming!⎈

    "hashicorp" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "hashicorp" chart repository
Update Complete. ⎈ Happy Helming!⎈

  2. Create a values.yaml file that sets the external servers to HCP Vault over the private address. This will deploy a Vault Agent injector into the EKS cluster.

    $ cat > values.yaml << EOF
injector:
   enabled: true
   externalVaultAddr: "${VAULT_PRIVATE_ADDR?}"
EOF

    $ cat > values.yaml << EOF
injector:
   enabled: true
   externalVaultAddr: "${VAULT_PRIVATE_ADDR?}"
EOF

  3. Validate that the values file is correctly populated with your HCP Vault private address value (which will differ from the example shown).

    $ cat values.yaml
injector:
enabled: true
externalVaultAddr: "https://vault-cluster-brian.private.vault.11eb3a47-8920-4714-ba99-0242ac11000e.aws.hashicorp.cloud:8200"

    $ cat values.yaml
injector:
enabled: true
externalVaultAddr: "https://vault-cluster-brian.private.vault.11eb3a47-8920-4714-ba99-0242ac11000e.aws.hashicorp.cloud:8200"

  4. Install the HashiCorp Vault Helm chart.

    $ helm install vault -f values.yaml hashicorp/vault

NAME: vault
LAST DEPLOYED: Thu Jun  9 08:54:48 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/

Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault

    $ helm install vault -f values.yaml hashicorp/vault

NAME: vault
LAST DEPLOYED: Thu Jun  9 08:54:48 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://www.vaultproject.io/docs/

Your release is named vault. To learn more about the release, try:

  $ helm status vault
  $ helm get manifest vault

  5. Once the helm install command completes, verify the Vault Agent injector pod deploys by issuing kubectl get pods.

    $ kubectl get pods

    $ kubectl get pods

    Example output:

    NAME                                   READY   STATUS    RESTARTS   AGE
vault-agent-injector-c8fd9fc5f-jhhw9   1/1     Running   0          2m11s

    NAME                                   READY   STATUS    RESTARTS   AGE
vault-agent-injector-c8fd9fc5f-jhhw9   1/1     Running   0          2m11s

Kubernetes configuration

Create a namespace, learn-vault.

$ kubectl create namespace learn-vault

$ kubectl create namespace learn-vault

Create a cluster role binding.

$ kubectl create clusterrolebinding oidc-reviewer \
--clusterrole=system:service-account-issuer-discovery \
--group=system:unauthenticated

$ kubectl create clusterrolebinding oidc-reviewer \
--clusterrole=system:service-account-issuer-discovery \
--group=system:unauthenticated

Create a service account named product in the learn-vault namespace.

$ kubectl create sa product -n learn-vault

$ kubectl create sa product -n learn-vault

Configure JWT auth method on HCP Vault

Your services need a service account token to authenticate to Vault, which they can obtain from the Vault JWT/OIDC Auth Method.

  1. Enable the JWT/OIDC auth method in HCP Vault using your terminal.

    $ vault auth enable jwt

    $ vault auth enable jwt

    Output:

    Success! Enabled jwt auth method at: jwt/

    Success! Enabled jwt auth method at: jwt/

  2. Fetch the service account signing public key from your cluster's JWKS URI.

    Start a proxy.

    $ kubectl proxy &

    $ kubectl proxy &

    Assign its value to the environment variable ISSUER.

    $ ISSUER="$(curl --fail --silent --show-error 127.0.0.1:8001/.well-known/openid-configuration | jq -r '.issuer')"

    $ ISSUER="$(curl --fail --silent --show-error 127.0.0.1:8001/.well-known/openid-configuration | jq -r '.issuer')"

    Stop the proxy by killing its process.

    $ kill %%

    $ kill %%

  3. Create the JWT auth method configuration, specifying the ISSUER environment variable as value of the OIDC Discovery URL parameter.

    $ vault write auth/jwt/config oidc_discovery_url="${ISSUER?}"

    $ vault write auth/jwt/config oidc_discovery_url="${ISSUER?}"

    Successful output example:

    Success! Data written to: auth/jwt/config

    Success! Data written to: auth/jwt/config

Deploy an example workload

Now that the clients have been deployed, it is time to deploy an application workload. This tutorial will use the HashiCups demo application.

Issue the following command to clone the repository to the development host.

$ git clone https://github.com/hashicorp/hcp-vault-eks-jwt-auth

$ git clone https://github.com/hashicorp/hcp-vault-eks-jwt-auth

Change into the project directory.

$ cd hcp-vault-eks-jwt-auth

$ cd hcp-vault-eks-jwt-auth

Deploy a database to Kubernetes

NOTE: This tutorial deploys a database into Kubernetes and exposes it to HCP Vault using a public LoadBalancer service type. In a production configuration, you should deploy this as a private load balancer restricting access to only HCP Vault.

  1. Deploy a PostgreSQL database. This contains data for various coffees related to a demo application, all hosted in the products database.

    $ kubectl apply -f postgres.yaml

    $ kubectl apply -f postgres.yaml

    Output:

    service/postgres created
deployment.apps/postgres created

    service/postgres created
deployment.apps/postgres created

  2. Verify you've deployed the PostgreSQL database in your Kubernetes cluster.

    $ kubectl get pods

    $ kubectl get pods

    Example output:

    NAME                                   READY   STATUS    RESTARTS   AGE
postgres-5bbfc8bb5c-9px77              1/1     Running   0          31s
vault-agent-injector-c8fd9fc5f-jhhw9   1/1     Running   0          2m34s

    NAME                                   READY   STATUS    RESTARTS   AGE
postgres-5bbfc8bb5c-9px77              1/1     Running   0          31s
vault-agent-injector-c8fd9fc5f-jhhw9   1/1     Running   0          2m34s

Add the database role to Vault

The product API needs to read the database username and password from Vault. Create the role for the product service account to generate database credentials.

  1. Enable the database secrets engine.

    $ vault secrets enable database

    $ vault secrets enable database

    Output:

    Success! Enabled the database secrets engine at: database/

    Success! Enabled the database secrets engine at: database/

  2. Set the POSTGRES_IP environment variable to the load balancer DNS hostname.

    $ export POSTGRES_IP=$(kubectl get service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' \
   postgres)

    $ export POSTGRES_IP=$(kubectl get service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' \
   postgres)

  3. Create the database configuration that allows HCP Vault to configure Postgres.

    $ vault write database/config/products \
    plugin_name=postgresql-database-plugin \
    allowed_roles="*" \
    connection_url="postgresql://{{username}}:{{password}}@${POSTGRES_IP}:5432/products?sslmode=disable" \
    username="postgres" \
    password="password"

    $ vault write database/config/products \
    plugin_name=postgresql-database-plugin \
    allowed_roles="*" \
    connection_url="postgresql://{{username}}:{{password}}@${POSTGRES_IP}:5432/products?sslmode=disable" \
    username="postgres" \
    password="password"

  4. Create a database role for product that allows Vault to issue database passwords.

    $ vault write database/roles/product \
    db_name=products \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    revocation_statements="ALTER ROLE \"{{name}}\" NOLOGIN;"\
    default_ttl="1h" \
    max_ttl="24h"

    $ vault write database/roles/product \
    db_name=products \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    revocation_statements="ALTER ROLE \"{{name}}\" NOLOGIN;"\
    default_ttl="1h" \
    max_ttl="24h"

    Output:

    Success! Data written to: database/roles/product

    Success! Data written to: database/roles/product

  5. Request a new set of PostgreSQL database credentials for the product role.

    $ vault read database/creds/product

    $ vault read database/creds/product

    Example output:

    Key                Value
---                -----
lease_id           database/creds/product/zeXVyjlT550wHYEDOhF5ckDC.8fmet
lease_duration     1h
lease_renewable    true
password           gT8FvhyMtLuF-SCbw4mF
username           v-token-hc-product-Qcg6fDERVvya93uwvbCy-1652888425

    Key                Value
---                -----
lease_id           database/creds/product/zeXVyjlT550wHYEDOhF5ckDC.8fmet
lease_duration     1h
lease_renewable    true
password           gT8FvhyMtLuF-SCbw4mF
username           v-token-hc-product-Qcg6fDERVvya93uwvbCy-1652888425

Configure Vault policy for database credentials

Create a new product ACL policy that allows the product service to read the database credentials specific to product.

$ vault policy write product - << EOF
path "database/creds/product" {
  capabilities = ["read"]
}
EOF

$ vault policy write product - << EOF
path "database/creds/product" {
  capabilities = ["read"]
}
EOF

Output:

Success! Uploaded policy: product

Success! Uploaded policy: product

Create the vault-jwt-product JWT auth method role to associate the product. This allows the product service account in Kubernetes to get a Vault token with the JWT auth method.

$ vault write auth/jwt/role/vault-jwt-product \
    role_type="jwt" \
    bound_audiences="https://kubernetes.default.svc" \
    user_claim="sub" \
    bound_subject="system:serviceaccount:learn-vault:product" \
    policies="product" \
    ttl="1h"

$ vault write auth/jwt/role/vault-jwt-product \
    role_type="jwt" \
    bound_audiences="https://kubernetes.default.svc" \
    user_claim="sub" \
    bound_subject="system:serviceaccount:learn-vault:product" \
    policies="product" \
    ttl="1h"

NOTE: The bound_audiences value in this example is for EKS, but could differ for other platforms. If you are using a platform other than EKS, be sure to consult the documentation for that platform for the correct value.

Deploy the product API

  1. Make sure that you include your HCP Vault namespace in the deployment that will access database credentials.

    $ grep ${VAULT_NAMESPACE} product.yaml

    $ grep ${VAULT_NAMESPACE} product.yaml

    Output:

    vault.hashicorp.com/namespace: "admin"

    vault.hashicorp.com/namespace: "admin"

  2. Deploy the product service.

    $ kubectl apply -f product.yaml

    $ kubectl apply -f product.yaml

    Output:

    service/product created
serviceaccount/product created
deployment.apps/product created

    service/product created
serviceaccount/product created
deployment.apps/product created

  3. The product deployment should initialize.

    $ kubectl get pods

    $ kubectl get pods

    Example output:

    NAME                                    READY   STATUS    RESTARTS   AGE
postgres-5bd8c648fd-q6jhr               1/1     Running   0          25m
product-6f7ff64c75-zrcwd                2/2     Running   0          27s
vault-agent-injector-7d4ccf785f-dkt7c   1/1     Running   0          29m

    NAME                                    READY   STATUS    RESTARTS   AGE
postgres-5bd8c648fd-q6jhr               1/1     Running   0          25m
product-6f7ff64c75-zrcwd                2/2     Running   0          27s
vault-agent-injector-7d4ccf785f-dkt7c   1/1     Running   0          29m

  4. Port forward the web service locally to port 9090.

    $ kubectl port-forward service/product 9090

    $ kubectl port-forward service/product 9090

    Output:

    Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

    Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

  5. Open another terminal and make a request to localhost:9090/coffees to check if the web service can pull coffee information from the database.

    $ curl -s localhost:9090/coffees | jq .

    $ curl -s localhost:9090/coffees | jq .

    Output:

    [
  {
    "id": 1,
    "name": "Packer Spiced Latte",
    "teaser": "Packed with goodness to spice up your images",
    "description": "",
    "price": 350,
    "image": "/packer.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 2
      },
      {
        "ingredient_id": 4
      }
    ]
  },
  {
    "id": 2,
    "name": "Vaulatte",
    "teaser": "Nothing gives you a safe and secure feeling like a Vaulatte",
    "description": "",
    "price": 200,
    "image": "/vault.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 2
      }
    ]
  },
  {
    "id": 3,
    "name": "Nomadicano",
    "teaser": "Drink one today and you will want to schedule another",
    "description": "",
    "price": 150,
    "image": "/nomad.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 3
      }
    ]
  },
  {
    "id": 4,
    "name": "Terraspresso",
    "teaser": "Nothing kickstarts your day like a provision of Terraspresso",
    "description": "",
    "price": 150,
    "image": "/terraform.png",
    "ingredients": [
      {
        "ingredient_id": 1
      }
    ]
  },
  {
    "id": 5,
    "name": "Vagrante espresso",
    "teaser": "Stdin is not a tty",
    "description": "",
    "price": 200,
    "image": "/vagrant.png",
    "ingredients": [
      {
        "ingredient_id": 1
      }
    ]
  },
  {
    "id": 6,
    "name": "Connectaccino",
    "teaser": "Discover the wonders of our meshy service",
    "description": "",
    "price": 250,
    "image": "/consul.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 5
      }
    ]
  }
]

    [
  {
    "id": 1,
    "name": "Packer Spiced Latte",
    "teaser": "Packed with goodness to spice up your images",
    "description": "",
    "price": 350,
    "image": "/packer.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 2
      },
      {
        "ingredient_id": 4
      }
    ]
  },
  {
    "id": 2,
    "name": "Vaulatte",
    "teaser": "Nothing gives you a safe and secure feeling like a Vaulatte",
    "description": "",
    "price": 200,
    "image": "/vault.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 2
      }
    ]
  },
  {
    "id": 3,
    "name": "Nomadicano",
    "teaser": "Drink one today and you will want to schedule another",
    "description": "",
    "price": 150,
    "image": "/nomad.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 3
      }
    ]
  },
  {
    "id": 4,
    "name": "Terraspresso",
    "teaser": "Nothing kickstarts your day like a provision of Terraspresso",
    "description": "",
    "price": 150,
    "image": "/terraform.png",
    "ingredients": [
      {
        "ingredient_id": 1
      }
    ]
  },
  {
    "id": 5,
    "name": "Vagrante espresso",
    "teaser": "Stdin is not a tty",
    "description": "",
    "price": 200,
    "image": "/vagrant.png",
    "ingredients": [
      {
        "ingredient_id": 1
      }
    ]
  },
  {
    "id": 6,
    "name": "Connectaccino",
    "teaser": "Discover the wonders of our meshy service",
    "description": "",
    "price": 250,
    "image": "/consul.png",
    "ingredients": [
      {
        "ingredient_id": 1
      },
      {
        "ingredient_id": 5
      }
    ]
  }
]

When you are done, return to the terminal with the port-forward command and type Ctrl + C to exit.

Clean up

  1. Delete the product API.

    $  kubectl delete -f product.yaml

    $  kubectl delete -f product.yaml

  2. Delete the product role.

    $ vault delete auth/kubernetes/role/product

    $ vault delete auth/kubernetes/role/product

  3. Delete the product policy.

    $ vault policy delete product

    $ vault policy delete product

  4. Delete the product database role.

    $ vault delete database/roles/product

    $ vault delete database/roles/product

  5. Revoke all leases for database credentials.

    $ vault lease revoke -prefix database/creds/product

    $ vault lease revoke -prefix database/creds/product

  6. Delete the PostgreSQL database.

    $ kubectl delete -f postgres.yaml

    $ kubectl delete -f postgres.yaml

  7. Delete the database secrets engine configuration.

    $ vault delete database/config/product

    $ vault delete database/config/product

  8. Delete the Helm installation for HCP Vault.

    $ helm delete vault

    $ helm delete vault

  9. Disable the database secrets engine in HCP Vault.

    $ vault secrets disable database

    $ vault secrets disable database

  10. Disable the Kubernetes auth method in HCP Vault.

    $ vault auth disable kubernetes

    $ vault auth disable kubernetes

  11. Delete your HCP Vault cluster and the associated HVN.

  12. Remove the scenario directory and all of its contents.

    $ rm -rf ${HC_LEARN_LAB?}

    $ rm -rf ${HC_LEARN_LAB?}

Next steps

In this tutorial, you connected Vault clients on Amazon EKS to HCP Vault and retrieved PostgreSQL database credential dynamically. To learn more about the Vault features introduced in this tutorial, refer to the following tutorials.

Previous
Next

This tutorial also appears in:

  • 11 tutorials
    HCP Vault Operations
    Learn how to provision and connect to HCP Vault clusters.
    • Vault