HCP Vault with Amazon Elastic Kubernetes Service

HashiCorp Cloud Platform (HCP) is a fully managed platform offering HashiCorp Products as a Service (HPaaS) to automate infrastructure on any cloud.

This tutorial will cover the process required to connect an Elastic Kubernetes Service (EKS) Cluster to HCP Vault on AWS.

Prerequisites

The following prerequisites are required:

• An HCP HashiCorp Virtual Network (HVN)
• A public HCP Vault deployment
• AWS CLI
• kubectl
• helm
• jq
• git
• An EKS Cluster deployed in the VPC associated with your HVN

For this tutorial, you will need to ensure that you have authenticated with the AWS CLI, and that the CLI is targeting the region where you have created your EKS cluster. Review the AWS documentation for instructions on how to configure the AWS CLI.

To ensure that communication between your HCP Vault cluster servers and the agent running in your EKS cluster is possible, you must complete the steps detailed in the manual deployment tutorial.

Your EKS cluster security group must allow traffic from the HCP Vault CIDR range. If your EKS cluster endpoint uses port 443, create a security group rule to allow ingress traffic from HCP Vault to the primary EKS cluster security group.

For this tutorial, you will configure Vault from your development host. As a result, the HCP Vault cluster needs to be publicly available. In production, you should configure HCP Vault over a bastion host.

Configure development host

Kubernetes stores cluster connection information in a special file called kubeconfig. You can retrieve the Kubernetes configuration settings for your EKS cluster and merge them into your local kubeconfig file.

1. Use the AWS CLI to retrieve the kubeconfig.

   $ aws eks --region <your-region> update-kubeconfig --name <your-cluster-name>
   

   $ aws eks --region <your-region> update-kubeconfig --name <your-cluster-name>
   

   Copy

2. You can use the HCP Portal to retrieve the client configuration information you need to connect the Vault agents in your EKS cluster to HCP Vault. Navigate to the Vault resource page in the HCP portal, and then select the Vault cluster.

3. Click "Generate Token". Copy the administrator token and set it in your terminal to an environment variable named VAULT_TOKEN.

   $ export VAULT_TOKEN=s.*******************
   

   $ export VAULT_TOKEN=s.*******************
   

   Copy

4. Click the clipboard next to "Public". This will copy the public Vault address to your clipboard. Set it as the VAULT_ADDR environment variable.

   $ export VAULT_ADDR=https://learn.vault.**************.aws.hashicorp.cloud:8200
   

   $ export VAULT_ADDR=https://learn.vault.**************.aws.hashicorp.cloud:8200
   

   Copy

5. Click the clipboard next to "Private". This will copy the private Vault address to your clipboard. Set the Vault private address as the VAULT_PRIVATE_ADDR environment variable. You will use this later in the tutorial to enable the EKS cluster to access HCP Vault over the HVN peering connection.

   $ export VAULT_PRIVATE_ADDR=https://learn.private.vault.**************.aws.hashicorp.cloud:8200
   

   $ export VAULT_PRIVATE_ADDR=https://learn.private.vault.**************.aws.hashicorp.cloud:8200
   

   Copy

6. Since HCP Vault uses namespaces, set the VAULT_NAMESPACE environment variable to admin.

   $ export VAULT_NAMESPACE=admin
   

   $ export VAULT_NAMESPACE=admin
   

   Copy

Install Vault agents on EKS

This uses the official vault-helm chart to install the Vault agents to your EKS cluster.

1. Retrieve the Helm chart from the HashiCorp Helm repository.

   $ helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update
   

   $ helm repo add hashicorp https://helm.releases.hashicorp.com && helm repo update
   

   Copy

   Example output:

   "hashicorp" has been added to your repositories
   Hang tight while we grab the latest from your chart repositories...
   ...Successfully got an update from the "hashicorp" chart repository
   Update Complete. ⎈ Happy Helming!⎈
   

   "hashicorp" has been added to your repositories
   Hang tight while we grab the latest from your chart repositories...
   ...Successfully got an update from the "hashicorp" chart repository
   Update Complete. ⎈ Happy Helming!⎈
   

2. Create a values.yaml file that sets the external servers to HCP Vault. This will deploy a Vault agent injector into the EKS cluster.

   $ cat > values.yaml << EOF
   injector:
      enabled: true
      externalVaultAddr: "${VAULT_PRIVATE_ADDR}"
   EOF
   

   $ cat > values.yaml << EOF
   injector:
      enabled: true
      externalVaultAddr: "${VAULT_PRIVATE_ADDR}"
   EOF
   

   Copy

3. Validate that the values file is populated correctly.

   $ more values.yaml
   

   $ more values.yaml
   

   Copy

4. Install the HashiCorp Vault Helm chart.

   $ helm install vault -f values.yaml hashicorp/vault
   

   $ helm install vault -f values.yaml hashicorp/vault
   

   Copy

5. Once the helm install command completes, verify the Vault agent injector pod deploys by issuing kubectl get pods.

   $ kubectl get pods
   

   $ kubectl get pods
   

   Copy

   Example output:

   NAME                                   READY   STATUS    RESTARTS   AGE
   vault-agent-injector-c8fd9fc5f-jhhw9   1/1     Running   0          2m11s
   

   NAME                                   READY   STATUS    RESTARTS   AGE
   vault-agent-injector-c8fd9fc5f-jhhw9   1/1     Running   0          2m11s
   

Configure Kubernetes auth method on HCP Vault

Your services need a Kubernetes service account token to authenticate to Vault.

1. Enable the Kubernetes auth method in HCP Vault using your terminal.

   $ vault auth enable kubernetes
   

   $ vault auth enable kubernetes
   

   Copy

   Output:

   Success! Enabled kubernetes auth method at: kubernetes/
   

   Success! Enabled kubernetes auth method at: kubernetes/
   

   Copy

2. The chart configures a Kubernetes service account named vault that you can use to enable Vault communication with Kubernetes. Get the JSON Web Token (JWT) for the vault service account and set it to the TOKEN_REVIEW_JWT environment variable.

   $ export TOKEN_REVIEW_JWT=$(kubectl get secret \
      $(kubectl get serviceaccount vault -o jsonpath='{.secrets[0].name}') \
      -o jsonpath='{ .data.token }' | base64 --decode)
   

   $ export TOKEN_REVIEW_JWT=$(kubectl get secret \
      $(kubectl get serviceaccount vault -o jsonpath='{.secrets[0].name}') \
      -o jsonpath='{ .data.token }' | base64 --decode)
   

   Copy

3. Get the Kubernetes certificate authority for the service account and set it to the KUBE_CA_CERT environment variable.

   $ export KUBE_CA_CERT=$(kubectl get secret \
      $(kubectl get serviceaccount vault -o jsonpath='{.secrets[0].name}') \
      -o jsonpath='{ .data.ca\.crt }' | base64 --decode)
   

   $ export KUBE_CA_CERT=$(kubectl get secret \
      $(kubectl get serviceaccount vault -o jsonpath='{.secrets[0].name}') \
      -o jsonpath='{ .data.ca\.crt }' | base64 --decode)
   

   Copy

4. Get the Kubernetes cluster endpoint and set it to the KUBE_HOST environment variable.

   $ export KUBE_HOST=$(kubectl config view --raw --minify --flatten \
      -o jsonpath='{.clusters[].cluster.server}')
   

   $ export KUBE_HOST=$(kubectl config view --raw --minify --flatten \
      -o jsonpath='{.clusters[].cluster.server}')
   

   Copy

5. Configure the Vault Kubernetes auth method to use the service account token.

   $ vault write auth/kubernetes/config \
      token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
      kubernetes_host="$KUBE_HOST" \
      kubernetes_ca_cert="$KUBE_CA_CERT"
   

   $ vault write auth/kubernetes/config \
      token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
      kubernetes_host="$KUBE_HOST" \
      kubernetes_ca_cert="$KUBE_CA_CERT"
   

   Copy

   Output:

   Success! Data written to: auth/kubernetes/config
   

   Success! Data written to: auth/kubernetes/config
   

   Copy

Deploy an example workload

Now that the clients have been deployed, it is time to deploy an application workload. This tutorial will use the HashiCups demo application.

1. Issue the following command to clone the repository to the development host.

   $ git clone https://github.com/hashicorp/vault-guides.git
   

   $ git clone https://github.com/hashicorp/vault-guides.git
   

   Copy

2. Change directory into the example repository.

   cd vault-guides/cloud/eks-hcp-vault/
   

   cd vault-guides/cloud/eks-hcp-vault/
   

   Copy

Deploy a database to Kubernetes

1. Deploy a PostgreSQL database. This contains data for various coffees related to a demo application, all hosted in the products database.

   $ kubectl apply -f postgres.yaml
   

   $ kubectl apply -f postgres.yaml
   

   Copy

   Output:

   service/postgres created
   deployment.apps/postgres created
   

   service/postgres created
   deployment.apps/postgres created
   

   Copy

2. Verify you've deployed the PostgreSQL database in your Kubernetes cluster.

   $ kubectl get pods
   

   $ kubectl get pods
   

   Copy

   Example output:

   NAME                                   READY   STATUS    RESTARTS   AGE
   postgres-5bbfc8bb5c-9px77              1/1     Running   0          31s
   vault-agent-injector-c8fd9fc5f-jhhw9   1/1     Running   0          2m34s
   

   NAME                                   READY   STATUS    RESTARTS   AGE
   postgres-5bbfc8bb5c-9px77              1/1     Running   0          31s
   vault-agent-injector-c8fd9fc5f-jhhw9   1/1     Running   0          2m34s
   

Add the database role to Vault

The product API needs to read the database username and password from Vault. Create the role for the product service account to generate database credentials.

1. Enable the database secrets engine.

   $ vault secrets enable database
   

   $ vault secrets enable database
   

   Copy

   Output:

   Success! Enabled the database secrets engine at: database/
   

   Success! Enabled the database secrets engine at: database/
   

   Copy

2. Set the POSTGRES_IP environment variable to the load balancer DNS hostname.

   $ export POSTGRES_IP=$(kubectl get service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' \
      postgres)
   

   $ export POSTGRES_IP=$(kubectl get service -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' \
      postgres)
   

   Copy

3. Create the database configuration that allows HCP Vault to configure Postgres.

   $ vault write database/config/products \
       plugin_name=postgresql-database-plugin \
       allowed_roles="*" \
       connection_url="postgresql://{{username}}:{{password}}@${POSTGRES_IP}:5432/products?sslmode=disable" \
       username="postgres" \
       password="password"
   

   $ vault write database/config/products \
       plugin_name=postgresql-database-plugin \
       allowed_roles="*" \
       connection_url="postgresql://{{username}}:{{password}}@${POSTGRES_IP}:5432/products?sslmode=disable" \
       username="postgres" \
       password="password"
   

   Copy

4. Create a database role for product that allows Vault to issue database passwords.

   $ vault write database/roles/product \
       db_name=products \
       creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
           GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
       revocation_statements="ALTER ROLE \"{{name}}\" NOLOGIN;"\
       default_ttl="1h" \
       max_ttl="24h"
   

   $ vault write database/roles/product \
       db_name=products \
       creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
           GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
       revocation_statements="ALTER ROLE \"{{name}}\" NOLOGIN;"\
       default_ttl="1h" \
       max_ttl="24h"
   

   Copy

   Output:

   Success! Data written to: database/roles/product
   

   Success! Data written to: database/roles/product
   

   Copy

5. Request a new set of PostgreSQL database credentials for the product role.

   $ vault read database/creds/product
   

   $ vault read database/creds/product
   

   Copy

   Example output:

   Key                Value
   ---                -----
   lease_id           database/creds/product/zeXVyjlT550wHYEDOhF5ckDC.8fmet
   lease_duration     1h
   lease_renewable    true
   password           gT8FvhyMtLuF-SCbw4mF
   username           v-token-hc-product-Qcg6fDERVvya93uwvbCy-1652888425
   

   Key                Value
   ---                -----
   lease_id           database/creds/product/zeXVyjlT550wHYEDOhF5ckDC.8fmet
   lease_duration     1h
   lease_renewable    true
   password           gT8FvhyMtLuF-SCbw4mF
   username           v-token-hc-product-Qcg6fDERVvya93uwvbCy-1652888425
   

Configure Vault policy for database credentials

1. Create a file called product.hcl that allows the product service to read the database credentials specific to product.

   $ cat > product.hcl << EOF
   path "database/creds/product" {
     capabilities = ["read"]
   }
   EOF
   

   $ cat > product.hcl << EOF
   path "database/creds/product" {
     capabilities = ["read"]
   }
   EOF
   

   Copy

2. Create a new product policy.

   $ vault policy write product ./product.hcl
   

   $ vault policy write product ./product.hcl
   

   Copy

   Output:

   Success! Uploaded policy: product
   

   Success! Uploaded policy: product
   

   Copy

3. Configure Vault to associate the product service with a Kubernetes service account. This allows the product service account in Kubernetes to get a Vault token.

   $ vault write auth/kubernetes/role/product \
       bound_service_account_names=product \
       bound_service_account_namespaces=default \
       policies=product \
       ttl=1h
   

   $ vault write auth/kubernetes/role/product \
       bound_service_account_names=product \
       bound_service_account_namespaces=default \
       policies=product \
       ttl=1h
   

   Copy

   Output:

   Success! Data written to: auth/kubernetes/role/product
   

   Success! Data written to: auth/kubernetes/role/product
   

   Copy

Deploy the product API

1. Make sure that you include your HCP Vault namespace in the deployment that will access database credentials.

   $ grep ${VAULT_NAMESPACE} product.yaml
   

   $ grep ${VAULT_NAMESPACE} product.yaml
   

   Copy

   Output:

   vault.hashicorp.com/namespace: "admin"
   

   vault.hashicorp.com/namespace: "admin"
   

   Copy

2. Deploy the product service.

   $ kubectl apply -f product.yaml
   

   $ kubectl apply -f product.yaml
   

   Copy

   Output:

   service/product created
   serviceaccount/product created
   deployment.apps/product created
   

   service/product created
   serviceaccount/product created
   deployment.apps/product created
   

   Copy

3. The product deployment should initialize.

   $ kubectl get pods
   

   $ kubectl get pods
   

   Copy

   Example output:

   NAME                                    READY   STATUS    RESTARTS   AGE
   postgres-5bd8c648fd-q6jhr               1/1     Running   0          25m
   product-6f7ff64c75-zrcwd                2/2     Running   0          27s
   vault-agent-injector-7d4ccf785f-dkt7c   1/1     Running   0          29m
   

   NAME                                    READY   STATUS    RESTARTS   AGE
   postgres-5bd8c648fd-q6jhr               1/1     Running   0          25m
   product-6f7ff64c75-zrcwd                2/2     Running   0          27s
   vault-agent-injector-7d4ccf785f-dkt7c   1/1     Running   0          29m
   

4. Port forward the web service locally to port 9090.

   $ kubectl port-forward service/product 9090
   

   $ kubectl port-forward service/product 9090
   

   Copy

   Output:

   Forwarding from 127.0.0.1:9090 -> 9090
   Forwarding from [::1]:9090 -> 9090
   

   Forwarding from 127.0.0.1:9090 -> 9090
   Forwarding from [::1]:9090 -> 9090
   

   Copy

5. Open another terminal and make a request to localhost:9090/coffees to check if the web service can pull coffee information from the database.

   $ curl -s localhost:9090/coffees | jq .
   

   $ curl -s localhost:9090/coffees | jq .
   

   Copy

   Output:

   [
     {
       "id": 1,
       "name": "Packer Spiced Latte",
       "teaser": "Packed with goodness to spice up your images",
       "description": "",
       "price": 350,
       "image": "/packer.png",
       "ingredients": [
         {
           "ingredient_id": 1
         },
         {
           "ingredient_id": 2
         },
         {
           "ingredient_id": 4
         }
       ]
     },
     {
       "id": 2,
       "name": "Vaulatte",
       "teaser": "Nothing gives you a safe and secure feeling like a Vaulatte",
       "description": "",
       "price": 200,
       "image": "/vault.png",
       "ingredients": [
         {
           "ingredient_id": 1
         },
         {
           "ingredient_id": 2
         }
       ]
     },
     {
       "id": 3,
       "name": "Nomadicano",
       "teaser": "Drink one today and you will want to schedule another",
       "description": "",
       "price": 150,
       "image": "/nomad.png",
       "ingredients": [
         {
           "ingredient_id": 1
         },
         {
           "ingredient_id": 3
         }
       ]
     },
     {
       "id": 4,
       "name": "Terraspresso",
       "teaser": "Nothing kickstarts your day like a provision of Terraspresso",
       "description": "",
       "price": 150,
       "image": "/terraform.png",
       "ingredients": [
         {
           "ingredient_id": 1
         }
       ]
     },
     {
       "id": 5,
       "name": "Vagrante espresso",
       "teaser": "Stdin is not a tty",
       "description": "",
       "price": 200,
       "image": "/vagrant.png",
       "ingredients": [
         {
           "ingredient_id": 1
         }
       ]
     },
     {
       "id": 6,
       "name": "Connectaccino",
       "teaser": "Discover the wonders of our meshy service",
       "description": "",
       "price": 250,
       "image": "/consul.png",
       "ingredients": [
         {
           "ingredient_id": 1
         },
         {
           "ingredient_id": 5
         }
       ]
     }
   ]
   

   [
     {
       "id": 1,
       "name": "Packer Spiced Latte",
       "teaser": "Packed with goodness to spice up your images",
       "description": "",
       "price": 350,
       "image": "/packer.png",
       "ingredients": [
         {
           "ingredient_id": 1
         },
         {
           "ingredient_id": 2
         },
         {
           "ingredient_id": 4
         }
       ]
     },
     {
       "id": 2,
       "name": "Vaulatte",
       "teaser": "Nothing gives you a safe and secure feeling like a Vaulatte",
       "description": "",
       "price": 200,
       "image": "/vault.png",
       "ingredients": [
         {
           "ingredient_id": 1
         },
         {
           "ingredient_id": 2
         }
       ]
     },
     {
       "id": 3,
       "name": "Nomadicano",
       "teaser": "Drink one today and you will want to schedule another",
       "description": "",
       "price": 150,
       "image": "/nomad.png",
       "ingredients": [
         {
           "ingredient_id": 1
         },
         {
           "ingredient_id": 3
         }
       ]
     },
     {
       "id": 4,
       "name": "Terraspresso",
       "teaser": "Nothing kickstarts your day like a provision of Terraspresso",
       "description": "",
       "price": 150,
       "image": "/terraform.png",
       "ingredients": [
         {
           "ingredient_id": 1
         }
       ]
     },
     {
       "id": 5,
       "name": "Vagrante espresso",
       "teaser": "Stdin is not a tty",
       "description": "",
       "price": 200,
       "image": "/vagrant.png",
       "ingredients": [
         {
           "ingredient_id": 1
         }
       ]
     },
     {
       "id": 6,
       "name": "Connectaccino",
       "teaser": "Discover the wonders of our meshy service",
       "description": "",
       "price": 250,
       "image": "/consul.png",
       "ingredients": [
         {
           "ingredient_id": 1
         },
         {
           "ingredient_id": 5
         }
       ]
     }
   ]
   

When you are done, return to the terminal with the port-forward command and type Ctrl + C to exit.

Clean up

1. Delete the product API.

   $  kubectl delete -f product.yaml
   

   $  kubectl delete -f product.yaml
   

   Copy

2. Delete the product role.

   $ vault delete auth/kubernetes/role/product
   

   $ vault delete auth/kubernetes/role/product
   

   Copy

3. Delete the product policy.

   $ vault policy delete product
   

   $ vault policy delete product
   

   Copy

4. Delete the product database role.

   $ vault delete database/roles/product
   

   $ vault delete database/roles/product
   

   Copy

5. Revoke all leases for database credentials.

   $ vault lease revoke -prefix database/creds/product
   

   $ vault lease revoke -prefix database/creds/product
   

   Copy

6. Delete the PostgreSQL database.

   $ kubectl delete -f postgres.yaml
   

   $ kubectl delete -f postgres.yaml
   

   Copy

7. Delete the database secrets engine configuration.

   $ vault delete database/config/product
   

   $ vault delete database/config/product
   

   Copy

8. Delete the Helm installation for HCP Vault.

   $ helm delete vault
   

   $ helm delete vault
   

   Copy

9. Disable the database secrets engine in HCP Vault.

   $ vault secrets disable database
   

   $ vault secrets disable database
   

   Copy

10. Disable the Kubernetes auth method in HCP Vault.

    $ vault auth disable kubernetes
    

    $ vault auth disable kubernetes
    

    Copy

11. Delete HCP Vault and the HVN.

Next steps

In this tutorial, you connected Vault clients on Amazon EKS to HCP Vault and retrieved PostgreSQL database credential dynamically. To learn more about the Vault features introduced in this tutorial, refer to the following tutorials.

• Dynamic Secrets: Database Secrets Engine
• Integrate a Kubernetes Cluster with an External Vault
• Provision an EKS Cluster (AWS)