HashiConf Our community conference is taking place in San Francisco and online October 10-12. Register now
Terraform
Configuration Language

Perform CRUD Operations with Providers

  • 21min

  • Terraform
  • Video

Note: We recommend using the Terraform Plugin Framework for new provider development because it offers significant advantages compared to the SDKv2. Refer to the Plugin Framework tutorials to learn how to create providers using the framework.

In this tutorial, you will use a Terraform provider to interact with a fictional coffee-shop application called Hashicups. In the process, you will learn how providers map target APIs to Terraform in order to create, read, update, and delete resources.

Later in the track, you will re-create the HashiCups provider discussed in this tutorial based on the Terraform Plugin SDK v2.

Terraform plugins

Terraform is comprised of Terraform Core and Terraform Plugins.

core-plugins-api

  1. Terraform Core reads the configuration and builds the resource dependency graph.
  2. Terraform Plugins (providers and provisioners) bridge Terraform Core and their respective target APIs. Terraform provider plugins implement resources via basic CRUD (create, read, update, and delete) APIs to communicate with third party services.

Tip

We recommend Terraform plugins consume an external API client library, as shown in the diagram above. If one doesn't exist, you should create one. This is aligned with modern coding practices of keeping software projects modular.

Upon terraform plan or terraform apply, Terraform Core asks the Terraform provider to perform an action via a RPC interface. The provider attempts to fulfill the request by invoking a CRUD operation against the target API's client library. This process enforces a clear separation of concerns. Providers are able to serve as an abstraction of a client library.

While most Terraform providers manage cloud infrastructure (e.g. AWS, Azure and GCP providers), providers can serve as an interface to any API and allow Terraform to potentially manage any resource.

Introduction to HashiCups

HashiCups is a demo application that allows you to view and order customized HashiCorp branded coffee.

The API has a number of unprotected and protected endpoints. You can list all coffees and ingredients for a particular coffee without authentication. Once authenticated, you can create, read, update and delete (CRUD) orders.

The Terraform HashiCups provider interfaces with the product's REST API through a GoLang client library. This allows you to manage HashiCups orders through Terraform.

Prerequisites

To follow this tutorial, you need:

It also assumes that you are familiar with the usual Terraform plan/apply workflow. If you're new to Terraform itself, refer first to the Getting Started track.

Initialize HashiCups locally

The HashiCups provider requires a running instance of HashiCups. In your terminal, clone the HashiCups Provider repository and navigate to it.

$ git clone https://github.com/hashicorp/learn-terraform-hashicups-provider && cd learn-terraform-hashicups-provider

Navigate to the docker_compose directory and run docker compose up to spin up a local instance of HashiCups on port :19090.

$ cd docker_compose && docker compose up

Leave this process running in your terminal window. This will contain HashiCups' log messages.

In another terminal window, verify that HashiCups is running by sending a request to its health check endpoint.

$ curl localhost:19090/health
ok

Install HashiCups provider

With Terraform 0.13+, you must specify all required providers and their respective source in your Terraform configuration. A provider source string is comprised of [hostname]/[namespace]/[name].

  • Both hostname and namespace are optional for providers in the hashicorp namespace in the Terraform Registry.
  • If hostname is omitted, Terraform will use the Terraform Registry hostname as the default hostname.
  • namespace is always required when hostname is set.

For third party Terraform providers, you can choose any arbitrary identifier (comprised of letters and hyphens) for the hostname and namespace.

When you run terraform init, Terraform will attempt to download the provider from the Terraform Registry.

If Terraform can't download the provider from the Terraform Registry (for example if the provider is local, or because of firewall restrictions), you can specify the installation method configuration explicitly. Otherwise, Terraform will implicitly attempt to find the provider locally in the appropriate subdirectory within the user plugins directory, ~/.terraform.d/plugins/${host_name}/${namespace}/${type}/${version}/${target} or %APPDATA%\terraform.d\plugins\${host_name}/${namespace}/${type}/${version}/${target}.

Note For Terraform versions before v0.13.0, Terraform will attempt to locate the provider by binary name in the root user plugins directory.

In order to use any third-party provider not published to the registry, you must download the binary then move it into the appropriate subdirectory within the user plugins directory. Because HashiCups is a third-party provider used primarily for demos and education, you will do this. The provider source string for the HashiCups provider will be hashicorp.com/edu/hashicups.

First, download the HashiCups provider binary. Because providers are Go binaries, you can download a pre-compiled binary or build it directly from source. Building from source requires some Go experience.

Download the HashiCups provider v0.3.1 binary that matches your system. A full list of binaries can be found on the Terraform HashiCups Provider release page. Rename the file to terraform-provider-hashicups.

Download the HashiCups provider for MacOS.

curl -LO https://github.com/hashicorp/terraform-provider-hashicups/releases/download/v0.3.1/terraform-provider-hashicups_0.3.1_darwin_amd64.zip

Create the appropriate subdirectory within the user plugins directory for the HashiCups provider.

$ mkdir -p ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.3.1/darwin_amd64

Then, unzip the downloaded binary into the appropriate user plugins directory.

$ unzip terraform-provider-hashicups_0.3.1_darwin_amd64.zip -d ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.3.1/darwin_amd64

Finally, make the binary executable.

$ chmod +x ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.3.1/darwin_amd64/terraform-provider-hashicups_v0.3.1

Now that the provider is in your user plugins directory, you can use the provider in your Terraform configuration.

Create new HashiCups user

HashiCups requires a username and password to generate a JSON web token (JWT) which is used to authenticate against protected endpoints. You will use this user to authenticate to the HashiCups provider to manage your orders.

Create a user on HashiCups named education with the password test123.

$ curl -X POST localhost:19090/signup -d '{"username":"education", "password":"test123"}'
{"UserID":1,"Username":"education","token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTEwNzgwODUsInVzZXJfaWQiOjIsInVzZXJuYW1lIjoiZWR1Y2F0aW9uIn0.CguceCNILKdjOQ7Gx0u4UAMlOTaH3Dw-fsll2iXDrYU"}

Then, authenticate to HashiCups. This will return the userID, username, and a JSON Web Token (JWT) authorization token. You will use the JWT authorization token later to retrieve your orders.

$ curl -X POST localhost:19090/signin -d '{"username":"education", "password":"test123"}'
{"UserID":1,"Username":"education","token":"eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTEwNzgwODUsInVzZXJfaWQiOjIsInVzZXJuYW1lIjoiZWR1Y2F0aW9uIn0.CguceCNILKdjOQ7Gx0u4UAMlOTaH3Dw-fsll2iXDrYU"}

Set the HASHICUPS_TOKEN environment variable to the token you retrieved from invoking the /signin endpoint. You will use this later in the tutorial to verify your HashiCups order has been created, updated and deleted.

$ export HASHICUPS_TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE1OTEwNzgwODUsInVzZXJfaWQiOjIsInVzZXJuYW1lIjoiZWR1Y2F0aW9uIn0.CguceCNILKdjOQ7Gx0u4UAMlOTaH3Dw-fsll2iXDrYU

The terminal containing your HashiCups logs will have recorded both operations.

api_1  | 2020-12-10T09:19:50.601Z [INFO]  Handle User | signup
api_1  | 2020-12-10T09:19:59.601Z [INFO]  Handle User | signin

Now that the HashiCups app is running, you're ready to interact with it using the Terraform provider.

Initialize workspace

Add the following to your main.tf file. This is required for Terraform 0.13+.

main.tf
terraform {
  required_providers {
    hashicups = {
      version = "~> 0.3.1"
      source  = "hashicorp.com/edu/hashicups"
    }
  }
}

Since HashiCups is a third-party provider, the hostname and namespace values in the source string are arbitrary. For more information on provider source, reference the provider source documentation.

Then, initialize your Terraform workspace. If your Hashicups provider is located in the correct directory, it should successfully initialize. Otherwise, move your HashiCups provider to the correct directory: ~/.terraform.d/plugins/${name}/${version}/${os}_${arch}.

$ terraform init

Create order

Add the following to your main.tf file.

This authenticates the HashiCups provider, creates an order and returns the order details as a Terraform output value. The order contains a total of 4 coffees: 2 of each coffee_id 3 and 2.

main.tf
provider "hashicups" {
  username = "education"
  password = "test123"
}

resource "hashicups_order" "edu" {
  items {
    coffee {
      id = 3
    }
    quantity = 2
  }
  items {
    coffee {
      id = 2
    }
    quantity = 2
  }
}

output "edu_order" {
  value = hashicups_order.edu
}

Run terraform apply to create the order. Notice how the execution plan shows a proposed order, with additional information about the order. Confirm the apply step with a yes.

$ terraform apply

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # hashicups_order.edu will be created
  + resource "hashicups_order" "edu" {
      + id           = (known after apply)
      + last_updated = (known after apply)

      + items {
          + quantity = 2

          + coffee {
              + description = (known after apply)
              + id          = 3
              + image       = (known after apply)
              + name        = (known after apply)
              + price       = (known after apply)
              + teaser      = (known after apply)
            }
        }
      + items {
          + quantity = 2

          + coffee {
              + description = (known after apply)
              + id          = 2
              + image       = (known after apply)
              + name        = (known after apply)
              + price       = (known after apply)
              + teaser      = (known after apply)
            }
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + edu_order = {
      + id           = (known after apply)
      + items        = [
          + {
              + coffee   = [
                  + {
                      + description = (known after apply)
                      + id          = 3
                      + image       = (known after apply)
                      + name        = (known after apply)
                      + price       = (known after apply)
                      + teaser      = (known after apply)
                    },
                ]
              + quantity = 2
            },
          + {
              + coffee   = [
                  + {
                      + description = (known after apply)
                      + id          = 2
                      + image       = (known after apply)
                      + name        = (known after apply)
                      + price       = (known after apply)
                      + teaser      = (known after apply)
                    },
                ]
              + quantity = 2
            },
        ]
      + last_updated = (known after apply)
    }

Once the apply completes, the provider saves the resource's state. If you're running Terraform locally, your terraform.tfstate file contains this state. You can also view the state by running terraform state show <resource_name>.

$ terraform state show hashicups_order.edu
# hashicups_order.edu:
resource "hashicups_order" "edu" {
  id = "1"

  items {
    quantity = 2

    coffee {
      id     = 3
      image  = "/nomad.png"
      name   = "Nomadicano"
      price  = 150
      teaser = "Drink one today and you will want to schedule another"
    }
  }
  items {
    quantity = 2

    coffee {
      id     = 2
      image  = "/vault.png"
      name   = "Vaulatte"
      price  = 200
      teaser = "Nothing gives you a safe and secure feeling like a Vaulatte"
    }
  }
}

The (known after apply) values in the execution plan during the terraform apply step have all been populated, since the order was successfully created.

Verify order created

When you created an order in HashiCups using Terraform, the terminal containing your HashiCups logs will have recorded operations invoked by the HashiCups provider.

api_1  | 2020-12-10T10:47:47.838Z [INFO]  Handle User | signin
api_1  | 2020-12-10T10:47:54.841Z [INFO]  Handle User | signin
api_1  | 2020-12-10T10:47:54.853Z [INFO]  Handle Orders | CreateOrder
api_1  | 2020-12-10T10:47:54.866Z [INFO]  Handle Orders | GetUserOrder

The provider invoked a total of 4 operations.

  1. The provider invoked the first signin operation when you ran terraform apply to retrieve the current state of the resources. Because there are no resources, it only authenticated the user.
  2. The provider invoked the second signin operation after you confirmed the apply run. The provider authenticated using the provided credentials to retrieve and save the JWT token.
  3. The provider invoked the CreateOrder operation to create the order defined by the Terraform configuration. Since this is a protected endpoint, it used the saved JWT token from the signin operation.
  4. After the order was created, the provider invoked the GetUserOrder operation to retrieve the order detail. Since this is a protected endpoint, it used the saved JWT token from the signin operation.

Verify the order was created by retrieving the order details via the API.

$ curl -X GET  -H "Authorization: ${HASHICUPS_TOKEN}" localhost:19090/orders/1
{"id":1,"items":[{"coffee":{"id":3,"name":"Nomadicano","teaser":"Drink one today and you will want to schedule another","description":"","price":150,"image":"/nomad.png","ingredients":null},"quantity":2},{"coffee":{"id":2,"name":"Vaulatte","teaser":"Nothing gives you a safe and secure feeling like a Vaulatte","description":"","price":200,"image":"/vault.png","ingredients":null},"quantity":2}]}

The order's properties should be the same as that of your hashicups_order.edu resource.

Update order

Now, change your order by updating the order resource through Terraform. In your main.tf, update the coffee quantities in the hashicups_order.edu block.

main.tf
resource "hashicups_order" "edu" {
  items {
    coffee {
      id = 3
    }
    quantity = 3
  }
  items {
    coffee {
      id = 2
    }
    quantity = 1
  }
}

Run terraform apply to update the order. Notice how the execution plan reflects the order change. Terraform is able to determine that these changes could be done in place rather than destroying the current order and creating a new one. This is because the modified property (quantity) does not have ForceNew set to true.

Confirm the apply step with a yes.

$ terraform apply
hashicups_order.edu: Refreshing state... [id=1]

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # hashicups_order.edu will be updated in-place
  ~ resource "hashicups_order" "edu" {
        id = "2"

      ~ items {
          ~ quantity = 2 -> 3

            # (1 unchanged block hidden)
        }
      ~ items {
          ~ quantity = 2 -> 1

            # (1 unchanged block hidden)
        }
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Changes to Outputs:
  ~ edu_order = {
      ~ items        = [
          ~ {
              ~ quantity = 2 -> 3
                # (1 unchanged element hidden)
            },
          ~ {
              ~ quantity = 2 -> 1
                # (1 unchanged element hidden)
            },
        ]
        # (2 unchanged elements hidden)
    }

Once the apply completes, view the output. Notice that while the order_id remained the same (1), the provider updated the order's coffee quantity and assigned the current time to the newly created last_updated field.

Outputs:

edu_order = {
  "id" = "1"
  "items" = tolist([
    {
      "coffee" = tolist([
        {
          "description" = ""
          "id" = 3
          "image" = "/nomad.png"
          "name" = "Nomadicano"
          "price" = 150
          "teaser" = "Drink one today and you will want to schedule another"
        },
      ])
      "quantity" = 3
    },
    {
      "coffee" = tolist([
        {
          "description" = ""
          "id" = 2
          "image" = "/vault.png"
          "name" = "Vaulatte"
          "price" = 200
          "teaser" = "Nothing gives you a safe and secure feeling like a Vaulatte"
        },
      ])
      "quantity" = 1
    },
  ])
  "last_updated" = tostring(null)
}

Verify order updated

Check the terminal containing your HashiCups logs for the recorded operations invoked by the HashiCups provider.

api_1  | 2020-12-10T11:13:14.964Z [INFO]  Handle User | signin
api_1  | 2020-12-10T11:13:14.973Z [INFO]  Handle Orders | GetUserOrder
api_1  | 2020-12-10T11:13:15.084Z [INFO]  Handle User | signin
api_1  | 2020-12-10T11:13:48.416Z [INFO]  Handle Orders | UpdateOrder
api_1  | 2020-12-10T11:13:48.421Z [INFO]  Handle Orders | GetUserOrder

The provider invoked a total of 5 operations.

  1. The provider invoked the first signin operation when you ran terraform apply to retrieve the current state of the resources. Because there are resources in the state...
  2. The provider invoked the GetUserOrder operation to reconcile any potential differences between the current state (state in HashiCups) and the state stored by Terraform.
  3. The provider invoked the second signin operation after Terraform's state was updated.
  4. The provider invoked the UpdateOrder operation to update the order to the one defined by the Terraform configuration. Since this is a protected endpoint, it used the saved JWT token from the signin operation.
  5. After the order was updated, the provider invoked the GetUserOrder operation to retrieve the order details. Since this is a protected endpoint, it used the saved JWT token from the signin operation.

Verify the order was updated by retrieving the order details via the API.

$ curl -X GET  -H "Authorization: ${HASHICUPS_TOKEN}" localhost:19090/orders/1
{"id":1,"items":[{"coffee":{"id":3,"name":"Nomadicano","teaser":"Drink one today and you will want to schedule another","description":"","price":150,"image":"/nomad.png","ingredients":null},"quantity":3},{"coffee":{"id":2,"name":"Vaulatte","teaser":"Nothing gives you a safe and secure feeling like a Vaulatte","description":"","price":200,"image":"/vault.png","ingredients":null},"quantity":1}]}

The order's properties should be the same as that of your updated hashicups_order.edu resource. There should be 3 Nomadicano and 1 Vaulatte.

Read coffee ingredients

Add a data block and an output block to your main.tf file. This will retrieve ingredients for the first coffee from your order. The data block retrieves additional information about a resource, which enables it to be referenced by another Terraform resource. In this example, it's used by an output block to display the coffee ingredients.

main.tf
data "hashicups_ingredients" "first_coffee" {
  coffee_id = hashicups_order.edu.items[0].coffee[0].id
}

output "first_coffee_ingredients" {
  value = data.hashicups_ingredients.first_coffee
}

Run terraform apply to retrieve the ingredients. Confirm the apply step with a yes.

$ terraform apply
hashicups_order.edu: Refreshing state... [id=1]
data.hashicups_ingredients.first_coffee: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

first_coffee_ingredients = {
  "coffee_id" = 3
  "id" = "3"
  "ingredients" = tolist([
    {
      "id" = 1
      "name" = "ingredient - Espresso"
      "quantity" = 20
      "unit" = "ml"
    },
    {
      "id" = 3
      "name" = "ingredient - Hot Water"
      "quantity" = 100
      "unit" = "ml"
    },
  ])
}

As you can see, ingredients of a Nomadicano is 20 ml espresso and 100 ml hot water.

Verify operation

Check the terminal containing your HashiCups logs for the recorded operations invoked by the HashiCups provider.

api_1  | 2020-12-10T11:36:17.907Z [INFO]  Handle User | signin
api_1  | 2020-12-10T11:36:17.915Z [INFO]  Handle Orders | GetUserOrder
api_1  | 2020-12-10T11:36:17.921Z [INFO]  Handle Coffee Ingredients
api_1  | 2020-12-10T11:36:18.035Z [INFO]  Handle User | signin
api_1  | 2020-12-10T11:36:27.513Z [INFO]  Handle User | signin

After the provider reconciles the state (first two operations), it invokes Handle Coffee Ingredients which retrieves the coffee ingredients.

Delete order

Finally, you can delete resources in Terraform. Run terraform destroy. Remember to confirm the destroy step with a yes.

$ terraform destroy

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  # hashicups_order.edu will be destroyed
  - resource "hashicups_order" "edu" {
      - id           = "1" -> null
      - last_updated = "Thursday, 10-Dec-20 04:43:04 UTC" -> null

      - items {
          - quantity = 3 -> null

          - coffee {
              - id     = 3 -> null
              - image  = "/nomad.png" -> null
              - name   = "Nomadicano" -> null
              - price  = 150 -> null
              - teaser = "Drink one today and you will want to schedule another" -> null
            }
        }
      - items {
          - quantity = 1 -> null

          - coffee {
              - id     = 2 -> null
              - image  = "/vault.png" -> null
              - name   = "Vaulatte" -> null
              - price  = 200 -> null
              - teaser = "Nothing gives you a safe and secure feeling like a Vaulatte" -> null
            }
        }
    }

Plan: 0 to add, 0 to change, 1 to destroy.

Changes to Outputs:
  - edu_order                = {
      - id           = "2"
      - items        = [
          - {
              - coffee   = [
                  - {
                      - description = ""
                      - id          = 3
                      - image       = "/nomad.png"
                      - name        = "Nomadicano"
                      - price       = 150
                      - teaser      = "Drink one today and you will want to schedule another"
                    },
                ]
              - quantity = 3
            },
          - {
              - coffee   = [
                  - {
                      - description = ""
                      - id          = 2
                      - image       = "/vault.png"
                      - name        = "Vaulatte"
                      - price       = 200
                      - teaser      = "Nothing gives you a safe and secure feeling like a Vaulatte"
                    },
                ]
              - quantity = 1
            },
        ]
      - last_updated = "Thursday, 10-Dec-20 04:43:04 UTC"
    } -> null
  - first_coffee_ingredients = {
      - coffee_id   = 3
      - id          = "3"
      - ingredients = [
          - {
              - id       = 1
              - name     = "ingredient - Espresso"
              - quantity = 20
              - unit     = "ml"
            },
          - {
              - id       = 3
              - name     = "ingredient - Hot Water"
              - quantity = 100
              - unit     = "ml"
            },
        ]
    } -> null

When the destroy step completes, the provider has destroyed the order resource, reflected by an empty state file.

Verify order deleted

Check the terminal containing your HashiCups logs for the recorded operations invoked by the HashiCups provider.

api_1  | 2020-12-10T11:42:59.915Z [INFO]  Handle User | signin
api_1  | 2020-12-10T11:42:59.924Z [INFO]  Handle Orders | GetUserOrder
api_1  | 2020-12-10T11:42:59.929Z [INFO]  Handle Coffee Ingredients
api_1  | 2020-12-10T11:43:01.683Z [INFO]  Handle User | signin
api_1  | 2020-12-10T11:43:01.689Z [INFO]  Handle Orders | DeleteOrder

After the provider reconciles the state (first three operations), it destroys the order.

Verify the order was deleted by retrieving the order details via the API.

$ curl -X GET  -H "Authorization: ${HASHICUPS_TOKEN}" localhost:19090/orders/1
{}

The order is blank, as expected.

Next steps

In this tutorial, you created, read, updated, and deleted an order resource in HashiCorp's demo application. In the process, you've learned how Terraform interacts with target APIs via providers.

A full list of official, partner, and community Terraform providers can be found on the Terraform Provider Registry. We encourage you to find a provider you're interested in and experiment!

terraform-registry-walkthrough

To learn more about provider source, refer to Automatic Installation of Third-Party Providers with Terraform 0.13 blog post.

To learn more about interpolation, refer to the Configuration Language documentation.

In the following tutorials, you will learn how to write a Terraform provider using the Plugins SDK v2 by re-implementing the HashiCups provider. In the process, you will create data sources, authenticate the provider to the HashiCups client, and create resources with CRUD functionality.

Previous
Next

This tutorial also appears in:

  • 10 tutorials
    Call APIs with Custom SDK Providers
    Interact with APIs using Terraform providers. In these tutorials, use a provider as a bridge between Terraform and a target API. Then, extend Terraform by developing a custom Terraform provider based on the Terraform Plugin SDK.
    • Terraform
  • 5 tutorials
    Terraform 0.13 tutorials
    Try the new capabilities in Terraform 0.13. Use count and for each with modules, log into Terraform Cloud with the CLI, and use a third-party provider with the new provider source syntax.
    • Terraform