Implement Complex Read

11min
Terraform

Note: We recommend using the Terraform Plugin Framework for new provider development because it offers significant advantages compared to the SDKv2. Refer to the Plugin Framework tutorials to learn how to create providers using the framework.

In this tutorial, you will retrieve an order, a protected endpoint, to verify whether a Terraform provider that interacts with the API of a fictional coffee-shop application called Hashicups, has been authenticated to the API successfully. The order schema contains nested objects, making it a more complicated data structure than the previously defined coffee data source. To do this, you will:

Create a new order via API
There must be an existing order for you to retrieve it.
Define order data source.
You will add a scaffold that defines an empty schema and functions to retrieve order information.
Define order schema.
The schema reflects the response from invoking the /order/{orderId} endpoint. The order schema is considered complex because it nests a list of OrderItems.
Implement complex read.
This read functionality uses multiple nesting functions to flatten the response from /order/{orderId} and map it to the order schema.
Add order data source to the provider schema.
This allows you to use the data source in your configuration.

Prerequisites

To follow this tutorial, you need:

a Golang 1.15+ installed and configured.
the Terraform 0.14+ CLI installed locally.
Docker and Docker Compose to run an instance of HashiCups locally.

Navigate to your terraform-provider-hashicups directory. Next, fetch new changes.

$ git fetch

Then, checkout the auth-configuration branch. This step is optional but recommended to ensure that your code matches the code from the previous tutorials.

$ git checkout auth-configuration

Clone the auth-configuration branch of the Terraform HashiCups Provider repository.

$ git clone --branch auth-configuration https://github.com/hashicorp/terraform-provider-hashicups

Change into the cloned repository.

$ cd terraform-provider-hashicups

Then, install all the provider's dependencies.

$ go mod tidy

Build provider binary

Next, build the binary and move it into your user Terraform plugins directory. This allows you to sideload and test the custom provider. Select the tab for your operating system for specific instructions.

$ make install
go build -o terraform-provider-hashicups
mv terraform-provider-hashicups ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.2/darwin_amd64

Build the provider binary.

$ go build -o terraform-provider-hashicups

Retrieve your system's OS and ARCH. You can find a full list of supported GOOS and GOARCH here.

$ export OS_ARCH="$(go env GOHOSTOS)_$(go env GOHOSTARCH)"

Create the appropriate subdirectory within the user plugins directory for the HashiCups provider if it doesn't exist already.

$ mkdir -p ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.2/$OS_ARCH

Next, move the binary to the appropriate subdirectory within your user plugins directory.

$ mv terraform-provider-hashicups ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.2/$OS_ARCH

Build the provider binary.

$ go build -o terraform-provider-hashicups.exe

Create the appropriate subdirectory within the user plugins directory for the HashiCups provider if it doesn't exist already, replacing [OS_ARCH] with your system's OS_ARCH. You can find a full list of supported GO_ARCH here.

$ mkdir -p %APPDATA%\terraform.d\plugins\hashicorp.com\edu\hashicups\0.2\[OS_ARCH]

Then, move the binary to the appropriate subdirectory within your user plugins directory, replacing [OS_ARCH] with your system's OS_ARCH.

$ move terraform-provider-hashicups.exe %APPDATA%\terraform.d\plugins\hashicorp.com\edu\hashicups\0.2\[OS_ARCH]

Tip

The Perform CRUD operations with Providers tutorial explains why and how to sideload custom providers. Refer to it to learn more about where to install custom providers and how to reference them in your configuration.

Run HashiCups API

The HashiCups provider requires an instance of HashiCups. Navigate to the docker_compose directory then run docker-compose up to spin up a local instance of HashiCups on port :19090.

$ cd docker_compose && docker-compose up

Leave this terminal running.

In another terminal, verify that HashiCups is running by sending a request to its health check endpoint.

$ curl localhost:19090/health
ok

Navigate to your terraform-provider-hashicups directory.

$ cd ..

Your directory should have the following structure.

$ tree -L 2
.
├── Makefile
├── README.md
├── docker_compose
│   ├── conf.json
│   └── docker-compose.yml
├── examples
│   ├── coffee
│   ├── main.tf
├── go.mod
├── go.sum
├── hashicups
│   ├── data_source_coffee.go
│   └── provider.go
├── main.go
└── vendor

If you're stuck at any point during this tutorial, refer to the implement-complex-read branch to see the changes implemented in this tutorial.

Create an `order` via API

Before you can query a user's order, you must first create an order. Replace the authorization token with the token you generated in the sign in step.

$ curl -X POST -H "Authorization: ${HASHICUPS_TOKEN}" localhost:19090/orders -d '[{"coffee": { "id":1 }, "quantity":4}, {"coffee": { "id":3 }, "quantity":3}]'
{
  "id": 1,
  "items": [
    {
      "coffee": {
        "id": 1,
        "name": "Packer Spiced Latte",
        "teaser": "Packed with goodness to spice up your images",
        "description": "",
        "price": 350,
        "image": "/packer.png",
        "ingredients": null
      },
      "quantity": 4
    },
    {
      "coffee": {
        "id": 3,
        "name": "Nomadicano",
        "teaser": "Drink one today and you will want to schedule another",
        "description": "",
        "price": 150,
        "image": "/nomad.png",
        "ingredients": null
      },
      "quantity": 3
    }
  ]
}

Query the order using the order ID you received in the response above as a HTML parameter (localhost:19090/orders/{orderID}). The response should be the same as the response above.

$ curl -X GET -H "Authorization: ${HASHICUPS_TOKEN}" localhost:19090/orders/1

Define `order` data resource

Now, create a file named hashicups/data_source_order.go in your hashicups directory and add the following snippet.

hashicups/data_source_order.go

package hashicups

import (
  "context"
  "strconv"

  hc "github.com/hashicorp-demoapp/hashicups-client-go"
  "github.com/hashicorp/terraform-plugin-sdk/v2/diag"
  "github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema"
)

func dataSourceOrder() *schema.Resource {
  return &schema.Resource{
    ReadContext: dataSourceOrderRead,
    Schema: map[string]*schema.Schema{
      "id": &schema.Schema{
        Type:     schema.TypeInt,
        Required: true,
      },
      "items": &schema.Schema{
        Type:     schema.TypeList,
        Computed: true,
        Elem: &schema.Resource{
          Schema: map[string]*schema.Schema{
            "coffee_id": &schema.Schema{
              Type:     schema.TypeInt,
              Computed: true,
            },
            "coffee_name": &schema.Schema{
              Type:     schema.TypeString,
              Computed: true,
            },
            "coffee_teaser": &schema.Schema{
              Type:     schema.TypeString,
              Computed: true,
            },
            "coffee_description": &schema.Schema{
              Type:     schema.TypeString,
              Computed: true,
            },
            "coffee_price": &schema.Schema{
              Type:     schema.TypeInt,
              Computed: true,
            },
            "coffee_image": &schema.Schema{
              Type:     schema.TypeString,
              Computed: true,
            },
            "quantity": &schema.Schema{
              Type:     schema.TypeInt,
              Computed: true,
            },
          },
        },
      },
    },
  }
}

Format your code.

$ go fmt ./...

Notice how the schema mirrors the JSON response — only id and items exist on the top level.

The id is required because an order ID must be set so the data source knows which order to retrieve.
The items schema is a computed list (schema.TypeList) of objects (schema.Resource) containing coffee attributes and quantity.
The coffee attributes is flattened and mapped accordingly (coffee.id is mapped to coffee_id, etc...)

You will learn an alternative method to nest maps in the Implement Create tutorial, when you create new orders.

Implement complex read

Now that you have defined the order schema, add the dataSourceOrderRead function to hashicups/data_source_order.go. This function will retrieve the order and map its values to the order schema defined above.

hashicups/data_source_order.go

func dataSourceOrderRead(ctx context.Context, d *schema.ResourceData, m interface{}) diag.Diagnostics {
  c := m.(*hc.Client)

  // Warning or errors can be collected in a slice type
  var diags diag.Diagnostics

  orderID := strconv.Itoa(d.Get("id").(int))

  order, err := c.GetOrder(orderID)
  if err != nil {
    return diag.FromErr(err)
  }

  orderItems := flattenOrderItemsData(&order.Items)
  if err := d.Set("items", orderItems); err != nil {
    return diag.FromErr(err)
  }

  d.SetId(orderID)

  return diags
}

Format your code.

$ go fmt ./...

Notice the m (meta) input parameter contains the HashiCups API Client set by the ConfigureContextFunc defined above. If provider uses an unauthenticated API Client, this function will fail and return an error message.

The API Client's GetOrder function returns an order object. However, you must flatten this response to accurately map the response to the order schema. An order consists of an order ID and a list of coffee objects and their respective quantities. As a result, the order object must go through one flattening function to populate the list of coffee objects and their quantities.

Add the flattenOrderItemsData function to your hashicups/data_source_order.go file. This returns a list of order items.

hashicups/data_source_order.go

func flattenOrderItemsData(orderItems *[]hc.OrderItem) []interface{} {
  if orderItems != nil {
    ois := make([]interface{}, len(*orderItems), len(*orderItems))

    for i, orderItem := range *orderItems {
      oi := make(map[string]interface{})

      oi["coffee_id"] = orderItem.Coffee.ID
      oi["coffee_name"] = orderItem.Coffee.Name
      oi["coffee_teaser"] = orderItem.Coffee.Teaser
      oi["coffee_description"] = orderItem.Coffee.Description
      oi["coffee_price"] = orderItem.Coffee.Price
      oi["coffee_image"] = orderItem.Coffee.Image
      oi["quantity"] = orderItem.Quantity

      ois[i] = oi
    }

    return ois
  }

  return make([]interface{}, 0)
}

Format your code.

$ go fmt ./...

The flattenOrderItemsData function takes an *[]hc.OrderItem as orderItems. If orderItems is not nil, it will iterate through the slice and map its values into a map[string]interface{}. Notice how the function assigns the coffee attributes directly to its corresponding flattened attribute (orderItem.Coffee.ID -> coffee_id).

Add data source to provider

Now that you've defined the order data source, you can add it to your provider.

In your hashicups/provider.go file, add the order data source to the DataSourcesMap of your Provider() function. Resources and data sources names must follow the <provider>_<resource_name> convention.

hashicups/provider.go

// Provider -
func Provider() *schema.Provider {
  return &schema.Provider{
    Schema: map[string]*schema.Schema{
      "username": &schema.Schema{
        Type:        schema.TypeString,
        Optional:    true,
        DefaultFunc: schema.EnvDefaultFunc("HASHICUPS_USERNAME", nil),
      },
      "password": &schema.Schema{
        Type:        schema.TypeString,
        Optional:    true,
        Sensitive:   true,
        DefaultFunc: schema.EnvDefaultFunc("HASHICUPS_PASSWORD", nil),
      },
    },
    ResourcesMap: map[string]*schema.Resource{},
    DataSourcesMap: map[string]*schema.Resource{
        "hashicups_coffees":     dataSourceCoffees(),
        "hashicups_order":       dataSourceOrder(),
    },
    ConfigureFunc: providerConfigure,
  }
}

Test the provider

Now that you've implemented read and created the coffees data source, verify that it works.

First, confirm that you are in the terraform-provider-hashicups root directory.

$ pwd
~/terraform-provider-hashicups

$ make install
go build -o terraform-provider-hashicups
mv terraform-provider-hashicups ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.2/darwin_amd64

Build the provider binary.

$ go build -o terraform-provider-hashicups

Retrieve your system's OS and ARCH. You can find a full list of supported GOOS and GOARCH here.

$ export OS_ARCH="$(go env GOHOSTOS)_$(go env GOHOSTARCH)"

Create the appropriate subdirectory within the user plugins directory for the HashiCups provider if it doesn't exist already.

$ mkdir -p ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.2/$OS_ARCH

Next, move the binary to the appropriate subdirectory within your user plugins directory.

$ mv terraform-provider-hashicups ~/.terraform.d/plugins/hashicorp.com/edu/hashicups/0.2/$OS_ARCH

Build the provider binary.

$ go build -o terraform-provider-hashicups.exe

$ mkdir -p %APPDATA%\terraform.d\plugins\hashicorp.com\edu\hashicups\0.2\[OS_ARCH]

Then, move the binary to the appropriate subdirectory within your user plugins directory, replacing [OS_ARCH] with your system's OS_ARCH.

$ move terraform-provider-hashicups.exe %APPDATA%\terraform.d\plugins\hashicorp.com\edu\hashicups\0.2\[OS_ARCH]

Tip

Navigate to the terraform-provider-hashicups/examples directory. This contains a sample Terraform configuration for the Terraform HashiCups provider.

$ cd examples

Add the following Terraform configuration to main.tf.

examples/main.tf

data "hashicups_order" "order" {
  id = 1
}

output "order" {
  value = data.hashicups_order.order
}

Finally, initialize your workspace to refresh your HashiCups provider, then apply. This should return the first order in your output.

Notice how the coffee attributes are flattened.

$ terraform init && terraform apply --auto-approve
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Outputs:

order = {
  "id" = 1
  "items" = [
    {
      "coffee_description" = ""
      "coffee_id" = 1
      "coffee_image" = "/packer.png"
      "coffee_name" = "Packer Spiced Latte"
      "coffee_price" = 350
      "coffee_teaser" = "Packed with goodness to spice up your images"
      "quantity" = 4
    },
    {
      "coffee_description" = ""
      "coffee_id" = 3
      "coffee_image" = "/nomad.png"
      "coffee_name" = "Nomadicano"
      "coffee_price" = 150
      "coffee_teaser" = "Drink one today and you will want to schedule another"
      "quantity" = 3
    },
  ]
}

Next steps

In this tutorial, you implemented a nested read function. This will be useful when you will create a resource using the HashiCups provider in the Implement Create tutorial.

If you were stuck during this tutorial, checkout the implement-complex-read branch to see the changes implemented in this tutorial.

The Terraform Provider Scaffold is a quick-start repository for creating a Terraform provider. Use this GitHub template when you're ready to create a custom provider.
To learn more about the Terraform Plugin SDK, refer to the Terraform Plugin SDK Documentation.

Prerequisites

Create an order via API

Define order data resource

Implement complex read

Add data source to provider

Test the provider

Next steps

Create an `order` via API

Define `order` data resource