Code Generation Design

This section describes the design principles for Terraform provider code generation, the solution, and more details on how existing provider development influenced this design.

Principles

Code generation can reduce the amount of manual effort that is required to maintain a Terraform provider. However, creating a scalable, generic, code generation solution requires following some design principles to integrate well into the existing provider development ecosystem.

Adaptability

The technology ecosystem is constantly evolving and improving. A solution should account for the following:

• API concepts, development practices, and tools will change over time.
• Terraform concepts, development practices, and tools will change over time.
• It is unreasonable for a single technology, either existing or new, to solve all issues over time.

Extensibility

Enabling adaptability requires enabling extensibility. A solution should account for the following:

• Code, tooling, and workflows that are accessible have already proven successful.
  • Refer to the existing Terraform Provider ecosystem - it is unreasonable for a single company or group of developers to understand, design, implement, and maintain all possible integrations.
• Foundational building blocks can help the ecosystem more than a prescribed solution.
• Interfaces can reduce required knowledge and increase consistency.
• Interfaces can be designed to be generic, yet work with specialized use cases.

Usability

Meeting provider developers where they are at, as both the API and Terraform Provider ecosystems already exist. A solution should account for the following:

• Acknowledge new and existing Terraform Provider use cases
  • Lowering the barrier of entry for creating new Terraform Providers
  • Reducing maintenance for existing Terraform Providers
• Acknowledge new and existing Terraform Provider developers
  • Lowering the barrier of entry for new developers
  • Providing more practitioner value for existing developers
• Acknowledge existing technology gaps
  • For example, API specifications have a wealth of useful information that provider developers can utilize.

Solution

This solution aims to provide scalable code generation that meets provider developers where they are at.

High level components

The solution can be summarized as creating an interface. The high level components introduced in this solution can be represented by the following workflow:

On one end, you have various sources of information, and on the other end you have Terraform Provider code. In the middle is our generic interface, which is surrounded by specialized tooling based on that interface.

Here is an expanded example of how components in this design could operate:

Here, a provider spec generator may accept API specification and/or API SDK information and on the other side, a provider code generator is based on a Terraform SDK.

Provider Code Specification

The versioned interface between source(s) and provider code; this interface creates a separation of concerns between any potential source(s) of information and potential consumer(s) of that information. It is intended to be:

• Programming language agnostic
  • Initial implementation is JSON, with JSON Schema validation
• Source agnostic
  • It does not matter what or how many source(s) are involved
• Consumer agnostic
  • It does not matter what or how many consumer(s) are involved
• Necessarily tailored for general API and specific Terraform concepts
• Focused initially on code generation, but extensible for other use-cases

Read more details in the Specification section.

Sources

The specification information can be filled with one or more sources, those could include but are not limited to:

• API Specification or IDL, such as OpenAPI, Protocol Buffers, etc.
• Developer (scripted or manual)
• API SDK information

Consumers

The specification information can be handled by one or more consumers, example use-cases being:

• Terraform SDK code generation, such as creating managed resources or data source schemas based on Terraform Plugin Framework.
• Testing generation
• Documentation generation

Example using OpenAPI

An example of implementing this code generation design was created for OpenAPI generating Terraform Plugin Framework code:

For this example, read more about each tool in it's linked section below:

• The source is an OpenAPI 3.x specification
• The provider spec generator is the OpenAPI Provider Spec Generator which creates a Provider Code Specification.
• The consumer is a Framework Code Generator which produces Terraform Plugin Framework Go code.

Influences

The design principles and solution were influenced by existing provider development practices and challenges.

Provider Development

A Terraform provider acts as a bridge between Terraform and an API, containing a collection of Terraform managed resources and data sources. A developer building a resource or data source for a provider utilizes a combination of a Terraform SDK and an API SDK for implementation.

Terraform SDK

A Terraform SDK contains the abstractions that bridge Terraform's concepts with API operations, including:

• An interface with the Terraform Plugin Protocol
• Terraform component concepts such as provider, managed resource, and data source
• Terraform data concepts such as configuration, plan, and state data.
  • These data concepts include a schema, which defines the structure and types of data; and a type system, which describes how to read and write that data.

Developing a managed resource or data source within a provider generally requires the following:

• Type name: Unique name to reference the resource or data source (e.g., aws_s3_bucket).
• Schema: Data model describing the shared shape of configuration, plan, and state data in Terraform. This data model must generally account for any normalization between Terraform configuration, the API's request objects, and the API's response objects.
• Operation logic: Create, Read, Update, and Delete (CRUD) logic for reading schema-based data, calling API operations, and storing updated schema-based data.
• API client configuration: Any necessary API client initialization or configuration

API SDK

Most API developers offer a native programming language interface to the API, rather than requiring consumers to build generic programming language concepts to match API expectations. This generally includes:

• API client type: Wrapper for HTTP; logic that handles the details required for authentication and endpoints.
• API operations: Functionality for calling API endpoints.
• API data types: Request, response, and other underlying data structures necessary to successfully make and receive API operations.

API Specification

Some APIs are built using an API specification, potentially utilizing an Interface Definition Language (IDL), such as OpenAPI, Protocol Buffers, or Smithy. These API specifications may influence a provider developer's workflow:

API developers may use tooling which can take definition file(s) and can perform automation for their APIs. This generally includes:

• Abstractions: Wrapping of multiple API concepts, such as defining a resource with a consistent data model and API behaviors.
• Behaviors: Endpoint definitions and error handling.
• Data models: Request, response, and other underlying data structures necessary to successfully make and receive API operations.
• SDK generation: Programmatic SDK code generation for multiple programming languages.

Challenges

There are challenges and realities with maintaining a high-quality Terraform provider that HashiCorp has noticed as the provider development ecosystem has grown.

APIs support more than just Terraform Providers

Many API services support an ecosystem of clients where a Terraform Provider is just one desirable integration. This may surface as:

• API components not matching Terraform's component model, such as implicitly existing resources that cannot be created or destroyed.
• API operations not matching Terraform's operational model, such as operations affecting multiple resources.
• API data handling not matching Terraform's data model and consistency rules.
• API error handling not optimized for Terraform, such as an API returning an unauthorized error instead of a not found error.

Developers support more than just Terraform Providers

While some developers can focus mainly on developing Terraform Providers, it is generally uncommon. This means that from the development perspective there may be unfamiliarity or uncertainty with:

• Terraform concepts
• Terraform SDKs
• Go programming language
• Additional provider quality features such as documentation and testing

Terraform Provider Code is often repetitive

Bridging between an API SDK and a Terraform SDK is often repetitive across resources and operations. APIs will typically follow certain conventions, which is great for consistency, but repetitive to interface with the generic Terraform SDKs. Manually developed providers, especially with external contributors, may experience:

• Inconsistent schema definitions, such as missing attributes or validation
• Inconsistent data modeling/mapping, such as invalid conversions between types
• Inconsistent error handling, such as missing resource state removal
• Duplicated code, such as favoring functionality over maintenance burden

Valid schema definition and data handling is critical, but high effort. Provider maintainers may cope with code consistency issues using:

• Specialized code helpers/packages
• Specialized static analysis (linting) tooling
• Specialized code generation