Terraform
Import existing resources in bulk
You can configure queries that instruct Terraform to search your existing infrastructure for unmanaged resources. Terraform can also generate configuration for importing the resources it finds so that you can import them to your Terraform workspace in bulk. For information about importing single resources or small batches of resources, refer to Import single resources for instructions.
Introduction
For organizations with large sets of infrastructure resources, manually identifying and importing them is tedious and labor intensive, even when using third-party tools or custom scripts. To alleviate this burden, you can write HCL-based queries and run them with the Terraform CLI to retrieve unmanaged resources so that you can import them in bulk.
Complete the following steps to find and import resources in bulk:
- Search for resources: Create
list
blocks to search for existing resources. - Generate configuration: Use the Terraform command to generate
resource
andimport
blocks for importing the resources. The configuration also includes the resource identity. - Import resources: Copy the generated configuration into your main.tf file and run a
terraform apply
command to import the resources. - After importing resources, you can remove the generated
import
block or keep it as an historical record.
Requirements
Terraform v1.12 or newer is required to search for resources according to their resource identities. HCP Terraform uses resource identities to determine when resources are managed by another workspace. Refer to Import existing resources to state for more information.
Supported providers
Verify that the resource type you want to search for is supported. For the initial beta release, you can use list
blocks to query for the following resource types. This list may change. Refer to your provider documentation for the most up-to-date information.
AWS provider
AWS CC provider
All resources in this provider support list
blocks. Because this provider is fully generated and doesn't accept provider-specific configurations, omit the config
block when defining queries for AWS CC resources.
Define a query
To search for resources, you must create a standard .tf
configuration and a .tfquery.hcl
configuration.
Standard configurations
If your .tf
configuration does not already have a required_providers
block, add it so that Terraform can install all provider plugins required to create and manage resources specified in the configuration. Refer to the required_providers
reference for details on how to configure this block.
If you have an HCP Terraform account, you can compare the resources Terraform discovers to resources you are already managing with Terraform in the HCP Terraform UI. To use this functionality, you must configure the cloud
block in your .tf
configuration to connect to HCP Terraform. Refer to the cloud
block reference for details on how to configure this block.
Query configurations
Create a file with a .tfquery.hcl
extension and add list
blocks to define queries that read existing infrastructure and return lists of unmanaged resources.
Each list
block requires an inline argument that specifies the type of resource you are querying. The type is specific to your provider. The block also requires an inline argument that declares a local name for the list of results returned by the query. Refer to the list
block reference for configuration details.
The following example defines a list
block named prod
that queries aws_instances
:
list "aws_instance" "prod" {
# . . .
}
Add the following arguments to your list
block:
provider
: This argument is required. It specifies which provider configuration Terraform should use to perform the query. Thelist
block retrieves the provider configuration from theterraform
block in yourmain.tf
configuration, but you can declare aprovider
block in your query file with an alternate configuration and reference it in theprovider
argument in yourlist
block.config
: Add theconfig
block to yourlist
block and define provider-specific arguments to build your query. Refer to your provider documentation for supported arguments and values.include_resource
: By default, Terraform retrieves only resource identities, but you can set theinclude_resource
argument totrue
so that Terraform can also retrieve all available resource attributes. To reference resource attributes retrieved by the list block, you must enable theinclude_resource
argument. Setting this argument totrue
may affect performance.limit
: By default, Terraform retrieves up to 100 resources per list block, but you can use thelimit
argument to specify a higher or lower number of results.
The list
block also supports several Terraform meta-arguments, which are arguments built into Terraform configuration language that configure how Terraform creates and manages infrastructure objects. For example, you can use the count
meta-argument to create multiple instances of the resource list returned by the query. Refer to Meta-arguments for more information.
In the following example, Terraform applies provder arguments to query for aws_instance
resources in the us-east-2
region. The query also filters results prefixed with prod-
and stage-
and that are in a running
state. The limit
argument, which is not a provider-specific argument, instructs Terraform to return the first 50 results:
list "aws_instance" "prod" {
provider = aws
limit = 50
config {
region = "us-east-2"
filter {
name = "tag:Name"
values = ["prod-*", "staging-*"]
}
filter {
name = "instance-state-name"
values = ["running"]
}
}
}
Parameterize your query configuration
You can parameterize your .tfquery.hcl
file so that you can reuse it with a different set of inputs. Refer to Set configuration parameters for instructions on how to parameterize configurations.
Add the following blocks to your query file to parameterize the configuration:
variable
: Add thevariable
block to your query file to define variables that people who run the query configuration can provide at runtime. Refer to thevariable
block reference for configuration details.locals
: Add thelocals
block to your query file to define temporary variables scoped to the query configuration. Refer to thelocals
block reference for configuration details.
Specify a custom provider configuration
The query file checks the main.tf
configuration for provider configurations, but you can also declare one or more provider
blocks in the query configuration file to define alternate provider configurations. To use one of the alternate provider configurations, add the provider
argument to your list
block and reference the name of the provider configuration. Refer to the provider
block reference for information about how to configure provider
blocks.
Query the infrastructure
After configuring the query configuration, you can search for resources using the Terraform CLI or using HCP Terraform if applicable.
HCP Terraform
To run queries from the HCP Terraform UI, you must configure a connnection to HCP Terraform and copy configuration files from your working directory to your workspace in HCP. Refer to Connect to HCP Terraform for instructions.
If HCP Terraform is connected to your version control system (VCS), you can also check your configurations into your VCS before running the query in HCP Terraform. Refer to Connect to VCS Providers for more information.
After copying the configuration to your HCP Terraform workspace, you can either run the query locally using the Terraform CLI or run the query in the HCP Terraform UI. Refer to Import existing resources to state in the HCP Terraform documentation for instructions on how to run queries and import resources from the UI.
Refer to Import existing resources to state in bulk in the HCP Terraform documentation for details.
Terraform CLI
Run the terraform query
command to retrieve the resource types specified in your list
blocks. The command prints the results to your console. By default, each result includes the following information:
- Reference to the
list
block that queried for resources formatted aslist.<type>.<label>
. - Identity of a discovered resource.
The provider may also include other information, such as a description of the resource. Refer to your provider documentation for details.
The number of results depends on the provider, as well as any query constraints you configured in the query configuration file.
To generate machine-readable results, you can include the -json
flag:
$ terraform query -json
Generate configuration for importing results
To generate configuration, run the terraform query
command and add the -generate-config-out
flag. The flag expects a path to a generated file named generated.tf
. The generated.tf
file contains the resource
and import
blocks, including resource identities, necessary for importing results.
The following example generates configuration to the working directory:
$ terraform query -generate-config-out=generated.tf
Even when connected to HCP Terraform or Terraform Enterprise, Terraform creates the generated.tf
file and stores it on the local workstation when running the terraform query
command with the -generate-config-out
flag.
After generating the results file, you must remove the file before rerunning the command to generate a new results file. Rerunning the command when a generated.tf
already exists at the specified path results in an error.
Import resources
Copy the import
and resource
blocks from the generated.tf
file to your main.tf
configuration and run the terraform apply
command to import the resources.
Next steps
You can discard the generated.tf
file after importing your resources. You can either remove import
blocks after you've imported them or leave them in your configuration as a record of the resource's origin for future module maintainers. For more information on maintaining configurations over time, refer to Refactoring.