ETL Mapping Customizations

You can update and customize the default ETL mapping rules offered by the Service Graph Connector for Terraform.

To ensure that your custom rules remain intact during future updates, you can clone the existing ETL record and maintain it separately from the default one.

This documentation guides you through the process of mapping a resource using an example of an AWS virtual private network (VPC). Although this resource is already covered by the application, the principles discussed apply to any new potential resource mapping.

Any customizations should be done from the application's scope: Service Graph Connector for Terraform.

Clone the ETL map

Navigate to the IntegrationHub ETL in the top menu. Check the SG-Terraform record and click Duplicate. Refer to the ServiceNow documentation to create a duplicate of an existing ETL transform map.

Build a resource in HCP Terraform

Create a new workspace in your HCP Terraform organization and create a Terraform resource that you would like to map. It helps to have a Terraform state record of the resource to ensure accurate mapping.

Configure a webhook and initiate a Terraform run.

Download Terraform State

Once the run is successfully completed, open your ServiceNow instance, click on All and navigate to Scheduled Imports.

Open the SG-Terraform Scheduled Process State record, search for the import set corresponding to the latest webhook request. Click the Import Set field to open the import set. Wait for the import set to be successfully processed.

Since there are no existing ETL rules configured for the new resource, it is ignored during the ETL process.

Open the Outbound Http Requests tab to list the requests sent from your ServiceNow instance to HCP Terraform and get the latest state of the workspace.

Open the record that starts with "http://archivist.terraform.io" by clicking on the timestamp. Copy the content of the URL field and open it you your browser to download the Terraform state file.

Locate the resource in the state object. This JSON record will serve as a source for the future mapping.

Identify the CI target

Pick a suitable Configuration Item (CI) target for your resource. For example, the AWS virtual private network (VPC) resource is mapped to Cloud Network (cmdb_ci_network) by the Service Graph Connector for Terraform. Refer to the ServiceNow CMDB documentation for more details on available CI tables.

Consult the CI Class Manager

After selecting an appropriate CI target, it is important to consult the CI Class Manager for guidance on dependent relationships. Many CMDB resources rely on other CI tables. If a related class is not properly mapped, the ETL job will generate errors or warnings and fail to import your resource into the CMDB.

In the top navigation, click on All, search for CI Class Manager, and click on Open Hierarchy. Search for your target CI Class and check Dependent Relationships tab to learn more about dependent mappings required by the resource.

For example, according to the CI Class Manager, Cloud Network should be hosted on Logical Datacenter and Cloud Service Account.

Set the mapping rules

Open the IntegrationHub ETL from the top navigation menu and select your cloned ETL map record prepared for customization. Refer to the ServiceNow documentation for instructions to create an ETL transform map.

Click on the first Specify Basic Details section of the ETL Transform Map Assistant. Select the import set number containing your resource from the Sample Import Set dropdown and click Mark as Complete.

Open the Preview and Prepare Data section and review the imported rows. Click Mark as Complete.

The third section provides the interface for mapping resource attributes. Click on Select CMDB Classes to Map Source Data. Click on Add Conditional Class button at the top. Set the rules that will identify your resource in the import set. Use the type field value from the Terraform state object to identify your resource (on the ServiceNow side, field name are prefaced with u_). Set the target CMDB CI Class name and click Save.

To modify the mapping for your new Conditional Class record, select Edit Mapping. On the right side of the interface, drag the relevant data pills and drop them into the corresponding CMDB fields on the left side. Refer to the Terraform state record to verify the presence of attributes. For uniqueness, the Source Native Key value is typically mapped to the arn field when dealing with AWS resources. All resources mapped in the Service Graph Connector for Terraform will have the Operational status and Name fields populated.

Once the mapping is completed, click on the left arrow at the top to return to the list of Conditional Classes. Map two more conditional classes in the same manner, according to the rules set in the CI Class Manager: Logical Datacenter (AWS Datacenter in case of AWS VPC) and Cloud Service Account. Since the AWS cloud provider is already covered by the application, these classes are already present. Click Edit Class to include your newly mapped resource into the listed conditional rules. Add another OR condition to each of them and click Save.

Click Mark as Complete to finalize the Select CMDB Classes to Map Source Data section.

Set the required relationships

Click Add Relationships to continue to the next section. Click the Add Conditional Relationship button at the top of the page. The following configuration tells the ETL that when a record with aws_vpc type is found in the import set, it should be hosted on AWS Datacenter 1. Click Save.

A similar dependent relationship needs to be established from AWS Datacenter to Cloud Service Account. Since the AWS cloud provider is already covered by the application, the relationship record is present in the application. Click Edit Relationship and add another OR condition containing your new resource to the list. Click Save.

Click Mark as Complete to finalize the Add Relationships section.

Run a test

There are two ways to test the new resource mapping. You can utilize the Test and Rollback Integration Results interface of the ETL Transform Map Assistant. Alternatively, you can initiate a new run in your HCP Terraform workspace that includes the deployment of the resource.