[POC] BTP Terraform Exporter

[CHERIANS]

What area do you want to see improved?

other

Is your feature request related to a problem? Please describe.

This is a PoC for Feature Request: #724
The Terraform Provider for SAP BTP currently supports the import of resources via the standard functionality of Terraform namely import blocks (link).
This allows our customers to import resources, but the process is quite cumbersome and error-prone.

It would be great to have a dedicated functionality to execute that export in an automated and streamlined way like:

btpexport --subaccount `name of the subaccount`

Describe the solution you would like

I would like to have a solution that fulfills the following requirements:

• Automatically identify resources to be imported
• Automatically generate the valid Terraform configuration file:
  • The resource should match its state in the remote platform (verified via terraform plan)
  • Defines the cross resource dependencies
• Option to automatically generate the Terraform state file
• Up-to-date support to the provider (Delay between new resources in the Provider and availability in the tool should be minimal)

There will be gaps as some resources might not support the import or the configuration might not be accessible (like for some service instances). This is acceptable as long as it is logged/tracked by the tool in a machine-readable manner e.g. by a dedicated JSON file that highlights the resources that could not be imported.

Describe alternatives you have considered

The Terraformer project initiated by GCP SRE might be an alternative. It is not clear how well maintained this project is and there is no governance described in the repository that outlines contribution and review flows.

Additional context

As a blueprint for the requirement the Microsoft Azure Export for Terraform CLI can be used, as it covers exactly the same requirements, but is of course restricted to Azure:

• Documentation
• GitHub Repository
• Original project outline and motivation - a bit outdated

[github-actions]

Thanks for the feature request. We evaluate it and update the issue accordingly.

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[ikenna-okeke]

Hi @CHERIANS, For a while now since i started using/looking at terrafom for BTP, I have been looking for ways to import already created resources into the terraform memory similar to what terraformer does(with GCP) instead of having to import each resource individually, as tis would be cumbersome and error prone like you highlighted for a customer account that has been running for instance more than 5 years ago. It would be really nice if the terraformer would be incorporated for the BTP.

[danilobovo]

Hi.

I'm working on a shell script that uses btp cli to export all the information and format as a terraform code and the terraform import command.
It works nice for subaccounts, entitlements, role_collections, subscriptions and environment instances, but It's hard to code the parameters for the btp_subaccount_service_instance.

I can share it in the next days.

[lechnerc77]

Hi @CHERIANS, For a while now since i started using/looking at terrafom for BTP, I have been looking for ways to import already created resources into the terraform memory similar to what terraformer does(with GCP) instead of having to import each resource individually, as tis would be cumbersome and error prone like you highlighted for a customer account that has been running for instance more than 5 years ago. It would be really nice if the terraformer would be incorporated for the BTP.

@ikenna-okeke Thanks for the feedback. If you would like to see this it would also help us if you can vote up the feature request per se (if not already done): #724

From a technical perspective the plan is to develop something that is conceptually closer to the Azure Terraform Exporter than to the terraformer project.

[lechnerc77]

Hi.

I'm working on a shell script that uses btp cli to export all the information and format as a terraform code and the terraform import command. It works nice for subaccounts, entitlements, role_collections, subscriptions and environment instances, but It's hard to code the parameters for the btp_subaccount_service_instance.

I can share it in the next days.

@danilobovo service instances are indeed a challenge and some might not be importable as they do not expose the parameters when fetching the data depending on their settings in the corresponding service offering.

[danilobovo]

@lechnerc77 Thanks for the confirmation.
I read about the "instances retrievable" parameter of btp_subaccount_service_offering data source, I'll use this information to try the import of some instances.

[lechnerc77]

@lechnerc77 Thanks for the confirmation. I read about the "instances retrievable" parameter of btp_subaccount_service_offering data source, I'll use this information to try the import of some instances.

@danilobovo That's the parameter. Be aware that even if the parameter is set to true some configuration data, namely the sensitive one (like admin password for HANA Cloud) won't be returned

[rkapoz-tech]

Hi,
as far as I understood, you are looking for a method to reverse-engineer already existing resources, and retrieve the tf definitions, right? For other platforms (e.g. AZURE) I often used aztfy (recently renamed aztfexport, as reported in the thread). I suggest you trying to use terraform plan -generate-config-out=generated_resources.tf flag. It is currently an experimental feature, but I used it. I tried using also on BTP and it worked ;) Afterwards you have to engineerize the resulting code, but I think it's a very good starting point. You have to specify the resource id you want to retrieve the definition for. It works also with terragrunt. In that case the derived cofnig file will be in the terragrunt-cahe directory. Hope this will help and let me know if it works.

Whai is missing, is a feature (written around btp cli?) that loops on an existing subaccount, collect ALL resources and related ids, and generate an import.tf file to be used as I describe above...

Roberto

[lechnerc77]

Hi, as far as I understood, you are looking for a method to reverse-engineer already existing resources, and retrieve the tf definitions, right? For other platforms (e.g. AZURE) I often used aztfy (recently renamed aztfexport, as reported in the thread). I suggest you trying to use terraform plan -generate-config-out=generated_resources.tf flag. It is currently an experimental feature, but I used it. I tried using also on BTP and it worked ;) Afterwards you have to engineerize the resulting code, but I think it's a very good starting point. You have to specify the resource id you want to retrieve the definition for. It works also with terragrunt. In that case the derived cofnig file will be in the terragrunt-cahe directory. Hope this will help and let me know if it works.

Whai is missing, is a feature (written around btp cli?) that loops on an existing subaccount, collect ALL resources and related ids, and generate an import.tf file to be used as I describe above...

Roberto

Hi @rkapoz-tech , thanks for your input. We also took a look into this Terraform functionality. Although this is already a better starting point than the usual import, it still has some shortcomings that you outline:

• you must define the IDs of the to be imported resources, namely define the import blocks.
• you get generated resources, but it is likely that you will have to do some additional work until a terraform plan would succeed.

As you are using aztfexport you probably agree that the convenience is much higher than what Hashicorp's functionality is providing. We would love to provide the users the same experience on the SAP BTP side.

[TieyanFu]

As discussed with @lechnerc77 , we decided to look into pulumi first to understand how the mapping is done by simply looking into the source code of terraform provider.

Currently raised an issue due to package reference in their guide.

[lechnerc77]

For the PoC we focus in a a first iteration on creating the import blocks for a subaccount and the entitlements of a subaccount.

The result would be a import.tf file that contains valid import blocks and can be called via terraform plan and terraform apply using the experimental flag -generate-config-out (see https://developer.hashicorp.com/terraform/language/import).

We do not need to create a provider.tf automatically in the first iteration.

[danilobovo]

As mentioned before, I'm using a shell script with btp cli to export some BTP resources in tf code.
I provide the code above, maybe it will help some of you.
https://github.com/danilobovo/btptfexport

[lechnerc77]

@danilobovo thanks for bringing your repository to our attention. However, due to the license you are using (GPL-3.0) and our internal Open Source guidelines on licenses, we are not allowed to use this in this project

[danilobovo]

I removed the license, my intention is to share the code to anyone who might find it useful.

[lechnerc77]

@danilobovo thanks, however as we have these corporate restrictions we must follow some governance:

• You repo now has no license at all which is also a no go
• The license change via PR is also not reliable at least might raise questions if not linked to a release ( e.g. "from release 1.1.0 the project is distributed under Apache 2.0")

Sorry for being this picky as any help is appreciated, but we also must make sure to be in sync with our OPSO governance.

[vipinvkmenon]

Initial assessment and feasibility analysis completed. Closing this issue and will start development through #724