Support terraform import via the cdk cli

[fmmoret]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Support importing existing resources into a stack's state file.

References

https://www.terraform.io/docs/cli/import/index.html

[fmmoret]

This is what I'm doing in my own project for now:

Created this script at bin/run_terraform_with_stack.sh in my local project & ran chmod +x on for execution permissions:

#!/bin/bash
if [[ $# -lt 2 ]] ; then
    echo "Must provide the commands of 'terraform [... commands ]' and a stack. Only got '$@'"
    exit 1
fi
TF_ARGS=${@:1:$(($#-1))}
STACK=${@: -1}
TF_COMMAND="terraform $TF_ARGS"
read -p "Will run '$TF_COMMAND' on the '$STACK' stack. Confirm (y/n) " -n 1 -r
echo ""
if [[ $REPLY =~ ^[Yy]$ ]]
then
  cd "cdktf.out/stacks/$STACK" && bash -c "$TF_COMMAND"
fi

And then made some scripts in my package.json:

This one is the base for the rest.

"tf": "./bin/run_terraform_with_stack.sh",

And here's some example shorthand utility scripts (useful if you have long commands that you run often:

"show-plan": "yarn tf show plan",
"untaint": "yarn tf untaint"

Then you can run these like yarn tf show plan my_stack_name or yarn show-plan my_stack_name.

Extending this all to the import example in the terraform documentation: https://www.terraform.io/docs/cli/import/usage.html

I haven't had to import anything yet, but I think you could do: yarn tf import aws_instance.example_name i-abcd1234 my_stack_name
This command locates the AWS instance with ID i-abcd1234. Then it attaches the existing settings of the instance, as described by the EC2 API, to the name aws_instance.example_name

No warranty / guarantees, but hope this helps someone 🎉

[remyleone]

That would be very useful to have a way to generate HCL configuration from existing configuration such as https://github.com/cycloidio/terracognita seems to provide for different vendors.

[DanielMSchmidt]

With the cdktf convert command you can take the output of terracognita and use it to create an equivalent CDK program

[rirze]

Having recently come into a new (AWS) environment where I have to import 100s of resources, I decided to build a solution that works for me using the CDK for Terraform. This is pretty much inspired by how Pulumi allows for resource imports from code, so I'm following that model pretty closely.

I have no idea if this approach will be amendable to being included into terraform-cdk core, but I'm listing it here in case it is, and/or if anyone else wants to adopt this workflow.

---

The idea I had is simple: for many of these resources, I know the resource_id that the Terraform resource corresponds to. The issue is tying the corresponding Terraform logical_id to the live resource_id. I first tried creating these links within my CDK code (trying to use Aspects), only to find out finding logical ids of resources from within the TerraformStack can be inconsistently complicated. So instead, I added an extra parameter to all of my higher-level resource abstraction classes (ex. my custom SecurityGroup class that creates 1:1 Terraform aws_security_group resource(s)). That extra parameter is import_, and setting it to the resource id needed for its import would create this snippet for the relevant resource in the generated cdk.tf.json:

    "aws_security_group": {
      "some_logical_id": {
        "//": {
          "metadata": {
            "path": "some_path",
            "uniqueId": "some_logical_id"
          },
         "importId": "sg-0123345123345"
        },
        "name": "my-sg"
      },

Once #1543 is merged, it should enable native support for users to add this importId into the comments key of this json format (["//"]) using add_override or something similar.

The code that inserts this can look something like this:

        self.my_sg = MySecurityGroup(
            self, 'some_logical_id',
            name="my-sg",
            import_="sg-0123345123345",
       ) 

Once the cdk.tf.json is generated, I then run a (Python) script that runs through the cdk.tf.json and:

1. finds any resource with a importId key
2. corresponds the given importId value with the logical_id of the resource it was found in.
3. create strings with the right format for importing resources
   a. terraform import <logical_id> <resource_id aka importId value>
4. output those strings into a .sh file so I remember to run it as a bash script.

Once that script generates a import.sh file containing the correct terraform import ... statements, I run it.

After I ensure the state is in a valid state, I remove the import_="some_resource_id" line from the code, since it is no longer needed. While failing to do this doesn't necessarily cause any future imports to fail, it makes any future debugging easier to analyze.

Hopefully that workflow makes sense, if there are any questions, feel free to ask.

---

Notes:

• Due to the unholy mess that is the Terraform AWS Provider and how it deals with aws_security_group_rules, my script that generates the import statements is a lot more developed than the approach above requires. In short, it needs to go out and make API calls and then stitch together an import statement using a combination of attached security group id and rule parameters (see this link for more details). I frankly don't know how this could be implemented into terraform-cdk core, since this is a provider-specific oddity where simply specifying the existing resource id wouldn't work.

[jsteinich]

@rirze Thanks for sharing your workflow.

I could see having some like importFrom available on all resources. This would basically just do what you are already doing with an override and a basic script of Terraform commands. That feels like a relatively low scope way of providing import support.
Could perhaps add support for some custom setting/generation for resources with strange semantics. Likely wouldn't want to go too far down this route though.

Might be interesting to have some sort of createResource method available on data sources which effectively does an import, but with the necessary id queried through friendlier properties. This would be harder to automatically generate and doesn't directly fit within current Terraform operations, but could be pretty slick.

[rirze]

@jsteinich Glad to see some feedback on this-- I think providing a basic level of "use this resource_id when using terraform import" would go a long way for this feature. I think due to the high level of you guys are trying to support, I think it'll be a while before a more native approach would manifest. Even for the peculiar case of aws_security_group_rule, I could support this workflow by making API calls during the synthesize process (which I'm not too keen on, but just saying it is possible).

If you want to move forward with this idea, do you see a particular way of exposing this to the user? I've seen @ansgarm detail an approach using an Imports.of(resource...).addImportId(resource_id) pattern (#1543 (comment)). That seems like the easiest way to implement this, although it might be a little verbose for someone not familiar with that style.

[jsteinich]

If you want to move forward with this idea, do you see a particular way of exposing this to the user? I've seen @ansgarm detail an approach using an Imports.of(resource...).addImportId(resource_id) pattern (#1543 (comment)). That seems like the easiest way to implement this, although it might be a little verbose for someone not familiar with that style.

I see that approach as more of just a friendlier way for a user to add additional functionality to a resource (in this case adding some import metadata). Building directly into the api we could add importFrom directly onto TerraformResource. This could either get synthesized out to a "comment" in the json file that is later acted upon, or be used by a new import command variant to generate and execute the native Terraform commands.

[ansgarm]

I see that approach as more of just a friendlier way for a user to add additional functionality to a resource (in this case adding some import metadata)

Yes, that was exactly I was after there. It was my proposal as an alternative to an "add any metadata" method. However, in case of the import use-case we're discussing here, I'd go with an API as proposed by @jsteinich.
And you're both right, we should dare to add such am API rather earlier than later. I was probably a bit too defensive in getting the workflow exactly right first 😅

Does anyone of you know whether there is any documentation on the id that one has to supply to terraform import? I remember that it sometimes took the arn but not always (for the AWS provider at least). If we had that information somewhere we could use it in docstrings and maybe make that part of the generated provider bindings. But I fear we don't (which is also fine, I think I just always wished these docs existed back when I used terraform import 😄).

[rirze]

Does anyone of you know whether there is any documentation on the id that one has to supply to terraform import? I remember that it sometimes took the arn but not always (for the AWS provider at least). If we had that information somewhere we could use it in docstrings and maybe make that part of the generated provider bindings. But I fear we don't (which is also fine, I think I just always wished these docs existed back when I used terraform import 😄).

It's my understanding that this is left up to the provider, so this heavily depends (like you noted) on the provider's strategy and particular resource import implementation. Take a look at aws_security_group_rule. Here, it's not as simple as giving it a security group rule ARN. I don't know if this Import category is available when terraform-cdk builds providers, but I would guess that's the easiest way to generate documentation for this feature.

[jsteinich]

I don't know if this Import category is available when terraform-cdk builds providers, but I would guess that's the easiest way to generate documentation for this feature.

It's not something that's directly available, but the idea of looking at the docs during generation has been brought up before. Might also be feasible to get it included in the upstream Terraform schema.

[nick0lay]

The following script was able to import the existing log group to the CDKTF state.

cdktf get
cdktf synth
cd cdktf.out/stacks/<stack name>
terraform init
terraform import aws_cloudwatch_log_group.lambda-logs "/aws/lambda/lambda-logs"
cd ../../..
cdktf deploy <stack name>

[RichiCoder1]

The AWS CDK has a surprisingly decent flow that could be emulated here: https://github.com/aws/aws-cdk/tree/main/packages/aws-cdk#cdk-import

TL;DR:

• Add resources to stack
• Run cdktf import
• cdktf synthesizes and then prompts for any resources that would be added (present in stack but not remote state); This is probably the Hard Part ™️. This could also be a place where the CLI surfaces doc links for imports
• cdktf does the necessary state imports for resources

[rirze]

• Add resources to stack
• Run cdktf import
• cdktf synthesizes and then prompts for any resources that would be added (present in stack but not remote state); This is probably the Hard Part ™️. This could also be a place where the CLI surfaces doc links for imports
• cdktf does the necessary state imports for resources

It's a very manual process imo. From what I remember, the CLI prompts for the ARN (AWS specific in this case) one by one. Might be a good initial step but I don't think this should be the long term solution.

[keidarcy]

Finally, I found this manual workaround to import existing resources.

1. assume resources exist, write the approximate cdktf code.
2. run cdkft synth in project root.
3. cd cdktf.out/stacks/<stack>
4. check uniqueId in cdk.tf.json, it's located at

  "resource": {
    "<Resource_Name>": {
      "<Unique_ID>": {
        "//": {
          "metadata": {
            "path": "<Path>",
            "uniqueId": "<Unique_ID>"
          }
        },

5. terraform init and terraform import <Resource_Name>.<Unique_ID> <ID>
6. cd ../../../, cdktf diff <stack>

You should see change instead of add.

[DanielMSchmidt]

We also recently published some docs around refactoring. If you move constructs between stacks you also need to import, maybe it is helpful to anyone reading this: https://developer.hashicorp.com/terraform/cdktf/examples-and-guides/refactoring

[PookMook]

Finally, I found this manual workaround to import existing resources.

1. assume resources exist, write the approximate cdktf code.
2. run cdkft synth in project root.
3. cd cdktf.out/stacks/<stack>
4. check uniqueId in cdk.tf.json, it's located at

  "resource": {
    "<Resource_Name>": {
      "<Unique_ID>": {
        "//": {
          "metadata": {
            "path": "<Path>",
            "uniqueId": "<Unique_ID>"
          }
        },

5. terraform init and terraform import <Resource_Name>.<Unique_ID> <ID>
6. cd ../../../, cdktf diff <stack>

You should see change instead of add.

Thanks for that message, works like a charm. Make sure you use the "DB instance ID" for on the DB you want to import, not the ARN. I guess it works the same for other types of ressources. Makes life easier and prevent the string too long errors

[Danielku15]

The workaround from @keidarcy works indeed nicely. We are using the CDK with C# and I wrote some code which automates this for us until there is an official feature.

1. We run the synth command
2. We then read all resources listed in the resulting cdk.tf.json
3. We read also the existing tfstate and remove all resources which are already tracked in the state.
4. We run a terraform init
5. We then try to run a terraform import for all resources which are not known in the state.

Of course there can be errors in cases where you mix actually new items and items to import in one run but we can deal with that. The biggest challenge was building the import IDs which depend on the provider and resource you are importing and you need to align this with the docs.

As we are doing a larger migration it was worth automating this for us.

[mackignacio]

Hi everyone. I created a CLI for CDKTF import using the solution provided from this comment of @keidarcy and @Danielku15 . This is still on development but this is better than doing this manually.

[DanielMSchmidt]

Hey @mackignacio, just FYI, we merged #2972 now that includes import functionality. We are soon releasing a new cdktf version, so we hope you'll find it useful (or that your import needs are already solved by your CLI :) )

[mackignacio]

Nice Thank you. For this moment I will just used mine until you release the new version.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days. This helps our maintainers find and focus on the active issues. If you've found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.