[Bug]: aws_db_instance - configured value for vpc_security_group_ids is incorrectly changed during TF plan

[ktham]

Terraform Core Version

1.7.5

AWS Provider Version

5.84.0

Affected Resource(s)

• aws_db_instance
• aws_rds_cluster

Expected Behavior

Terraform is expected to be able to render the full detail of a partially-unknown set (See: hashicorp/terraform#36221) if the provider is indeed providing one, and it seems like the AWS provider isn't currently doing that here for aws_db_instance and aws_rds_cluster.

We should expect to see Terraform plan output that looks like this when a new security group is added to vpc_security_group_ids

      ~ vpc_security_group_ids                = [
          + "sg-46c08c0f",
          + (known after apply),
        ]

Actual Behavior

However, the aws_db_instance resource is doing something unexpected, we instead see this

      ~ vpc_security_group_ids                = [
          - "sg-46c08c0f",
        ] -> (known after apply)

👉 👉 sg-46c08c0f should be retained, but in the plan renderer, what we're seeing is that the vpc_security_group_ids list as a whole is being replaced with an unknown value.

This is very concerning behavior because it gives the impression that the AWS provider might potentially be revoking security group IDs that we expect to be retained.

Relevant Error/Panic Output Snippet

Terraform Configuration Files

resource "aws_security_group" "example" {
  # ...
}

resource "aws_db_instance" "example" {
  vpc_security_group_ids = [
    aws_security_group.example.id,
  ]
  # ...
}

Terraform configuration (step 2)

resource "aws_security_group" "example" {
  # ...
}


# Add this:
resource "aws_security_group" "additional" {
  # ...
}

resource "aws_db_instance" "example" {
  vpc_security_group_ids = [
    aws_security_group.example.id,

    # Add this:
    aws_security_group.additional.id,
  ]
  # ...
}

Steps to Reproduce

Run terraform apply for step 1 Terraform configuration, then run terraform apply for step 2 configuration from above.

A possible workaround is a targeted apply of the resources that contain the attributes that are unknown (such as the security group), in order to avoid the scenario of adding unknown elements to an existing set attribute, but that is not a feasible/scalable solution to have to do a targeted TF apply.

References

#41047

I'm so sorry for re-opening a new ticket, but it looks additional comments on closed tickets are difficult for maintainers to see and I believe the original ticket was pre-maturely closed with an incorrect understanding of expected Terraform Core behavior.

This is not a problem with Terraform Core, but a problem with the provider itself, please see:

• Terraform plans should show set members that will be retained post-apply terraform#36221
• OpenTofu plans should show set members that will be retained post-apply opentofu/opentofu#2296

👉 It looks like the AWS provider is likely producing an invalid plan here and part of the fix is to migrate this resource to the TF plugin framework according to @jbardin. If the provider is correctly returning the partial set back to Terraform core, the TF plan rendered can then render the plan in the expected way described above.

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[ktham]

I'm so sorry for re-opening a new ticket for #41047, but it looks additional comments on closed tickets are difficult for maintainers to see and I believe the original ticket was pre-maturely closed with an incorrect understanding of expected Terraform Core behavior

[justinretzolk]

Hey @ktham 👋 Thanks for following up here and opening a new issue so we could see it -- that's extremely helpful! Also, apologies that I'd misunderstood a bit here. Having spent some time re-reading through the linked threads, I see now that I'd moved a bit too quickly in closing the previous thread. For the sake of trying to keep the conversation in one place, I'm going to close this thread and re-open the last one.

[github-actions]

Warning

This issue has been closed, meaning that any additional comments are hard for our team to see. Please assume that the maintainers will not see them.

Ongoing conversations amongst community members are welcome, however, the issue will be locked after 30 days. Moving conversations to another venue, such as the AWS Provider forum, is recommended. If you have additional concerns, please open a new issue, referencing this one where needed.