Fix "EKS-Private" SubnetType tagging workaround

[jaskaransarkaria]

Background

Currently, we have a workaround where we retroactively overwrite the tags for the new EKS nodes-only subnets at the terraform vpc layer. However, the vpc tf module tries to rewrite the tag every time. We can see this with the failing check-divergence-networking pipeline.

https://concourse.cloud-platform.service.justice.gov.uk/teams/main/pipelines/divergence/jobs/divergence-networking/builds/1903

  # module.vpc.aws_subnet.private[5] will be updated in-place

  ~ resource "aws_subnet" "private" {
        id                                             = "subnet-0a55c8052635d4af6"
      ~ tags                                           = {
          ~ "SubnetType"                                                         = "EKS-Private" -> "Private"
            # (8 unchanged elements hidden)
        }

      ~ tags_all                                       = {
          ~ "SubnetType"                                                         = "EKS-Private" -> "Private"
            # (13 unchanged elements hidden)
        }
        # (15 unchanged attributes hidden)
    }

Plan: 3 to add, 3 to change, 3 to destroy.

This creates another problem where it means that when the vpc module rewrites the tags back to "Private" during the time before the next terraform resource corrects it to "EKS-Private" again. If rds, elasticache or opensearch resources are being deployed there's at that very moment they could be deployed to the wrong subnets. This would be bad as it would contribute to subnet fragmentation again.

Investigate moving the current module into a database or elasticache subnet, consider that the current old subnets contain different resources databases, elasticache and opensearch and importantly also contains the route table for the transit gateway. Make sure none of these resources are affected.

Figure out what the difference is between a database subnet and an elasticache subnet, if possible terraform move the current "Private" subnets into the new subnet value and then tag them from there. Then you can remove the subnet_tags file and tag the "EKS-Private" subnets from the vpc module.

https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/latest

Definition of done

• readme has been updated
• user docs have been updated
• another team member has reviewed
• smoke tests are green
• prepare demo for the team

Reference

How to write good user stories

[kyphutruong]

Working branch