[Bug]: DBInstanceAutomatedBackupsReplication cannot change the value of the argument "kms_key_id" after successful creation

[lajchon]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

DBInstanceAutomatedBackupsReplication.rds.aws.upbound.io/v1beta1

Resource MRs required to reproduce the bug

No response

Steps to Reproduce

Create KMS Key and RDS DBInstanceAutomatedBackupsReplication with a kmsKeyIdSelector.

apiVersion: kms.aws.upbound.io/v1beta1
kind: Key
metadata:
  labels:
    usage: replication
spec:
  forProvider:
    deletionWindowInDays: 0
    enableKeyRotation: False
    keyUsage: ENCRYPT_DECRYPT
    multiRegion: True

apiVersion: rds.aws.upbound.io/v1beta1
kind: DBInstanceAutomatedBackupsReplication
spec:
  forProvider:
    kmsKeyIdSelector:
      matchControllerRef: True
      matchLabels:
        usage: replication
    sourceDbInstanceArnSelector:
      matchControllerRef: True
    retentionPeriod: 7

What happened?

After DBInstanceAutomatedBackupsReplication returns a Ready 'True' state, it immediately enters a Synced 'False' state with the following status:

status:
  atProvider:
    id: >-
      arn:aws:rds:us-west-2:REMOVED:auto-backup:ab-oo3bee2fkoilq5y4xjnsil6ce3c2iakwc3bsr5y
    kmsKeyId: >-
      arn:aws:kms:us-west-2:REMOVED:key/mrk-5780b5bf1213453dbc7c2b83c1d1ee55
    retentionPeriod: 7
    sourceDbInstanceArn: arn:aws:rds:us-east-1:REMOVED:db:test-instance
  conditions:
    - lastTransitionTime: '2024-08-01T15:31:54Z'
      reason: Available
      status: 'True'
      type: Ready
    - lastTransitionTime: '2024-08-01T15:31:53Z'
      message: >-
        update failed: async update failed: refuse to update the external
        resource because the following update requires replacing it: cannot
        change the value of the argument "kms_key_id" from
        "arn:aws:kms:us-west-2:REMOVED:key/mrk-5780b5bf1213453dbc7c2b83c1d1ee55"
        to "mrk-5780b5bf1213453dbc7c2b83c1d1ee55"
      reason: ReconcileError
      status: 'False'
      type: Synced
    - lastTransitionTime: '2024-08-01T15:31:53Z'
      message: >-
        async update failed: refuse to update the external resource because the
        following update requires replacing it: cannot change the value of the
        argument "kms_key_id" from
        "arn:aws:kms:us-west-2:REMOVED:key/mrk-5780b5bf1213453dbc7c2b83c1d1ee55"
        to "mrk-5780b5bf1213453dbc7c2b83c1d1ee55"
      reason: AsyncUpdateFailure
      status: 'False'
      type: LastAsyncOperation
spec:
  deletionPolicy: Delete
  forProvider:
    kmsKeyId: mrk-5780b5bf1213453dbc7c2b83c1d1ee55
    kmsKeyIdRef:
      name: test-instance-vhnhg-wz9rg
    kmsKeyIdSelector:
      matchControllerRef: true
      matchLabels:
        usage: replication
    region: us-west-2
    retentionPeriod: 7
    sourceDbInstanceArn: arn:aws:rds:us-east-1:REMOVED:db:test-instance
    sourceDbInstanceArnRef:
      name: test-instance-vhnhg-c5pgr
    sourceDbInstanceArnSelector:
      matchControllerRef: true
  initProvider: {}
  managementPolicies:
    - '*'
  providerConfigRef:
    name: provider-aws-REMOVED

Relevant Error Output Snippet

No response

Crossplane Version

1.16.0

Provider Version

1.10.0

Kubernetes Version

No response

Kubernetes Distribution

EKS

Additional Info

No response

[blakeromano]

I don't think this is a bug from what I understand...

Crossplane Providers will not update a resource if it requires a re-creation (or at least in my experience upjet ones do not for sure). I think you'd have to create a new resource, then delete this resource (or however TF deals with replacements of this resource assuming you feel that makes sense).

[lajchon]

This is happening immediately after the DBInstanceAutomatedBackupsReplication resource is reconciled, and is not the result of trying to update to use a new KMS Key.

The resource is successfully created with kmsKeyId: arn:aws:kms:us-west-2:REMOVED:key/mrk-5780b5bf1213453dbc7c2b83c1d1ee55, as shown in the status.atProvider. But as soon as Ready becomes True, Synced becomes False because it's attempting to change the kmsKeyId. This seems like it's an issue with the kmsKeyIdSelector where it originally used the resource's Arn, then attempts to update to use the Id.

[lajchon]

Expanding my search of other issues, this appears to be part of a larger problem when resources reference kms_key_id.

The following issue has a possible workaround, using a ToCompositeFieldPath patch to set the spec.forProvider.kmsKeyId field, rather than using kmsKeyIdSelector. I'll test the workaround and report back the results.

#892 (comment)

[blakeromano]

Ahh I see. I think the issue is on the Terraform side from what I can tell. https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/service/rds/instance_automated_backups_replication.go#L228 specifically it seems that they set status based on the KMSKeyID returned from AWS SDK https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/[email protected]/types#DBInstanceAutomatedBackup which the AWS SDK says can return "id, arn or alias" so I think on the TF side they need to make it consistent on the ARN or maybe we in crossplane need to have a custom diff maybe as two solutions to the underlying problem.

[lajchon]

Tested the following workaround:

• Push KMS Key Arn into environment field:

              - type: ToEnvironmentFieldPath
                fromFieldPath: status.atProvider.arn
                toFieldPath: replicationKMSKeyArn
                policy:
                  fromFieldPath: Required

• Set DBInstanceAutomatedBackupsReplication kmsKeyId from environment field:

              - type: FromEnvironmentFieldPath
                fromFieldPath: replicationKMSKeyArn
                toFieldPath: spec.forProvider.kmsKeyId
                policy:
                  fromFieldPath: Required

The DBInstanceAutomatedBackupsReplication resource created successfully and did not enter a Synced 'False' state.

[github-actions]

This provider repo does not have enough maintainers to address every issue. Since there has been no activity in the last 90 days it is now marked as stale. It will be closed in 14 days if no further activity occurs. Leaving a comment starting with /fresh will mark this issue as not stale.