Bug: Resource created with reconcile policy skip and updated to manage keeps AzureResourceNotFound status until long running operation finishes

[nojnhuh]

Version of Azure Service Operator

v2.5.0

Describe the bug
Creating a resource with reconcile-policy: skip and then updating that to manage while the resource's Ready condition is false because of AzureResourceNotFound does not transition the reason to Reconciling when the ensuing PUT begins. Creating the same resource with reconcile-policy: manage from the start shows a Reconciling reason for the duration of the operation like I would expect.

To Reproduce
Steps to reproduce the behavior:

1. Apply the following resources:

apiVersion: resources.azure.com/v1api20200601
kind: ResourceGroup
metadata:
  name: jon-test
spec:
  location: eastus
---
apiVersion: containerservice.azure.com/v1api20231001
kind: ManagedCluster
metadata:
  annotations:
    serviceoperator.azure.com/reconcile-policy: skip
  name: jon-test
spec:
  agentPoolProfiles:
    - vmSize: standard_d2_v2
      name: pool0
      count: 1
      mode: System
  dnsPrefix: jon-test
  identity:
    type: SystemAssigned
  location: eastus
  owner:
    name: jon-test
  servicePrincipalProfile:
    clientId: msi

2. Observe the ManagedCluster's Ready condition fail with reason: AzureResourceNotFound (as expected)
3. Update the reconcile policy to manage:

kubectl patch managedcluster jon-test --type merge -p '{"metadata": {"annotations": {"serviceoperator.azure.com/reconcile-policy": "manage"}}}'

4. ASO starts a PUT to create the resource, but does not update the reason for the Ready condition to Reconciling

Expected behavior

Immediately after step 3, I expect the ManagedCluster's Ready condition to still be false but with reason: Reconciling to reflect that ASO has successfully begun to create the resource. This is the behavior I observe when creating the ManagedCluster with the manage reconcile policy straight away.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

CAPZ is basing some logic on the reason for the Ready condition, treating Reconciling as less severe than AzureResourceNotFound. This doesn't break any functionality for us, it only leads to some overly negative status reporting on CAPZ resources.

[nojnhuh]

My best guess is that this has to do with the logic for overwriting conditions:

azure-service-operator/v2/pkg/genruntime/conditions/conditions.go

Lines 230 to 237 in 543a399

	// SetConditionReasonAware sets the provided Condition on the Conditioner. This is similar to SetCondition
	// with one difference: SetConditionReasonAware understands common Reasons used by ASO and allows some of them to
	// modify the standard Condition priority rules. This is primarily used to allow the Reconciling condition to overwrite
	// Warning conditions raised by the operator that have been fixed. This is useful because sometimes getting a success or
	// error from Azure can take a long time, and workflows like: submit -> warning -> fix warning -> call Azure -> wait -> success
	// otherwise would continue reporting the Warning Condition until the final success step (possibly many minutes after the
	// warning was resolved).
	func SetConditionReasonAware(o Conditioner, new Condition) {

[matthchr]

Agree w/ your guess @nojnhuh. We'll take a look at fixing this