Bug: ASO pod fails to start with invalid values messages regarding "v1api20240601storage"

[JDonaghy]

Describe the bug

ASO was installed via the helm chart and the instance with "azureOperatorMode: webhooks" fails with error such as:

"msg"="failed to apply CRDs" "error"="failed to apply CRD privatednszonesaaaarecords.network.azure.com: CustomResourceDefinition.apiextensions.k8s.io "privatednszonesaaaarecords.network.azure.com" is invalid: status.storedVersions[0]: Invalid value: "v1api20240601storage": must appear in spec.versions" "logger"="setup"

Each time the failing pod is restarted the error is reported for a different CRD. Occasionally it will start successfully.

Azure Service Operator Version:

mcr.microsoft.com/k8s/azureserviceoperator:v2.9.0

Expected behavior

The pos should start up reliably, or give more useful information about how to resolve the issue.

To Reproduce

I'm unsure what triggered this problem because I'm not aware of anything that has changed. It has been working reliable until very recently.

Additional context

The operator installation is managed by ArgoCD.

[theunrepentantgeek]

"msg"="failed to apply CRDs" "error"="failed to apply CRD privatednszonesaaaarecords.network.azure.com: CustomResourceDefinition.apiextensions.k8s.io "privatednszonesaaaarecords.network.azure.com" is invalid: status.storedVersions[0]: Invalid value: "v1api20240601storage": must appear in spec.versions" "logger"="setup"

Version v1api20240601storage of privatednszonesaaaarecords.network.azure.com was only just released as a part of version v2.11.0 (ref).

Do you have both v2.9.0 and v2.11.0 running at the same time? If so, they'll be contesting for control over the CRDs which would lead to the kinds of failures you're seeing.

[JDonaghy]

Thank you @theunrepentantgeek, that was the problem. I have a pipeline that installs ASO via helm, but I forgot to specify the version there, so it installs the latest every time I run it. I also have ArgoCD manage it, and I did specify 2.9.0 there. So it was working until I reran the pipeline that installed 2.11.0.

I'm planning to just leverage ArgoCD for the install going forward and use sync-waves to make sure things get installed in the right order.