OCPBUGS-55967: Add hot loop detection in the boot image controller

[djoshy]

- What I did
I added a simple hot loop detection counter in the boot image controller. If a machineset is updated more than 3 times to the same target boot image, the MSBIC will error and degrade the cluster. To fix this, one could:

1. Opt the cluster out of boot image updates. You can stop here if you do not want/care about boot image updates.
2. Fix the other actor in your cluster that is also reconciling the boot image. This will vary depending on the platform.
3. Opt the cluster back in for boot image updates.

- How to verify it

1. Bring up a cluster on GCP/AWS.
2. Update the boot image to a different value. On GCP, this would be the disk.image field in the providerSpec and on AWS, this would the AMI.ID field in the providerSpec. The MSBIC should immediately update the boot image back to the correct value.
3. Repeat this 3 more times. This should cause the MSBIC to error and degrade the operator.
4. Now, opt-out of boot image management - this should clear the degrade.

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-55967, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.20.0) matches configured target version for branch (4.20.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

- What I did
I added a simple hot loop detection counter in the boot image controller. If a machineset is updated more than 3 times to the same target boot image, the MSBIC will error and degrade the cluster. To fix this, one could:

1. Opt the cluster out of boot image updates. You can stop here if you do not want/care about boot image updates.
2. Fix the other actor in your cluster that is also reconciling the boot image. This will vary depending on the platform.
3. Opt the cluster back in for boot image updates.

- How to verify it

1. Bring up a cluster on GCP/AWS.
2. Update the boot image to a different value. On GCP, this would be the disk.image field in the providerSpec and on AWS, this would the AMI.ID field in the providerSpec. The MSBIC should immediately update the boot image back to the correct value.
3. Repeat this 3 more times. This should cause the MSBIC to error and degrade the operator.
4. Now, opt-out of boot image management - this should clear the degrade.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[ptalgulk01]

Pre-merge verification:

Verified using IPI based AWS and GCP based 4.20 cluster.

1.Manually change the boot image for a MachineSet for more than 3 times:

• On GCP: Change the disk.image in the providerSpec.
• On AWS: Change the ami.id in the providerSpec.

$ oc edit machinesets.machine.openshift.io  -n openshift-machine-api ci-ln-wrsgd3k-76ef8-hsbfn-worker-us-east-2b
....
      providerSpec:
       .....
          disks:
        .....
            image: projects/rhcos-cloud/global/images/rhcos-abcd

2. At first few time the value is changed to original. Later able to see the edited value with machine-config operator been degraded.

$ oc get co machine-config 
NAME             VERSION                                                AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
machine-config   4.20.0-0-2025-05-12-144624-test-ci-ln-dlgwm9b-latest   True        False         True       110m    Failed to resync 4.20.0-0-2025-05-12-144624-test-ci-ln-dlgwm9b-latest because: bootimage update failed: 1 Degraded MAPI MachineSets | 0 Degraded CAPI MachineSets | 0 CAPI MachineDeployments | Error(s): error syncing MAPI MachineSet ppt-12-20-h69rh-worker-a: refusing to reconcile machineset ppt-12-20-h69rh-worker-a, hot loop detected. Please opt-out of boot image updates, adjust your machine provisioning workflow to prevent hot loops and opt back in to resume boot image updates

$ oc  get machineconfigurations -o yaml
  - lastTransitionTime: "2025-05-13T06:58:52Z"
    message: '1 Degraded MAPI MachineSets | 0 Degraded CAPI MachineSets | 0 CAPI MachineDeployments
      | Error(s): error syncing MAPI MachineSet ppt-12-20-h69rh-worker-a: refusing
      to reconcile machineset ppt-12-20-h69rh-worker-a, hot loop detected. Please
      opt-out of boot image updates, adjust your machine provisioning workflow to
      prevent hot loops and opt back in to resume boot image updates'
    reason: MAPIMachinesetUpdated
    status: "True"
    type: BootImageUpdateDegraded

3. Opt-out of boot-mage by editing the machieconfiguration

....
  spec:
    logLevel: Normal
    managedBootImages:
      machineManagers:
      - apiGroup: machine.openshift.io
        resource: machinesets
        selection:
          mode: None
    managementState: Managed
    operatorLogLevel: Normal
  status:
....
    managedBootImagesStatus:
      machineManagers:
      - apiGroup: machine.openshift.io
        resource: machinesets
        selection:
          mode: None

/label qe-approved

[openshift-ci-robot]

@djoshy: This pull request references Jira Issue OCPBUGS-55967, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.20.0) matches configured target version for branch (4.20.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

- What I did
I added a simple hot loop detection counter in the boot image controller. If a machineset is updated more than 3 times to the same target boot image, the MSBIC will error and degrade the cluster. To fix this, one could:

1. Opt the cluster out of boot image updates. You can stop here if you do not want/care about boot image updates.
2. Fix the other actor in your cluster that is also reconciling the boot image. This will vary depending on the platform.
3. Opt the cluster back in for boot image updates.

- How to verify it

1. Bring up a cluster on GCP/AWS.
2. Update the boot image to a different value. On GCP, this would be the disk.image field in the providerSpec and on AWS, this would the AMI.ID field in the providerSpec. The MSBIC should immediately update the boot image back to the correct value.
3. Repeat this 3 more times. This should cause the MSBIC to error and degrade the operator.
4. Now, opt-out of boot image management - this should clear the degrade.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[yuqi-zhang]

/lgtm

Sufficient solution as a safety, will at least allow us some detection. Do you think it's worth adding metrics?

[yuqi-zhang]

Hmm, there should be no valid scenario where this would be updating back right? Or e.g. a fake "update" from the client?

[djoshy]

Addressed in slack; but basically we should only be calling the check here if there is a diff between the old/new bootimage, so we should be covered from empty syncs.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [djoshy,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[djoshy]

/lgtm

Sufficient solution as a safety, will at least allow us some detection. Do you think it's worth adding metrics?

I think there might be value in that as a follow-up! I'll make a card for it.

[openshift-ci]

@djoshy: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@djoshy: An error was encountered updating to the MODIFIED state for bug OCPBUGS-55967 on the Jira server at https://issues.redhat.com/. No known errors were detected, please see the full error message for details.

Full error message.

No response returned: Post "https://issues.redhat.com/rest/api/2/issue/OCPBUGS-55967/transitions": POST https://issues.redhat.com/rest/api/2/issue/OCPBUGS-55967/transitions giving up after 5 attempt(s)

Please contact an administrator to resolve this issue, then request a bug refresh with /jira refresh.

In response to this:

- What I did
I added a simple hot loop detection counter in the boot image controller. If a machineset is updated more than 3 times to the same target boot image, the MSBIC will error and degrade the cluster. To fix this, one could:

1. Opt the cluster out of boot image updates. You can stop here if you do not want/care about boot image updates.
2. Fix the other actor in your cluster that is also reconciling the boot image. This will vary depending on the platform.
3. Opt the cluster back in for boot image updates.

- How to verify it

1. Bring up a cluster on GCP/AWS.
2. Update the boot image to a different value. On GCP, this would be the disk.image field in the providerSpec and on AWS, this would the AMI.ID field in the providerSpec. The MSBIC should immediately update the boot image back to the correct value.
3. Repeat this 3 more times. This should cause the MSBIC to error and degrade the operator.
4. Now, opt-out of boot image management - this should clear the degrade.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

Distgit: ose-machine-config-operator
This PR has been included in build ose-machine-config-operator-container-v4.20.0-202505140744.p0.ga09f116.assembly.stream.el9.
All builds following this will include this PR.

[djoshy]

/cherry-pick release-4.19

[openshift-cherrypick-robot]

@djoshy: new pull request created: #5050

In response to this:

/cherry-pick release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.