OCPBUGS-55638: OCPBUGS-55637: Add admin_acks handling in the MCO #5027

djoshy · 2025-05-02T19:55:51Z

- What I did
I added a new sync loop in the operator for the admin-gates configmap in the openshift-config-managed namespace. This sync loop will now begin to add the following keys:

ack-4.18-boot-image-opt-out-in-4.19 when a AWS/GCP cluster does not have a boot image configuration.
ack-4.18-aarch64-bootloader-4.19 when an aarch64/arm64 node is detected in the cluster.

These gates are only relevant for upgrades to 4.19, so they will only need to exist in the MCO's release-4.18 branch. I've also added a few units for this functionality.

- How to verify it
Before testing either case, ensure that any non MCO keys in admin-gates have been acknowledged. For example, if there is a key called ack-4.18-kube-1.32-api-removals-in-4.19 in admin-gates, add this key to the admin-acks configmap in the openshift-config namespace with its value set to true like so:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    include.release.openshift.io/hypershift: "true"
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    kubernetes.io/description: Record administrator acknowledgments of update gates
      defined in the admin-gates ConfigMap in the openshift-config-managed namespace.
    release.openshift.io/create-only: "true"
  creationTimestamp: "2025-05-13T07:51:46Z"
  name: admin-acks
  namespace: openshift-config
  resourceVersion: "2284"
  uid: fdd6b65e-5c83-4d41-a08d-cba2bbdf1e91
data:
  ack-4.18-kube-1.32-api-removals-in-4.19: "true"

AWS/GCP boot image opt-out:
- Launch this PR against a 4.18 cluster on the AWS or GCP platform without a boot image configuration defined. The CVO should trip Upgradeable=False (check via oc get clusterversion -o yaml). Now, add a boot image configuration - the CVO should no longer be setting Upgradeable to false.
- This gate should not show up on any other platforms.
aarch64 bootloader:
- Launch this PR against a 4.18 aarch64 cluster. The CVO should trip Upgradeable=False (check via oc get clusterversion -o yaml).
- This gate should not show up on clusters of any other architectures.

openshift-ci-robot · 2025-05-02T19:55:57Z

@djoshy: This pull request references Jira Issue OCPBUGS-55638, which is invalid:

release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
expected Jira Issue OCPBUGS-55638 to depend on a bug targeting a version in 4.19.0 and in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

[DNM, currently testing]

- What I did

Added a sync loop for admin-guards configmap with support for AWS/GCP boot image guard.

- How to verify it

Launch this PR against a 4.18 cluster on the AWS or GCP platform without a boot image configurationm defined. The CVO should trip Upgradeable=False. Now, add a boot image configuration - this should set Upgradeable back to true.

This PR should have no effect on other platforms.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

djoshy · 2025-05-06T20:00:55Z

/retest-required

djoshy · 2025-05-06T20:28:27Z

/retest-required

yuqi-zhang

Generally lgtm, will of course need to wait for the docs to exist before we can merge

Some minor questions inline, but I think they're mostly clear

yuqi-zhang · 2025-05-07T03:29:19Z

pkg/operator/admin_ack.go

+	BootImageAWSGCPMsg = "GCP or AWS platform with no boot image configuration detected. OCP will automatically opt-in all GCP and AWS clusters that currently do not a boot image configuration in 4.19. Please add a configuration to disable boot image updates if this is not desired. See [insert-doc-link] "
+
+	AArch64BootImageKey = "ack-4.18-aarch64-bootloader-4.19"
+	AArch64BootImageMsg = "aarch64 nodes detected. Please ensure boot image are updated by following the KCS:(insertlink) prior to upgrading to 4.19."


nit: updated sounds like it's an action that always needs to be taken, maybe we should say something like "ensure boot image is not too old" instead? (I couldn't think of a better way to phrase it)

Yeah, that makes sense to me! Perhaps we should run the messages through a doc/Trevor review, they might have better insight/worked on this before

yuqi-zhang · 2025-05-07T03:36:07Z

pkg/operator/admin_ack.go

+	}
+
+	// If no machine managers exist, no configuration has been defined
+	return mcop.Spec.ManagedBootImages.MachineManagers == nil, nil


So this is saying that if the user already has opted in in a previous version, we don't raise the admin ack?

(makes sense, just thinking out loud)

Not necessarily opt-ed in, but if they have any sort of opinion(all/none/partial), we'll either remove the admin-ack if one exists, or otherwise a no-op

yuqi-zhang · 2025-05-07T03:38:59Z

pkg/operator/admin_ack.go

+// updateGuardKeyIfNeeded adds a key with a message depending on:
+// If guard is needed and key doesn't exist => create it
+// If guard is needed and key exists => nothing to do
+// If guard is not needed and key exists => delete it


This is interesting, in the sense that we do clear stale admin acks, but only before you do the upgrade, since these syncs will be gone after 4.18.

(again, this is fine, just thinking out loud)

Right, so I wanted the user to able to clear the gate either by addressing it via the admin-ack configmap or via defining an boot image opinion. In 4.19, this should not have any effect per Trevor since these keys only target 4.18(but we can double check that again)

yuqi-zhang · 2025-05-07T03:45:07Z

pkg/operator/admin_ack_test.go

+			adminCM: buildAdminAckConfigMapWithData(nil),
+		},
+		{
+			name:    "AWS platform, with a boot image configuration",


These tests all have a "disabled" boot image configuration (empty machinemanagers), and thereby do not need the admin ack, right?

Right, disabled meaning the user wanted to opt out so no need to raise the alarm. I can add comments/make this more verbose if its not very clear in the current state 😄

djoshy · 2025-05-07T12:57:48Z

/retest-required

yuqi-zhang

Leaving an approval for the code, but should wait until docs are in place to merge

openshift-ci · 2025-05-08T00:25:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [djoshy,yuqi-zhang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

djoshy · 2025-05-13T14:06:50Z

Updated to include doc links.

ack-4.18-boot-image-opt-out-in-4.19 => https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/machine_configuration/mco-update-boot-images#mco-update-boot-images
ack-4.18-aarch64-bootloader-4.19 => https://access.redhat.com/solutions/7119697 (still WIP)

openshift-ci · 2025-05-13T18:18:20Z

@djoshy: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-op-ocl	`9f5367d`	link	false	`/test e2e-gcp-op-ocl`
ci/prow/e2e-vsphere-ovn-upi-zones	`9f5367d`	link	false	`/test e2e-vsphere-ovn-upi-zones`
ci/prow/e2e-aws-ovn-upgrade-out-of-change	`9f5367d`	link	false	`/test e2e-aws-ovn-upgrade-out-of-change`
ci/prow/e2e-vsphere-ovn-upi	`9f5367d`	link	false	`/test e2e-vsphere-ovn-upi`
ci/prow/e2e-vsphere-ovn-zones	`9f5367d`	link	false	`/test e2e-vsphere-ovn-zones`
ci/prow/e2e-gcp-op-techpreview	`9f5367d`	link	false	`/test e2e-gcp-op-techpreview`
ci/prow/e2e-azure-ovn-upgrade-out-of-change	`9f5367d`	link	false	`/test e2e-azure-ovn-upgrade-out-of-change`
ci/prow/e2e-gcp-op	`9f5367d`	link	true	`/test e2e-gcp-op`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

ptalgulk01 · 2025-05-16T12:19:53Z

Pre-merge verified :
Verified using IPI based AWS cluster

launch 4.18,openshift/machine-config-operator#5027 aws

Edit the admin-acks -n openshift-config

oc edit cm admin-acks -n openshift-config -o yaml
apiVersion: v1
data:
  ack-4.18-kube-1.32-api-removals-in-4.19: "true"
kind: ConfigMap
metadata:
  annotations:
    include.release.openshift.io/hypershift: "true"
    include.release.openshift.io/ibm-cloud-managed: "true"
    include.release.openshift.io/self-managed-high-availability: "true"
    kubernetes.io/description: Record administrator acknowledgments of update gates
      defined in the admin-gates ConfigMap in the openshift-config-managed namespace.
    release.openshift.io/create-only: "true"
  creationTimestamp: "2025-05-16T09:20:54Z"
  name: admin-acks
  namespace: openshift-config
  resourceVersion: "36986"
  uid: 167b0343-a43b-4e6d-b948-fa8c5a22cccc

Able to see Upgradeable = False before editing the bootImage value
Edit the machineconfiguration

....
  spec:
    logLevel: Normal
    managedBootImages:
      machineManagers:
      - apiGroup: machine.openshift.io
        resource: machinesets
        selection:
          mode: All
    managementState: Managed
    operatorLogLevel: Normal
  status:

After edit as far now able to see the Upgradeable is disappered from clusterversion and to make it appear again edit the data field from admin-acks -n openshift-config

yuqi-zhang · 2025-05-16T19:32:40Z

/label backport-risk-assessed

Not an actual backport

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 2, 2025

openshift-ci bot requested review from cheesesashimi and umohnani8 May 2, 2025 19:56

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 2, 2025

djoshy force-pushed the 4.18-boot-image-admin-ack branch from 0bbd233 to eaebaed Compare May 5, 2025 14:23

djoshy changed the title ~~[DNM] OCPBUGS-55638: Add admin_acks handling in the MCO~~ OCPBUGS-55638: Add admin_acks handling in the MCO May 5, 2025

djoshy force-pushed the 4.18-boot-image-admin-ack branch from eaebaed to 910ea66 Compare May 6, 2025 19:33

openshift-merge-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 6, 2025

operator: add admin_ack handling

ec75d6e

djoshy force-pushed the 4.18-boot-image-admin-ack branch from 910ea66 to 157c091 Compare May 6, 2025 19:37

openshift-merge-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 6, 2025

djoshy changed the title ~~OCPBUGS-55638: Add admin_acks handling in the MCO~~ OCPBUGS-55638: OCPBUGS-55637: Add admin_acks handling in the MCO May 6, 2025

yuqi-zhang reviewed May 7, 2025

View reviewed changes

yuqi-zhang approved these changes May 8, 2025

View reviewed changes

operator: add admin_ack unit tests

9f5367d

djoshy force-pushed the 4.18-boot-image-admin-ack branch from 157c091 to 9f5367d Compare May 13, 2025 14:05

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label May 16, 2025

openshift-ci bot assigned anuragthehatter, eurijon and mhanss May 16, 2025

openshift-ci bot assigned ptalgulk01, rbbratta, rioliu-rh, rlobillo, sergiordlr and zhaozhanqi May 16, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCPBUGS-55638: OCPBUGS-55637: Add admin_acks handling in the MCO #5027

OCPBUGS-55638: OCPBUGS-55637: Add admin_acks handling in the MCO #5027

djoshy commented May 2, 2025 •

edited

Loading

openshift-ci-robot commented May 2, 2025

djoshy commented May 6, 2025

djoshy commented May 6, 2025

yuqi-zhang left a comment

yuqi-zhang May 7, 2025

djoshy May 7, 2025

yuqi-zhang May 7, 2025

djoshy May 7, 2025

yuqi-zhang May 7, 2025

djoshy May 7, 2025

yuqi-zhang May 7, 2025

djoshy May 7, 2025

djoshy commented May 7, 2025

yuqi-zhang left a comment

openshift-ci bot commented May 8, 2025

djoshy commented May 13, 2025

openshift-ci bot commented May 13, 2025

ptalgulk01 commented May 16, 2025

yuqi-zhang commented May 16, 2025

OCPBUGS-55638: OCPBUGS-55637: Add admin_acks handling in the MCO #5027

Are you sure you want to change the base?

OCPBUGS-55638: OCPBUGS-55637: Add admin_acks handling in the MCO #5027

Conversation

djoshy commented May 2, 2025 • edited Loading

openshift-ci-robot commented May 2, 2025

djoshy commented May 6, 2025

djoshy commented May 6, 2025

yuqi-zhang left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djoshy commented May 7, 2025

yuqi-zhang left a comment

Choose a reason for hiding this comment

openshift-ci bot commented May 8, 2025

djoshy commented May 13, 2025

openshift-ci bot commented May 13, 2025

ptalgulk01 commented May 16, 2025

yuqi-zhang commented May 16, 2025

djoshy commented May 2, 2025 •

edited

Loading