OKD 4.13.0 update cannot be verified against keyrings: verifier-public-key-ci

[p-fruck]

I recently setup a clean OKD cluster using 4.13.0-0.okd-2023-06-04-080300. Trying to upgrade to any of the later releases yields the following error:

Retrieving payload failed version="4.13.0-0.okd-2023-06-24-145750" image="registry.ci.openshift.org/origin/release@sha256:873c173b9ccb1f78bf2c5aa285227259c373191a722bcc3915375006903088a5" failure=The update cannot be verified: unable to verify sha256:873c173b9ccb1f78bf2c5aa285227259c373191a722bcc3915375006903088a5 against keyrings: verifier-public-key-ci

I tried updating to the following releases:

version="4.13.0-0.okd-2023-08-04-164726"
version="4.13.0-0.okd-2023-07-23-051208"
version="4.13.0-0.okd-2023-07-09-062029"
version="4.13.0-0.okd-2023-06-24-145750"

Reading through existing discussions, I know that there has been an expiring GPG key around april, but this shouldn't effect the releases I used. Any ideas why this is happening and how I can resolve the issue? I'd like to verify the signature before upgrading and therefore avoid using --force

[vrutkovs]

This is expected. In order to upgrade without the warning both current and planned upgrade versions should include the new key. Until mid of May, unfortunately, only the old expired key was included in the payloads.

Since #1612 you need to force the upgrade once - and

[p-fruck]

Thanks! I missed your comment on the release discussion. I started upgrading to 06-24. Unfortunately, my cluster failed to upgrade showing the following error message:

$ oc adm upgrade
Failing=True:

  Reason: MultipleErrors
  Message: Multiple errors are preventing progress:
  * Cluster operators authentication, console are not available
  * Cluster operators ingress, openshift-apiserver are degraded

info: An upgrade is in progress. Unable to apply 4.13.0-0.okd-2023-06-24-145750: an unknown error has occurred: MultipleErrors

Upgradeable=False

  Reason: PoolUpdating
  Message: Cluster operator machine-config should not be upgraded between minor versions: One or more machine config pools are updating, please see `oc get mcp` for further details

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4

Running MCP I can see that the render-master is not degraded, but still needs to update one master node:

$ oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
master   rendered-master-9aacbd03720ea6ff63c86dd7c4409edb   False     True       False      3              2                   2                     0                      2d3h
worker   rendered-worker-7b4344f18e0051f2f17ded65208d79af   True      False      False      3              3                   3                     0                      2d3h

The missing node seems to be my master 1

$ oc get nodes
NAME           STATUS                     ROLES                         AGE    VERSION
okd-master-1   Ready,SchedulingDisabled   control-plane,master,worker   2d3h   v1.26.5+7a891f0
okd-master-2   Ready                      control-plane,master,worker   2d3h   v1.26.6+a7ee68b
okd-master-3   Ready                      control-plane,master,worker   2d3h   v1.26.6+a7ee68b
okd-worker-1   Ready                      worker                        27h    v1.26.6+a7ee68b
okd-worker-2   Ready                      worker                        27h    v1.26.6+a7ee68b
okd-worker-3   Ready                      worker                        27h    v1.26.6+a7ee68b

[vrutkovs]

Lets create a new discussion on this - and please collect a must-gather for it and upload to any file sharing site for us to examine