Cluster Version Operator overrides prevents OKD upgrades #1462

robercondori · 2023-01-12T18:22:19Z

robercondori
Jan 12, 2023

Describe the bug
OKD upgrade is blocked due to Precondition fails.

Precondition error message:
sync_worker.go:348] Precondition error (force false, block true): Precondition "ClusterVersionUpgradeable" failed because of "ClusterVersionOverridesSet": Disabling ownership via cluster version overrides prevents upgrades. Please remove overrides before continuing.

"oc adm upgrade" output:

info: An upgrade is in progress. Working towards 4.11.0-0.okd-2022-10-28-153352

Upgradeable=False

Reason: ClusterVersionOverridesSet
Message: Disabling ownership via cluster version overrides prevents upgrades. Please remove overrides before continuing.

ReleaseAccepted=False

Reason: PreconditionChecks
Message: Preconditions failed for payload loaded version="4.11.0-0.okd-2022-10-28-153352" image="registry.ci.openshift.org/origin/release@sha256:13fceee2e4e919d942b8252e8a86b1bd62371cb88cfcb45746d6f232a7b9acba": Precondition "ClusterVersionUpgradeable" failed because of "ClusterVersionOverridesSet": Disabling ownership via cluster version overrides prevents upgrades. Please remove overrides before continuing.

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4

Recommended updates:

VERSION IMAGE
4.11.0-0.okd-2022-10-28-153352 registry.ci.openshift.org/origin/release@sha256:13fceee2e4e919d942b8252e8a86b1bd62371cb88cfcb45746d6f232a7b9acba
4.11.0-0.okd-2022-10-15-073651 registry.ci.openshift.org/origin/release@sha256:1f72fb4fc1452da5c1300f7f7ba376599d9cea41e00a141ba738d2620b51d542

There are Overrides on CVO which is blocking the upgrade process. Check the output of "oc get clusterversion -o yaml":

apiVersion: v1
items:

apiVersion: config.openshift.io/v1
kind: ClusterVersion
metadata:
creationTimestamp: "2023-01-04T16:46:58Z"
generation: 3
name: version
resourceVersion: "5226742"
uid: 2e3bb7ae-f261-44bd-818a-47804ef508b7
spec:
channel: stable-4
clusterID: 241e108c-102e-4a0d-a43c-4e4562088e78
desiredUpdate:
force: false
image: registry.ci.openshift.org/origin/release@sha256:13fceee2e4e919d942b8252e8a86b1bd62371cb88cfcb45746d6f232a7b9acba
version: 4.11.0-0.okd-2022-10-28-153352
overrides:
- group: ""
  kind: ConfigMap
  name: cloud-provider-config
  namespace: openshift-config
  unmanaged: true
- group: ""
  kind: ConfigMap
  name: cluster-config-v1
  namespace: kube-system
  unmanaged: true
- group: config.openshift.io
  kind: DNS
  name: cluster
  namespace: ""
  unmanaged: true
- group: config.openshift.io
  kind: Infrastructure
  name: cluster
  namespace: ""
  unmanaged: true
- group: config.openshift.io
  kind: Ingress
  name: cluster
  namespace: ""
  unmanaged: true
- group: apiextensions.k8s.io
  kind: CustomResourceDefinition
  name: networks.config.openshift.io
  namespace: ""
  unmanaged: true
- group: config.openshift.io
  kind: Network
  name: cluster
  namespace: ""
  unmanaged: true
- group: config.openshift.io
  kind: Proxy
  name: cluster
  namespace: ""
  unmanaged: true
- group: config.openshift.io
  kind: Scheduler
  name: cluster
  namespace: ""
  unmanaged: true
- group: ""
  kind: Secret
  name: kube-cloud-cfg
  namespace: kube-system
  unmanaged: true
- group: ""
  kind: ConfigMap
  name: root-ca
  namespace: kube-system
  unmanaged: true
- group: ""
  kind: Secret
  name: machine-config-server-tls
  namespace: openshift-machine-config-operator
  unmanaged: true
- group: ""
  kind: Secret
  name: pull-secret
  namespace: openshift-config
  unmanaged: true
- group: ""
  kind: Secret
  name: vsphere-creds
  namespace: kube-system
  unmanaged: true
- group: ""
  kind: Secret
  name: kubeadmin
  namespace: kube-system
  unmanaged: true
- group: rbac.authorization.k8s.io
  kind: Role
  name: vsphere-creds-secret-reader
  namespace: kube-system
  unmanaged: true
- group: ""
  kind: ConfigMap
  name: openshift-install-manifests
  namespace: openshift-config
  unmanaged: true
  upstream: https://amd64.origin.releases.ci.openshift.org/graph
  status:
  availableUpdates:
- image: registry.ci.openshift.org/origin/release@sha256:13fceee2e4e919d942b8252e8a86b1bd62371cb88cfcb45746d6f232a7b9acba
  version: 4.11.0-0.okd-2022-10-28-153352
- image: registry.ci.openshift.org/origin/release@sha256:1f72fb4fc1452da5c1300f7f7ba376599d9cea41e00a141ba738d2620b51d542
  version: 4.11.0-0.okd-2022-10-15-073651
  capabilities: {}
  conditions:
- lastTransitionTime: "2023-01-04T16:47:01Z"
  status: "True"
  type: RetrievedUpdates
- lastTransitionTime: "2023-01-04T16:47:01Z"
  message: Disabling ownership via cluster version overrides prevents upgrades.
  Please remove overrides before continuing.
  reason: ClusterVersionOverridesSet
  status: "False"
  type: Upgradeable
- lastTransitionTime: "2023-01-04T16:47:01Z"
  message: Capabilities match configured spec
  reason: AsExpected
  status: "False"
  type: ImplicitlyEnabledCapabilities
- lastTransitionTime: "2023-01-11T16:48:50Z"
  message: 'Preconditions failed for payload loaded version="4.11.0-0.okd-2022-10-28-153352"
  image="registry.ci.openshift.org/origin/release@sha256:13fceee2e4e919d942b8252e8a86b1bd62371cb88cfcb45746d6f232a7b9acba":
  Precondition "ClusterVersionUpgradeable" failed because of "ClusterVersionOverridesSet":
  Disabling ownership via cluster version overrides prevents upgrades. Please
  remove overrides before continuing.'
  reason: PreconditionChecks
  status: "False"
  type: ReleaseAccepted
- lastTransitionTime: "2023-01-04T17:46:01Z"
  message: Done applying 4.11.0-0.okd-2022-08-20-022919
  status: "True"
  type: Available
- lastTransitionTime: "2023-01-04T17:46:01Z"
  status: "False"
  type: Failing
- lastTransitionTime: "2023-01-11T21:10:21Z"
  message: Working towards 4.11.0-0.okd-2022-10-28-153352
  status: "True"
  type: Progressing
  desired:
  image: registry.ci.openshift.org/origin/release@sha256:13fceee2e4e919d942b8252e8a86b1bd62371cb88cfcb45746d6f232a7b9acba
  version: 4.11.0-0.okd-2022-10-28-153352
  history:
- completionTime: null
  image: registry.ci.openshift.org/origin/release@sha256:13fceee2e4e919d942b8252e8a86b1bd62371cb88cfcb45746d6f232a7b9acba
  startedTime: "2023-01-11T21:10:21Z"
  state: Partial
  verified: false
  version: 4.11.0-0.okd-2022-10-28-153352
- completionTime: "2023-01-04T17:46:01Z"
  image: quay.io/openshift/okd@sha256:7a99c7af2f3e6107107a6cf92669fef74ebbd0cc1780fbd187be2410b003a429
  startedTime: "2023-01-04T16:47:01Z"
  state: Completed
  verified: false
  version: 4.11.0-0.okd-2022-08-20-022919
  observedGeneration: 3
  versionHash: ljJhEGzflFM=
  kind: List
  metadata:
  resourceVersion: ""

There are a lot of Resources with an "unmanaged" state like Ingress, Network, DNS, secrets and configmaps which are causing Overrides on CVO. For example, even though I removed "kubeadmin" secret on "kube-system" ns to disabled kubeadmin user, it still shows up as overrides in CVO.

Is this some kind of issue on OKD? or how can I troubleshooting it? Cluster Operator Pod on ns "openshift-cluster-version" doesn't provide helpful information. Also, Cluster Operators are in a ready state and working as expected.

NAME VERSION AVAILABLE PROGRESSING DEGRADED SINCE MESSAGE
authentication 4.11.0-0.okd-2022-08-20-022919 True False False 6h46m
baremetal 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
cloud-controller-manager 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
cloud-credential 4.11.0-0.okd-2022-08-20-022919 True False False 8d
cluster-autoscaler 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
config-operator 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
console 4.11.0-0.okd-2022-08-20-022919 True False False 19h
csi-snapshot-controller 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
dns 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
etcd 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
image-registry 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
ingress 4.11.0-0.okd-2022-08-20-022919 True False False 19h
insights 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
kube-apiserver 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
kube-controller-manager 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
kube-scheduler 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
kube-storage-version-migrator 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
machine-api 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
machine-approver 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
machine-config 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
marketplace 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
monitoring 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
network 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
node-tuning 4.11.0-0.okd-2022-08-20-022919 True False False 134m
openshift-apiserver 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
openshift-controller-manager 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
openshift-samples 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
operator-lifecycle-manager 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
operator-lifecycle-manager-catalog 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
operator-lifecycle-manager-packageserver 4.11.0-0.okd-2022-08-20-022919 True False False 19h
service-ca 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h
storage 4.11.0-0.okd-2022-08-20-022919 True False False 7d23h

Version
4.11.0-0.okd-2022-08-20-022919
IPI installation on vSphere 7.0.3

How reproducible
oc adm upgrade --to-latest

vrutkovs · 2023-01-13T08:09:06Z

vrutkovs
Jan 13, 2023
Maintainer

CVO won't start upgrade if payload is modified via overrides, yes

2 replies

jarolcacique Jul 25, 2023

Hello... I have the same behavior ... briefly, what do I need to do to continue with the update process?

vrutkovs Jul 25, 2023
Maintainer

Overrides need to be removed from ClusterVersion

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cluster Version Operator overrides prevents OKD upgrades #1462

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 2 replies

{{title}}

{{title}}

{{title}}

Select a reply

Cluster Version Operator overrides prevents OKD upgrades #1462

robercondori Jan 12, 2023

Replies: 1 comment · 2 replies

vrutkovs Jan 13, 2023 Maintainer

jarolcacique Jul 25, 2023

vrutkovs Jul 25, 2023 Maintainer

robercondori
Jan 12, 2023

Replies: 1 comment 2 replies

vrutkovs
Jan 13, 2023
Maintainer

vrutkovs Jul 25, 2023
Maintainer