Issue during kubeflow upgrade from 1.6 to 1.7

[shan100github]

Existing environment:
Kubernetes Environment before upgrade: Master and Worker are in v1.22.15+rke2r1
Kubeflow version: 1.6

Upgraded environment:
Kubernetes Environment before upgrade: Master and Worker are in v1.24.12+rke2r1
Kubeflow version: 1.7

Issue faced:
During the upgrade the following error occured.

resource mapping not found for name: "eventing-webhook" namespace: "knative-eventing" from "STDIN": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2"
ensure CRDs are installed first
resource mapping not found for name: "activator" namespace: "knative-serving" from "STDIN": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2"
ensure CRDs are installed first

The solution to overcome the above issue:
Uninstalled the following

cd manifests1.6.x
kustomize build common/knative/knative-serving/overlays/gateways | kubectl delete -f -
kustomize build common/istio-1-14/cluster-local-gateway/base | kubectl delete -f -
kustomize build common/knative/knative-eventing/base | kubectl delete -f -

Retry install kubeflow 1.7

Overall upgrade information:

• Upgrade kubeflow from 1.6 to 1.7
• Upgrade kubernetes from 1.22 to 1.23 and 1.23 to 1.24

Ref: https://kubeflow.slack.com/archives/C01EY3L525N/p1680644579763759

[aii-shanker-jj]

After upgrading k8s to 1.24, upon retrying following command installed Knative and KServe

while ! kustomize build example | awk '!/well-defined/' | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

[Lejboelle]

Just want to highlight another issue I ran into:

I am running k8s 1.24.9 and wanted to upgrade from KF 1.6.1 -> 1.7.0.
Upgrade went fine, but after completion my inference service started scaling up like crazy (like 50 replicas!).
The issue was the old statefulset kserve-controller-manager which was still present.

Solution:
kubectl delete statefulset kserve-controller-manager -n kubeflow

After this operation everything is behaving normally.

[juliusvonkohout]

duplicate of #2216

/close

[google-oss-prow]

@juliusvonkohout: Closing this issue.

In response to this:

duplicate of #2216

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[juliusvonkohout]

Solved by #2717

[shan100github]

It's probably an old write-up, might help someone.
https://medium.com/@shan-ops/mlops-platform-on-rancher-rke2-kubernetes-cluster-bare-metal-environment-kubeflow-1-7-upgrade-8def9be07c1d