Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue during kubeflow upgrade from 1.6 to 1.7 #2453

Closed
shan100github opened this issue May 2, 2023 · 6 comments
Closed

Issue during kubeflow upgrade from 1.6 to 1.7 #2453

shan100github opened this issue May 2, 2023 · 6 comments

Comments

@shan100github
Copy link

Existing environment:
Kubernetes Environment before upgrade: Master and Worker are in v1.22.15+rke2r1
Kubeflow version: 1.6

Upgraded environment:
Kubernetes Environment before upgrade: Master and Worker are in v1.24.12+rke2r1
Kubeflow version: 1.7

Issue faced:
During the upgrade the following error occured.

resource mapping not found for name: "eventing-webhook" namespace: "knative-eventing" from "STDIN": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2"
ensure CRDs are installed first
resource mapping not found for name: "activator" namespace: "knative-serving" from "STDIN": no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2"
ensure CRDs are installed first

The solution to overcome the above issue:
Uninstalled the following

cd manifests1.6.x
kustomize build common/knative/knative-serving/overlays/gateways | kubectl delete -f -
kustomize build common/istio-1-14/cluster-local-gateway/base | kubectl delete -f -
kustomize build common/knative/knative-eventing/base | kubectl delete -f -

Retry install kubeflow 1.7

Overall upgrade information:

  • Upgrade kubeflow from 1.6 to 1.7
  • Upgrade kubernetes from 1.22 to 1.23 and 1.23 to 1.24

Ref: https://kubeflow.slack.com/archives/C01EY3L525N/p1680644579763759

@aii-shanker-jj
Copy link

After upgrading k8s to 1.24, upon retrying following command installed Knative and KServe

while ! kustomize build example | awk '!/well-defined/' | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done

@Lejboelle
Copy link

Lejboelle commented Jun 6, 2023

Just want to highlight another issue I ran into:

I am running k8s 1.24.9 and wanted to upgrade from KF 1.6.1 -> 1.7.0.
Upgrade went fine, but after completion my inference service started scaling up like crazy (like 50 replicas!).
The issue was the old statefulset kserve-controller-manager which was still present.

Solution:
kubectl delete statefulset kserve-controller-manager -n kubeflow

After this operation everything is behaving normally.

@juliusvonkohout
Copy link
Member

duplicate of #2216

/close

@google-oss-prow
Copy link

@juliusvonkohout: Closing this issue.

In response to this:

duplicate of #2216

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@juliusvonkohout
Copy link
Member

Solved by #2717

@shan100github
Copy link
Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants