KF 1.5 tracking

[kimwnasptd]

Will use this issue for exposing high level updates for the KF 1.5 release.

We will split the communications for each WG in its respective issue, to keep relative information in the correct places. The first thing to note is a link to the schedule: kubeflow/community#549

You can also find the WG discussions in the following issues:

• AutoML WG and Kubeflow 1.5 release #2106
• Notebooks WG and Kubeflow 1.5 release #2109
• Pipelines WG and Kubeflow 1.5 release #2111
• Training Operator WG and Kubeflow 1.5 release #2105
• KServe and Kubeflow 1.5 release #2110

cc @kubeflow/release-team

[kimwnasptd]

A heads up regarding common dependencies and K8s version, from the Manifests WG.

I'm reviewing the following PRs, which correspond to the following versions:

• Updating cert-manager to v1.5.0 to enable KF on Kubernetes 1.22 #2076
• Updating istio to v1.11 to support Kubernetes 1.22 #2077
  These are the only big features we will be reviewing and trying to merge throughout the feature freeze phase

Regarding K8s version, we are targeting 1.20 and 1.21 for this release. This comes hand in hand with the automated tests effort #2099

[kimwnasptd]

cc @andreyvelich @johnugeorge @zijianjoy @gaocegege

[zijianjoy]

Question: Looks like there are some major effort to support Kubernetes 1.22. But we are still targetting 1.20 and 1.21. May I ask what is left for the whole Kubeflow to support Kubernetes 1.22?

[kimwnasptd]

May I ask what is left for the whole Kubeflow to support Kubernetes 1.22?

I'll open a specific issue for this this, so that we can track the state of each WG and the remaining work for this.

[kimwnasptd]

Regarding the state of the release, we've cut the v1.5.0-rc.0 tag of the release yesterday.

From Manifests WG we are working full throttle to introduce the basic tests #2099 to help with the manifests testing phase, after the Feature Freeze.

For the other WGs, feel free to tag the @kubeflow/release-team to any issue you think that might affect the release or that we need to track.

[kimwnasptd]

I've bumped into 2 important issues in the Notebook Controller #2109 (comment). Hopefully I'll have the fixes merged tomorrow to have them included in the next RC tomorrow

[Tomcli]

Hi @kimwnasptd, can you help us update the kfp-tekton release to v1.1.1? We patched some fixes for cache deployer to KFP for k8s 1.22 and we have to also update the kfp-tekton manifests to sync those changes. Thanks.
https://github.com/kubeflow/kfp-tekton/tree/v1.1.1/manifests/kustomize

Our team at IBM is testing on k8s 1.22 now and we will post issues on other WGs as well.

[juliusvonkohout]

i have encountered two regressions from 1.4 to 1.5rc1 so far :

1. use oidc-authservice(keycloak) centraldashboard(ui) logout not working kubeflow#6372
2. 1.5rc1 jupyter GPU support and poddefaults support is broken kubeflow#6382

[kimwnasptd]

Thanks for the heads up @juliusvonkohout, looking into them now.

[kimwnasptd]

A heads up for the next RC2 of manifests; we'll create a (hopefully) final RC that will include the following updates:

1. Bump to KFP 1.8.1 Update kubeflow/pipelines manifests from 1.8.1 #2152
2. Updated manifests for using the KServe app in the CentralDashboard centraldashboard: Add KServe overlay kubeflow#6383 manifests: Update gateways and prefix kserve/models-web-app#32
3. Bumps to the final versions of the other components
   • For Notebooks we are tracking the above items KF 1.5 tracking #2112 (comment)

[kimwnasptd]

A heads up, I've cut the v1.5.0-rc.2 in the Manifests repo. All of the components, aside from the Notebooks that live in kubeflow/kubeflow are in their final state.

We are expecting this RC to become the final one on March 9th 2022.

I've also run the full test script with this RC and everything is passing. But if you bump into any issues please report so we can address them accordingly.

Changes from previous RC

The notable changes between this RC and the previous one are:

• Sync kubeflow katib manifests v0.13.0 #2156
• Update kubeflow/pipelines manifests from 1.8.1 #2152
• Making the KServe Models web app the default
  • kserve: Update the Models web app #2158
  • kserve: Add distinct file for cluster-roles #2153
• Fix Notebook the controller error logs flood
  • 1.5rc1 jupyter GPU support and poddefaults support is broken kubeflow#6382
  • Update kubeflow/kubeflow manifests from v1.5.0-rc.2 #2159

KServe and KFServing LTS

Lastly, regarding KServe 0.7 and KFServing 0.6.1:

1. Both are included in the example kustomization
2. The KServe models web app is used by default
3. To switch between the two web apps an admin will need to select which overlay of the CentralDashboard to use.

I didn't have the time to update the Knative version, so we'll remain with 0.22.1.

cc @kubeflow/release-team @kubeflow/wg-automl-leads @kubeflow/wg-notebooks-leads @kubeflow/wg-pipeline-leads @kubeflow/wg-training-leads @zijianjoy

[juliusvonkohout]

#2160 could be included into 1.5rc2 without building new images. So far Jupyter (the name notebook is wrong), KServe, Katib etc. is working. I did not test the 1.8.1 pipelines extensively, because i am testing a multi-user namespace isolation implementation with one Minio bucket per namespace.

[juliusvonkohout]

expecting this RC to become the final one on March 9th 2022.

There still is the logout bug kubeflow/kubeflow#6372

[surajkota]

kubeflow/kubeflow#6339 could be included without building images as well. It is being tracked for 1.5 docs as well

[jbottum]

@kimwnasptd are we going to include the new issues identified above that are not in RC2? Will there be an RC3 and if so, what will be the date for that ?

[yhwang]

@kimwnasptd thanks for the update. About the RC.2. does updating the knative version need lots of effort? I know for testing definitely needs time to go through all the test cases. Just really hope we can have knative 0.22.3 in kubeflow v1.5.

[chintha-v]

Please help us with Info: When can we expect final 1.5 version released? Will it be compatible with Kubernetes 1.22? We need to plan our activities accordingly.

[chintha-v]

Because of kubeflow in our product we are not able to upgrade kubernetes to 1.22 version. Which version of Kubeflow will be having Kubernetes 1.22 compatibilty? When can we expect this?

[surajkota]

Help needed to resolve this issue: #2146 (comment)

AWS distribution testing is blocked because of this

[chintha-v]

Can someone help me with info: Which version of Kubeflow will be having Kubernetes 1.22 compatibilty and When can we expect that?

[juliusvonkohout]

Can someone help me with info: Which version of Kubeflow will be having Kubernetes 1.22 compatibilty and When can we expect that?

1.5rc2 still throws deprecation warnings. You could build the manifest search for v1beta1 and create a pull request to fix it. Otherwise good luck waiting six months.

[annajung]

@chintha-v All components except notebook support k8s v1.22. You can track the notebook WG's 1.22 status here

K8s 1.22 support is on the roadmap for the next Kubeflow release.

The release is scheduled for today (March 9th), but there has been a request to delay the release in the last few hours which will need to be reviewed by the release team. After review, if there has been a delay, we'll send out an announcement via google group and update the thread accordingly.

[kimwnasptd]

Small update regarding the timing of the release in #2146 (comment)

[annajung]

Another release update, copying comment from here

We've gotten to the bottom of the issue. This is a problem with any K8s cluster that does not support using signerName: kubernetes.io/kubelet-serving in CertificateSigningRequests, and EKS is such a case.

I want to further understand the following first:

1. What is the best practice around such certificates?
2. Is it a problem to give a certificate, aimed to be used by kubelet, to the cache-deployer webhook?
3. What is the long term solution and how quickly could it be implemented?

I'd like to first have an answer for the above, before pushing the release button. For this I'll be delaying the release just for one more day, to take a look with a more clear mind and have answers on the above and a solid plan going forward.

We'll also add more technical details into #2165, which we'll at some point bring back to the KFP repo to discuss next steps.

[chintha-v]

@annajung Thanks for the info. We have been planing to upgrade kubernetes to 1.22 but waiting for compatible kubeflow release. To plan our activities accordingly, we need exact date or at least month by which we can get next kubeflow release having 1.22 compatibility. Please share the required information.

[kimwnasptd]

A heads up, we are moving on with cutting the KF 1.5.0 release now.

We are still discussing about a proper fix for #2165, which could be included in KF 1.5.1.

[kimwnasptd]

@chintha-v we are targeting 1.22 support in the KF 1.6 release. You can see an overview of the release cycle in https://github.com/kubeflow/community/blob/master/releases/handbook.md#timeline.

It's a 4month release cadence, but the dates might slide depending on how quickly the 1.6 release will start. And of course there's always a chance of delays, depending on reported bugs.

[juliusvonkohout]

@kimwnasptd i noticed that some commits from March 8 and March 10 are missing from the v1.5-branch branch even though you released 1.5. Especially 8120cdb is missing

[juliusvonkohout]

/close

There has been no activity for a long time. Please reopen if necessary.

[google-oss-prow]

@juliusvonkohout: Closing this issue.

In response to this:

/close

There has been no activity for a long time. Please reopen if necessary.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.