Backport ModelRegistry Component #329

ykaliuta · 2024-08-27T14:33:52Z

Jira: https://issues.redhat.com/browse/RHOAIENG-4272

ykaliuta · 2024-08-28T10:22:54Z

From the log https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/red-hat-data-services_rhods-operator/329/pull-ci-red-hat-data-services-rhods-operator-main-rhods-operator-e2e/1828679610703810560/artifacts/rhods-operator-e2e/gather-extra/artifacts/pods/redhat-ods-operator_redhat-ods-operator-controller-manager-d5f8fbcbd-dmfq4_manager.log

{"level":"error","ts":"2024-08-28T07:13:25Z","logger":"opendatahub.controllers.DSCInitialization","msg":"error to set networkpolicy in operator namespace","path":"/opt/manifests/monitoring/networkpolicy","error":"networkpolicies.networking.k8s.io \"redhat-ods-operator\" is forbidden: cannot set blockOwnerDeletion in this case because cannot find RESTMapping for APIVersion dscinitialization.opendatahub.io/v1 Kind DSCInitialization: no matches for kind \"DSCInitialization\" in version \"dscinitialization.opendatahub.io/v1\"","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/controllers/dscinitialization.(*DSCInitializationReconciler).reconcileDefaultNetworkPolicy\n\t/workspace/controllers/dscinitialization/utils.go:202\ngithub.com/opendatahub-io/opendatahub-operator/v2/controllers/dscinitialization.(*DSCInitializationReconciler).createOdhNamespace\n\t/workspace/controllers/dscinitialization/utils.go:121\ngithub.com/opendatahub-io/opendatahub-operator/v2/controllers/dscinitialization.(*DSCInitializationReconciler).Reconcile\n\t/workspace/controllers/dscinitialization/dscinitialization_controller.go:180\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}

{"level":"error","ts":"2024-08-28T07:15:34Z","logger":"opendatahub.controllers.DSCInitialization","msg":"failed applying service mesh resources","error":"2 errors occurred:\n\t* failed applying FeatureHandler features. cause: 1 error occurred:\n\t* failed to get resource istio-system/data-science-smcp: no matches for kind \"ServiceMeshControlPlane\" in version \"maistra.io/v2\"\n\n\n\t* failed applying FeatureHandler features. cause: 1 error occurred:\n\t* 2 errors occurred:\n\t* failed to find Service Mesh Control Plane: no matches for kind \"ServiceMeshControlPlane\" in version \"maistra.io/v2\"\n\t* service mesh control plane is not ready\n\n\n\n\n\n","stacktrace":"github.com/opendatahub-io/opendatahub-operator/v2/controllers/dscinitialization.(*DSCInitializationReconciler).configureServiceMesh\n\t/workspace/controllers/dscinitialization/servicemesh_setup.go:44\ngithub.com/opendatahub-io/opendatahub-operator/v2/controllers/dscinitialization.(*DSCInitializationReconciler).Reconcile\n\t/workspace/controllers/dscinitialization/dscinitialization_controller.go:296\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/opt/app-root/src/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:235"}

Sounds weird

ykaliuta · 2024-08-28T10:29:28Z

/test rhods-operator-e2e

ykaliuta · 2024-09-02T17:05:20Z

Finally e2e passed :)

VaishnaviHire · 2024-09-13T18:09:09Z

@dhirajsb Should we go ahead with adding Model registry as component for 2.14?

zdtsw · 2024-09-14T05:34:27Z

@dhirajsb Should we go ahead with adding Model registry as component for 2.14?

should we wait for a weeek before merge this PR?
we can have downstream nightly on pure webhook first and give odh nightly one week to test modelreg.

dhirajsb · 2024-09-16T17:47:11Z

I'll leave it up to platform team. We (MR and Dashboard teams) can test on nightly builds and 2.18 release this week.
@tonyxrmdavidson cc

ykaliuta · 2024-09-18T15:59:22Z

/test rhods-operator-e2e

ykaliuta · 2024-09-18T18:04:58Z

I see the operator pod in the artefacts https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/red-hat-data-services_rhods-operator/329/pull-ci-red-hat-data-services-rhods-operator-main-rhods-operator-e2e/1836434864258158592/artifacts/rhods-operator-e2e/gather-extra/artifacts/pods/redhat-ods-operator_redhat-ods-operator-controller-manager-6d8cff779-8zx5p_manager.log . Wondering what the test cannot detect

…tasciencecluster.opendatahub.io_datascienceclusters.yaml Co-authored-by: Wen Zhou <[email protected]>

…ahub.io_datascienceclusters.yaml Co-authored-by: Wen Zhou <[email protected]>

Co-authored-by: Wen Zhou <[email protected]>

dhirajsb

Added more changes to use rhoai MR namespace rhoai-model-registries and to enable testing model registry reconcile.

dhirajsb · 2024-09-25T19:38:10Z

components/modelregistry/modelregistry.go

+type ModelRegistry struct {
+	components.Component `json:""`
+
+	// Namespace for model registries to be installed, configurable only once when model registry is enabled, defaults to "odh-model-registries"


Suggested change

// Namespace for model registries to be installed, configurable only once when model registry is enabled, defaults to "odh-model-registries"

// Namespace for model registries to be installed, configurable only once when model registry is enabled, defaults to "rhoai-model-registries"

i think the reason we keep it as odh-* is to ensure the source code is the same for upstream and downstream. to only config it with different values from csv etc

that could be just revert to the previous version instead of all that suggestions.

config/samples/datasciencecluster_v1_datasciencecluster.yaml

dhirajsb · 2024-09-25T19:43:46Z

tests/e2e/helper_test.go

@@ -168,6 +168,11 @@ func setupDSCInstance(name string) *dscv1.DataScienceCluster {
 						ManagementState: operatorv1.Removed,
 					},
 				},
+				ModelRegistry: modelregistry.ModelRegistry{
+					Component: components.Component{
+						ManagementState: operatorv1.Removed,


Suggested change

ManagementState: operatorv1.Removed,

ManagementState: operatorv1.Managed,

Enable testing model registry reconcile in e2e tests.

we can keep it as Removed as other non-GA component,

Co-authored-by: Dhiraj Bokde <[email protected]>

VaishnaviHire · 2024-09-26T12:43:33Z

@dhirajsb @ykaliuta We probably need to update the api-docs here -

make generate manifests api-docs

ykaliuta · 2024-09-26T12:50:32Z

@dhirajsb @ykaliuta We probably need to update the api-docs here -
make generate manifests api-docs

So, it now

 % git grep odh-model-registries
components/modelregistry/modelregistry.go:      // Namespace for model registries to be installed, configurable only once when model registry is enabled, defaults to "odh-model-registries"
docs/upgrade-testing.md:      registriesNamespace: "odh-model-registries"

After api update it is

% git grep odh-model-registries   
components/modelregistry/modelregistry.go:      // Namespace for model registries to be installed, configurable only once when model registry is enabled, defaults to "odh-model-registries"
config/crd/bases/datasciencecluster.opendatahub.io_datascienceclusters.yaml:                          to "odh-model-registries"
docs/api-overview.md:| `registriesNamespace` _string_ | Namespace for model registries to be installed, configurable only once when model registry is enabled, defaults to "odh-model-registries" | rhoai-model-registries | MaxLength: 63 <br />Pattern: `^([a-z0-9]([-a-z0-9]*[a-z0-9])?)?$` <br /> |
docs/upgrade-testing.md:      registriesNamespace: "odh-model-registries"

I already lost what modelregistry want and were, so @dhirajsb , please update it up to your requirements.

odh-model-registries -> rhoai-model-registries like it is mentioned below: // +kubebuilder:default="rhoai-model-registries" Signed-off-by: Yauheni Kaliuta <[email protected]>

Signed-off-by: Yauheni Kaliuta <[email protected]>

ykaliuta · 2024-09-26T13:39:30Z

@dhirajsb @ykaliuta We probably need to update the api-docs here -
make generate manifests api-docs

forget my last comment. I updated the 2 entries I had if it's what is required (will revert, if not). But about the generated files, the tree is clean after the command.

VaishnaviHire · 2024-09-26T13:57:54Z

/hold

@dhirajsb If the changes look good to you, please add your lgtm

VaishnaviHire · 2024-09-26T13:57:59Z

/lgtm

dhirajsb

Need to change the model registry SOP doc url.

config/monitoring/prometheus/apps/prometheus-configs.yaml

Co-authored-by: Dhiraj Bokde <[email protected]>

openshift-ci · 2024-09-26T15:49:27Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ykaliuta, zdtsw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ykaliuta,zdtsw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

VaishnaviHire · 2024-09-26T17:10:54Z

/hold cancel

openshift-ci bot added the approved label Aug 27, 2024

ykaliuta added the rhoai-2.14 label Aug 27, 2024

ykaliuta force-pushed the rhoai-modelregistry branch from 83e3e39 to e02e335 Compare August 28, 2024 06:22

ykaliuta mentioned this pull request Aug 28, 2024

Backport ModelRegistry Component #212

Closed

ykaliuta force-pushed the rhoai-modelregistry branch 7 times, most recently from ad6b38e to cb6597e Compare August 29, 2024 22:44

openshift-merge-robot added the needs-rebase label Sep 1, 2024

ykaliuta force-pushed the rhoai-modelregistry branch from cb6597e to 9fbd31c Compare September 2, 2024 09:56

openshift-merge-robot removed the needs-rebase label Sep 2, 2024

ykaliuta force-pushed the rhoai-modelregistry branch from 9fbd31c to b9e8dc8 Compare September 2, 2024 10:57

VaishnaviHire requested a review from dhirajsb September 13, 2024 18:09

openshift-merge-robot added the needs-rebase label Sep 14, 2024

zdtsw added rhoai-2.14 and removed rhoai-2.14 labels Sep 17, 2024

ykaliuta force-pushed the rhoai-modelregistry branch from b9e8dc8 to 270971d Compare September 18, 2024 11:58

openshift-merge-robot removed the needs-rebase label Sep 18, 2024

ykaliuta force-pushed the rhoai-modelregistry branch from 270971d to 104e83c Compare September 18, 2024 13:03

dhirajsb and others added 5 commits September 25, 2024 12:35

Replace registriesNamespace to rhoai namespace in bundle/manifests/da…

c3fb212

…tasciencecluster.opendatahub.io_datascienceclusters.yaml Co-authored-by: Wen Zhou <[email protected]>

Replace registriesNamespace to rhoai namespace in config/crd/bases/da…

9741be2

…tasciencecluster.opendatahub.io_datascienceclusters.yaml Co-authored-by: Wen Zhou <[email protected]>

Use rhoai MR namespace in config/crd/bases/datasciencecluster.opendat…

b41258c

…ahub.io_datascienceclusters.yaml Co-authored-by: Wen Zhou <[email protected]>

Use rhoai MR namespace in docs/DESIGN.md

6543aee

Co-authored-by: Wen Zhou <[email protected]>

Use rhoai MR namespace in docs/api-overview.md

897343f

Co-authored-by: Wen Zhou <[email protected]>

dhirajsb requested changes Sep 25, 2024

View reviewed changes

openshift-ci bot assigned dhirajsb Sep 25, 2024

Update config/samples/datasciencecluster_v1_datasciencecluster.yaml

36c70e7

Co-authored-by: Dhiraj Bokde <[email protected]>

zdtsw mentioned this pull request Sep 26, 2024

[WIP] feat: add managed model registry prometheus job, metrics, and alering rules, fixes RHOAIENG-4273 #318

Closed

5 tasks

ykaliuta added 2 commits September 26, 2024 16:31

modelregistry: change default registries namespace in the comment

aa195fe

odh-model-registries -> rhoai-model-registries like it is mentioned below: // +kubebuilder:default="rhoai-model-registries" Signed-off-by: Yauheni Kaliuta <[email protected]>

docs/uprgade-testing: update default model registries namespace to rhoai

8a1c45c

Signed-off-by: Yauheni Kaliuta <[email protected]>

openshift-ci bot added the do-not-merge/hold label Sep 26, 2024

openshift-ci bot assigned VaishnaviHire Sep 26, 2024

openshift-ci bot added the lgtm label Sep 26, 2024

dhirajsb requested changes Sep 26, 2024

View reviewed changes

config/monitoring/prometheus/apps/prometheus-configs.yaml Outdated Show resolved Hide resolved

config/monitoring/prometheus/apps/prometheus-configs.yaml Outdated Show resolved Hide resolved

config/monitoring/prometheus/apps/prometheus-configs.yaml Outdated Show resolved Hide resolved

openshift-ci bot removed the lgtm label Sep 26, 2024

ykaliuta and others added 3 commits September 26, 2024 18:47

Update config/monitoring/prometheus/apps/prometheus-configs.yaml

311fab2

Co-authored-by: Dhiraj Bokde <[email protected]>

Update config/monitoring/prometheus/apps/prometheus-configs.yaml

33cce26

Co-authored-by: Dhiraj Bokde <[email protected]>

Update config/monitoring/prometheus/apps/prometheus-configs.yaml

7635cc3

Co-authored-by: Dhiraj Bokde <[email protected]>

zdtsw approved these changes Sep 26, 2024

View reviewed changes

openshift-ci bot added the lgtm label Sep 26, 2024

openshift-ci bot removed the do-not-merge/hold label Sep 26, 2024

openshift-merge-bot bot merged commit 166cc19 into red-hat-data-services:main Sep 26, 2024
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backport ModelRegistry Component #329

Backport ModelRegistry Component #329

ykaliuta commented Aug 27, 2024

ykaliuta commented Aug 28, 2024 •

edited

Loading

ykaliuta commented Aug 28, 2024

ykaliuta commented Sep 2, 2024

VaishnaviHire commented Sep 13, 2024

zdtsw commented Sep 14, 2024

dhirajsb commented Sep 16, 2024

ykaliuta commented Sep 18, 2024

ykaliuta commented Sep 18, 2024

dhirajsb left a comment

dhirajsb Sep 25, 2024

zdtsw Sep 26, 2024

ykaliuta Sep 26, 2024

dhirajsb Sep 25, 2024

dhirajsb Sep 25, 2024

zdtsw Sep 26, 2024

VaishnaviHire commented Sep 26, 2024

ykaliuta commented Sep 26, 2024

ykaliuta commented Sep 26, 2024

VaishnaviHire commented Sep 26, 2024 •

edited

Loading

VaishnaviHire commented Sep 26, 2024

dhirajsb left a comment

openshift-ci bot commented Sep 26, 2024

VaishnaviHire commented Sep 26, 2024

	// Namespace for model registries to be installed, configurable only once when model registry is enabled, defaults to "odh-model-registries"
	// Namespace for model registries to be installed, configurable only once when model registry is enabled, defaults to "rhoai-model-registries"

	ManagementState: operatorv1.Removed,
	ManagementState: operatorv1.Managed,

Backport ModelRegistry Component #329

Backport ModelRegistry Component #329

Conversation

ykaliuta commented Aug 27, 2024

ykaliuta commented Aug 28, 2024 • edited Loading

ykaliuta commented Aug 28, 2024

ykaliuta commented Sep 2, 2024

VaishnaviHire commented Sep 13, 2024

zdtsw commented Sep 14, 2024

dhirajsb commented Sep 16, 2024

ykaliuta commented Sep 18, 2024

ykaliuta commented Sep 18, 2024

dhirajsb left a comment

Choose a reason for hiding this comment

dhirajsb Sep 25, 2024

Choose a reason for hiding this comment

zdtsw Sep 26, 2024

Choose a reason for hiding this comment

ykaliuta Sep 26, 2024

Choose a reason for hiding this comment

dhirajsb Sep 25, 2024

Choose a reason for hiding this comment

dhirajsb Sep 25, 2024

Choose a reason for hiding this comment

zdtsw Sep 26, 2024

Choose a reason for hiding this comment

VaishnaviHire commented Sep 26, 2024

ykaliuta commented Sep 26, 2024

ykaliuta commented Sep 26, 2024

VaishnaviHire commented Sep 26, 2024 • edited Loading

VaishnaviHire commented Sep 26, 2024

dhirajsb left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Sep 26, 2024

VaishnaviHire commented Sep 26, 2024

ykaliuta commented Aug 28, 2024 •

edited

Loading

VaishnaviHire commented Sep 26, 2024 •

edited

Loading