Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated and tested Seldon Core 1.15.0 #2326

Merged
merged 33 commits into from
Dec 1, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
371944c
Updated and tested Seldon Core 1.15.0
axsaucedo Nov 22, 2022
093e9b8
Added testing section
axsaucedo Nov 22, 2022
df88a5d
Added upgrading reference
axsaucedo Nov 22, 2022
43d5ab3
Added automated test
axsaucedo Nov 22, 2022
7d3c120
Added automated tests and documentation
axsaucedo Nov 22, 2022
1bee9af
Added upgrade documentation
axsaucedo Nov 23, 2022
8571418
Added description of product
axsaucedo Nov 23, 2022
969c855
Updated contrib to ensure group maps correctly
axsaucedo Nov 23, 2022
0fa1236
Added updated instructions for non-linux
axsaucedo Nov 23, 2022
a58a044
Added gh action for seldon test
axsaucedo Nov 25, 2022
c214ec2
Added gh action for seldon test
axsaucedo Nov 25, 2022
8efc369
Added description of product
axsaucedo Nov 25, 2022
adac982
Updating path for kind
axsaucedo Nov 25, 2022
1dbd88f
Updating path for kind
axsaucedo Nov 25, 2022
441da77
Updating path for kind
axsaucedo Nov 25, 2022
ab1f9d1
Added kind cluster and rebase
axsaucedo Nov 25, 2022
56a402e
Updated back to running kind script
axsaucedo Nov 25, 2022
5f73b6d
Fixed typo
axsaucedo Nov 25, 2022
c204f63
Updated to include latest manifests
axsaucedo Nov 25, 2022
5ff592a
Create namespace if not exists
axsaucedo Nov 25, 2022
e9dd11b
Adding certmanager
axsaucedo Nov 25, 2022
9ce1d9d
Added namespaces
axsaucedo Nov 25, 2022
28b0504
Added waiting for resources
axsaucedo Nov 25, 2022
b0fe651
Updating to path
axsaucedo Nov 25, 2022
e274072
Added wait for condition
axsaucedo Nov 25, 2022
dee4068
Added wait for condition
axsaucedo Nov 25, 2022
5d0d952
Added wait for condition
axsaucedo Nov 25, 2022
e139864
Added wait for condition
axsaucedo Nov 25, 2022
1adf573
Added wait for condition
axsaucedo Nov 25, 2022
c0fbcf6
Added 1.24 kind tests plus note in the readme
axsaucedo Nov 29, 2022
9d60727
Extending timeout of wait for
axsaucedo Nov 29, 2022
5cf6a0e
Update contrib/seldon/README.md
axsaucedo Nov 29, 2022
b2f1f79
Reverted update on readme
axsaucedo Nov 29, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .github/workflows/seldon_kind_test.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
name: Build & Apply Seldon manifests in KinD
on:
pull_request:
paths:
- contrib/seldon/**
Comment on lines +2 to +5
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be okay for now, but @kimwnasptd we should think about other ways to trigger /contrib tests in the future and not just trigger based on changes to the contrib/seldon directory.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was thinking the same thing! We could probably use the workflow_dispatch event as well in these workflows, to be able to manually trigger them https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_dispatch

Let's play around with this more once we have the components update their manifests and have some basic tests in place.


jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3

- name: Install KinD
run: ./tests/gh-actions/install_kind.sh

- name: Create KinD Cluster
run: kind create cluster --config tests/gh-actions/kind-cluster-1-24.yaml

- name: Install kustomize
run: ./tests/gh-actions/install_kustomize.sh

- name: Install Istio
run: ./tests/gh-actions/install_istio.sh

- name: Install cert-manager
run: ./tests/gh-actions/install_cert_manager.sh

- name: Build & Apply manifests
run: |
cd contrib/seldon/
make test
46 changes: 32 additions & 14 deletions contrib/seldon/Makefile
Original file line number Diff line number Diff line change
@@ -1,26 +1,44 @@

# Copy values.yaml from desired seldon-core release and modify
# If updating values.yaml
# certManager: true
# Istio.enabled: true
# istio gateway
# kubeflow: true

# Presently need to remove generated status field from resources.yaml file manually
SELDON_VERSION ?= 1.15.0

seldon-core-operator/base: clean-kustomize
mkdir -p seldon-core-operator/base
cd seldon-core-operator/base && helm template -f ../../values.yaml seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --namespace kubeflow --version 1.6.0 > resources.yaml
#kfctl strangeness means these variables don't always get resolved see https://github.com/kubeflow/manifests/issues/1660
#sed -i 's#cert-manager.io/inject-ca-from:.*#cert-manager.io/inject-ca-from: $$(CERTIFICATE_NAMESPACE)/$$(CERTIFICATE_NAME)#g' seldon-core-operator/base/resources.yaml
cd seldon-core-operator/base && helm template -f ../../values.yaml seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --namespace kubeflow --version ${SELDON_VERSION} > resources.yaml
sed -i 's#cert-manager.io/inject-ca-from:.*#cert-manager.io/inject-ca-from: kubeflow/seldon-serving-cert#g' seldon-core-operator/base/resources.yaml
sed -i "s#'seldon-webhook-service.kubeflow.svc.cluster.local'#"'$$(SERVICE_NAME).$$(SERVICE_NAMESPACE).svc.cluster.local#' seldon-core-operator/base/resources.yaml
sed -i "s#commonName: 'seldon-webhook-service.kubeflow.svc'#"'commonName: $$(SERVICE_NAME).$$(CERTIFICATE_NAMESPACE).svc#' seldon-core-operator/base/resources.yaml
sed -i "s#'seldon-webhook-service.kubeflow.svc'#"'$$(SERVICE_NAME).$$(SERVICE_NAMESPACE).svc#' seldon-core-operator/base/resources.yaml
sed -i "s#'seldon-webhook-service.kubeflow.svc.cluster.local'#"'seldon-webhook-service.$$(SERVICE_NAMESPACE).svc.cluster.local#' seldon-core-operator/base/resources.yaml
axsaucedo marked this conversation as resolved.
Show resolved Hide resolved
sed -i "s#commonName: 'seldon-webhook-service.kubeflow.svc'#"'commonName: seldon-webhook-service.$$(CERTIFICATE_NAMESPACE).svc#' seldon-core-operator/base/resources.yaml
sed -i "s#'seldon-webhook-service.kubeflow.svc'#"'seldon-webhook-service.$$(SERVICE_NAMESPACE).svc#' seldon-core-operator/base/resources.yaml
sed -i 's#kubeflow/kubeflow-gateway#$$(SERVICE_NAMESPACE)/kubeflow-gateway#' seldon-core-operator/base/resources.yaml
cp kustomization.tpl seldon-core-operator/base/kustomization.yaml
cp kustomizeconfig.tpl seldon-core-operator/base/kustomizeconfig.yaml

.PHONY:clean-kustomize
clean-kustomize:
rm -rf seldon-core-operator/base

test:
kimwnasptd marked this conversation as resolved.
Show resolved Hide resolved
# Create seldon namespace
kubectl create namespace seldon || echo "seldon namespace exists"
# Create kubeflow namespace if not exists
kubectl create namespace kubeflow || echo "kubeflow namespace exists"
# Creating resources
kustomize build seldon-core-operator/base | kubectl apply -n kubeflow -f -
# Waiting for resources
kubectl wait --for condition=Available -n kubeflow deployment/seldon-controller-manager
# Wait for manager resources to be created
sleep 5
# Create example deployment
kubectl apply -n seldon -f example.yaml
# Wait for manager resources to be created
sleep 5
kubectl get deploy -A
sleep 5
kubectl logs -n kubeflow deployment/seldon-controller-manager
sleep 5
kubectl get deploy -A
# Wait for deployment
kubectl wait --for condition=Available -n seldon deployment/echo-default-0-classifier --timeout=360s
# Remove deployment
kubectl delete -n seldon -f example.yaml
# Cleaning resources
kustomize build seldon-core-operator/base | kubectl delete -n kubeflow -f -
82 changes: 78 additions & 4 deletions contrib/seldon/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
# Seldon Kustomize
# Seldon Core

## Install Seldon Operator
[Seldon Core](https://github.com/SeldonIO/seldon-core/) is a framework to deploy your machine learning models on Kubernetes at scale.

# Requirements

* Kubernetes 1.18 - 1.24

Support for Kubernetes 1.25 is currently part of [SeldonIO/seldon-core#4172](https://github.com/SeldonIO/seldon-core/pull/4172)

## Install Seldon Core Operator

* The yaml assumes you will install in kubeflow namespace
* You need to have installed istio first
Expand All @@ -11,8 +19,74 @@ kustomize build seldon-core-operator/base | kubectl apply -n kubeflow -f -

## Updating

This kustomize spec was created from the seldon-core-operator helm chart with:
See [UPGRADE.md](UPGRADE.md)

## Testing

```
make test
axsaucedo marked this conversation as resolved.
Show resolved Hide resolved
```

# Overview

We can create a test model once the "Install Seldon Operator" is configured

```
# Create namespace for model
kubectl create namespace seldon
```

We can create an echo model with the following command:

```
make clean seldon-core-operator/base
kubectl apply -f - << ENDapiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
name: echo
namespace: seldon
spec:
predictors:
- name: default
replicas: 1
graph:
name: classifier
type: MODEL
componentSpecs:
- spec:
containers:
- image: seldonio/echo-model:1.15.0-dev
name: classifier
END
```

We can verify that model is running:

```
kubectl get pods -n seldon

NAME READY STATUS RESTARTS AGE
echo-default-0-classifier-679cb5fb68-qd4nm 2/2 Running 0 25m
```

Also we can verify that the correct virtualservice was created:

```
kubectl get virtualservice -n seldon

NAME GATEWAYS HOSTS AGE
echo ["kubeflow/kubeflow-gateway"] ["*"] 42m
```

Finally we can send a request (you will need to fetch the Dex Auth Token / Cookie):

```
export CLUSTER_IP=# Your cluster IP
export SESSION=# Your dex session

curl -H "Content-Type: application/json" -H "Cookie: authservice_session=${SESSION}" \
-d '{"data": {"ndarray":[[1.0, 2.0, 5.0]]}}' \
http://{CLUSTER_IP}/seldon/seldon/echo/api/v1.0/predictions

{"data":{"names":["t:0","t:1","t:2"],"ndarray":[[1.0,2.0,5.0]]},"meta":{"metrics":[{"key":"mycounter","type":"COUNTER","value":1},{"key":"mygauge","type":"GAUGE","value":100},{"key":"mytimer","type":"TIMER","value":20.2}],"requestPath":{"classifier":"seldonio/echo-model:1.15.0-dev"}}}
```

33 changes: 33 additions & 0 deletions contrib/seldon/UPGRADE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Upgrading Documentation
axsaucedo marked this conversation as resolved.
Show resolved Hide resolved

## Updating manifests

In order to update manifests make sure you are running the commands in linux.

If you are running in another OS, please make sure to update the Makefile commands.

You can refresh the configuration by running:

```
make seldon-core-operator/base
```

## Updating to specific version

Upgrading the version can be done by setting the `SELDON_VERSION` environment variable, such as:

```
# Set the desired version
export SELDON_VERSION=1.14.0

# Rebuild the kustomize base
make seldon-core-operator/base

# Run new manifests against cluster
kustomize build seldon-core-operator/base | kubectl apply -f -
```

## Instructions for breaking changes

The [core upgrading docs](https://docs.seldon.io/projects/seldon-core/en/latest/reference/upgrading.html.) provide step by step overview of breaking changes across minor and patch versions.

15 changes: 15 additions & 0 deletions contrib/seldon/example.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
name: echo
spec:
predictors:
- name: default
graph:
name: classifier
type: MODEL
componentSpecs:
- spec:
containers:
- image: seldonio/echo-model:1.15.0-dev
name: classifier
14 changes: 5 additions & 9 deletions contrib/seldon/kustomization.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -11,25 +11,21 @@ vars:
objref:
kind: Certificate
group: cert-manager.io
version: v1alpha2
version: v1
name: seldon-serving-cert # this name should match the one in certificate.yaml
fieldref:
fieldpath: metadata.namespace
- name: CERTIFICATE_NAME
objref:
kind: Certificate
group: cert-manager.io
version: v1alpha2
version: v1
name: seldon-serving-cert # this name should match the one in certificate.yaml
- name: SERVICE_NAMESPACE # namespace of the service
objref:
kind: Service
kind: Deployment
version: v1
name: seldon-webhook-service
group: apps
name: seldon-controller-manager
fieldref:
fieldpath: metadata.namespace
- name: SERVICE_NAME
objref:
kind: Service
version: v1
name: seldon-webhook-service
17 changes: 0 additions & 17 deletions contrib/seldon/seldon-core-operator/base/kubeflow-edit-seldon.yaml

This file was deleted.

15 changes: 5 additions & 10 deletions contrib/seldon/seldon-core-operator/base/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@
# and emits as a YAML string
resources:
- resources.yaml
- kubeflow-edit-seldon.yaml

configurations:
- kustomizeconfig.yaml
Expand All @@ -12,25 +11,21 @@ vars:
objref:
kind: Certificate
group: cert-manager.io
version: v1alpha2
version: v1
name: seldon-serving-cert # this name should match the one in certificate.yaml
fieldref:
fieldpath: metadata.namespace
- name: CERTIFICATE_NAME
objref:
kind: Certificate
group: cert-manager.io
version: v1alpha2
version: v1
name: seldon-serving-cert # this name should match the one in certificate.yaml
- name: SERVICE_NAMESPACE # namespace of the service
objref:
kind: Service
kind: Deployment
version: v1
name: seldon-webhook-service
group: apps
name: seldon-controller-manager
fieldref:
fieldpath: metadata.namespace
- name: SERVICE_NAME
objref:
kind: Service
version: v1
name: seldon-webhook-service
Loading