Helm installation from published packages (#158)

* Provide default installation example * Add website Make target Co-authored-by: Paolo Ambrosio - Sky UK <[email protected]>
sky-uk · Sep 2, 2022 · 06bdf9b · 06bdf9b
1 parent 6a3543b
commit 06bdf9b
Show file tree

Hide file tree

Showing 12 changed files with 143 additions and 38 deletions.
diff --git a/Makefile b/Makefile
@@ -212,6 +212,10 @@ docker-push-argo:
 	$(MAKE) -C argo/kfp-sdk docker-push
 	$(MAKE) -C argo/events docker-push
 
+##@ Website
+website:
+	$(MAKE) -C docs-gen
+
 package-all: docker-build docker-build-argo helm-package
 	$(MAKE) -C docs-gen
 

diff --git a/docs-gen/content/docs/getting-started/installation.md b/docs-gen/content/docs/getting-started/installation.md
@@ -7,14 +7,25 @@ We recommend the installation using Helm as it allows a declarative approach to
 
 This guide assumes you are familiar with [Helm](https://helm.sh/).
 
+## Prerequisites
+
+- KFP installed in [standalone mode](https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/). Default endpoints are used below.
+- Argo service account has been granted permissions to reference `ClusterWorkflowTemplate` resources ([example](https://github.com/sky-uk/kfp-operator/blob/master/docs-gen/includes/quickstart/resources/argo-rbac.yaml)).
+- GCS bucket or alternative storage location to store pipeline artifacts. Referenced as `{STORAGE_LOCATION}` below.
+- (Optional) [Argo-Events](https://argoproj.github.io/argo-events/installation/) for eventing support.
+
 ## Build and Install
 
-At the moment, you will have to build and publish the container images to run the operator manually.
-We are looking to publish images to a public repository in the near future.
-Please follow the [Development Guide](https://github.com/sky-uk/kfp-operator/blob/master/DEVELOPMENT.md#building-and-publishing) to publish these images.
+Create basic `values.yaml` with the following content:
+
+```yaml
+{{% readfile file="includes/quickstart/resources/values.yaml" %}}
+```
+
+Install the latest version of the operator
 
-```bash
-helm install -f values.yaml kfp-operator <YOUR_CHART_REPOSITORY>/kfp-operator
+```sh
+helm install oci://ghcr.io/kfp-operator/kfp-operator -f values.yaml
 ```
 
 ## Configuration Values

diff --git a/docs-gen/content/docs/reference/configuration.md b/docs-gen/content/docs/reference/configuration.md
@@ -5,12 +5,12 @@ weight: 1
 
 The Kubeflow Pipelines operator can be configured with the following parameters:
 
-| Parameter name      | Description                                                                                                                         | Example                                                |
-|---------------------|-------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------|
-| `defaultBeamArgs`   | Default Beam arguments to which the pipeline-defined ones will be added                                                             | <pre>- name: project<br/>  value: my-gcp-project</pre> |
-| `defaultExperiment` | Default Experiment name to be used for creating pipeline runs                                                                       | `Default`                                              |
-| `debug`             | Default debugging options                                                                                                           | See [Debugging](./debugging.md)                        |
-| `kfpEndpoint`       | The KFP endpoint available to the operator                                                                                          | `kubeflow-ui.kubeflow-pipelines:8080`                  |
-| `pipelineStorage`   | The storage location used by [TFX](https://www.tensorflow.org/tfx/guide/build_tfx_pipeline) to store pipeline artifacts and outputs | `gcs://kubeflow-pipelines-bucket`                      |
+| Parameter name      | Description                                                                                                                                                                                                                        | Example                                                |
+|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------|
+| `defaultBeamArgs`   | Default Beam arguments to which the pipeline-defined ones will be added                                                                                                                                                            | <pre>- name: project<br/>  value: my-gcp-project</pre> |
+| `defaultExperiment` | Default Experiment name to be used for creating pipeline runs                                                                                                                                                                      | `Default`                                              |
+| `debug`             | Default debugging options                                                                                                                                                                                                          | See [Debugging](./debugging.md)                        |
+| `kfpEndpoint`       | The KFP endpoint available to the operator                                                                                                                                                                                         | `kubeflow-ui.kubeflow-pipelines:8080`                  |
+| `pipelineStorage`   | The storage location used by [TFX (`pipeline-root`)](https://www.tensorflow.org/tfx/guide/build_tfx_pipeline) to store pipeline artifacts and outputs - this should be a top-level directory and not specific to a single pipeline | `gcs://kubeflow-pipelines-bucket`                      |
 
 An example can be found in the [local run configuration](../config/manager/controller_manager_config.yaml).
diff --git a/docs-gen/includes/quickstart/resources/argo-rbac.yaml b/docs-gen/includes/quickstart/resources/argo-rbac.yaml
@@ -0,0 +1,29 @@
+# Originally from https://github.com/argoproj/argo-workflows/blob/master/manifests/quick-start/base/cluster-workflow-template-rbac.yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: argo-clusterrole-binding
+subjects:
+  - kind: ServiceAccount
+    name: argo
+    namespace: kubeflow
+roleRef:
+  kind: ClusterRole
+  name: argo-role
+  apiGroup: rbac.authorization.k8s.io
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  annotations:
+  name: argo-role
+rules:
+  - apiGroups:
+      - argoproj.io
+    resources:
+      - clusterworkflowtemplates
+      - workflowtemplates
+    verbs:
+      - get
+      - list
+      - watch
diff --git a/docs-gen/includes/quickstart/resources/values.yaml b/docs-gen/includes/quickstart/resources/values.yaml
@@ -0,0 +1,13 @@
+fullnameOverride: kfp-operator
+manager:
+  argo:
+    serviceAccount: pipeline-runner
+  configuration:
+    defaultExperiment: Default
+    kfpEndpoint: http://ml-pipeline.kubeflow:8888
+    pipelineStorage: {STORAGE_LOCATION}
+# Uncomment below for eventing-support
+#eventsourceServer:
+#  create: true
+#  mlmdUrl: metadata-grpc-service.kubeflow:8080
+#  kfpUrl: ml-pipeline.kubeflow:8887
diff --git a/docs/docs/getting-started/index.xml b/docs/docs/getting-started/index.xml
@@ -53,11 +53,30 @@
 &lt;/ul></description></item><item><title>Docs: Installation</title><link>https://sky-uk.github.io/kfp-operator/docs/getting-started/installation/</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://sky-uk.github.io/kfp-operator/docs/getting-started/installation/</guid><description>
 &lt;p>We recommend the installation using Helm as it allows a declarative approach to managing Kubernetes resources.&lt;/p>
 &lt;p>This guide assumes you are familiar with &lt;a href="https://helm.sh/">Helm&lt;/a>.&lt;/p>
+&lt;h2 id="prerequisites">Prerequisites&lt;/h2>
+&lt;ul>
+&lt;li>KFP installed in &lt;a href="https://www.kubeflow.org/docs/components/pipelines/installation/standalone-deployment/">standalone mode&lt;/a>. Default endpoints are used below.&lt;/li>
+&lt;li>Argo service account has been granted permissions to reference &lt;code>ClusterWorkflowTemplate&lt;/code> resources (&lt;a href="https://github.com/sky-uk/kfp-operator/blob/master/docs-gen/includes/quickstart/resources/argo-rbac.yaml">example&lt;/a>).&lt;/li>
+&lt;li>GCS bucket or alternative storage location to store pipeline artifacts. Referenced as &lt;code>{STORAGE_LOCATION}&lt;/code> below.&lt;/li>
+&lt;li>(Optional) &lt;a href="https://argoproj.github.io/argo-events/installation/">Argo-Events&lt;/a> for eventing support.&lt;/li>
+&lt;/ul>
 &lt;h2 id="build-and-install">Build and Install&lt;/h2>
-&lt;p>At the moment, you will have to build and publish the container images to run the operator manually.
-We are looking to publish images to a public repository in the near future.
-Please follow the &lt;a href="https://github.com/sky-uk/kfp-operator/blob/master/DEVELOPMENT.md#building-and-publishing">Development Guide&lt;/a> to publish these images.&lt;/p>
-&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>helm install -f values.yaml kfp-operator &amp;lt;YOUR_CHART_REPOSITORY&amp;gt;/kfp-operator
+&lt;p>Create basic &lt;code>values.yaml&lt;/code> with the following content:&lt;/p>
+&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">fullnameOverride&lt;/span>: &lt;span style="color:#ae81ff">kfp-operator&lt;/span>
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">manager&lt;/span>:
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">argo&lt;/span>:
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">serviceAccount&lt;/span>: &lt;span style="color:#ae81ff">pipeline-runner&lt;/span>
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">configuration&lt;/span>:
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">defaultExperiment&lt;/span>: &lt;span style="color:#ae81ff">Default&lt;/span>
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">kfpEndpoint&lt;/span>: &lt;span style="color:#ae81ff">http://ml-pipeline.kubeflow:8888&lt;/span>
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">pipelineStorage&lt;/span>: {&lt;span style="color:#ae81ff">STORAGE_LOCATION}&lt;/span>
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Uncomment below for eventing-support&lt;/span>
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">#eventsourceServer:&lt;/span>
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># create: true&lt;/span>
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># mlmdUrl: metadata-grpc-service.kubeflow:8080&lt;/span>
+&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># kfpUrl: ml-pipeline.kubeflow:8887&lt;/span>
+&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Install the latest version of the operator&lt;/p>
+&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sh" data-lang="sh">&lt;span style="display:flex;">&lt;span>helm install oci://ghcr.io/kfp-operator/kfp-operator -f values.yaml
 &lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h2 id="configuration-values">Configuration Values&lt;/h2>
 &lt;p>Valid configuration options to override the &lt;a href="https://github.com/sky-uk/kfp-operator/blob/master/helm/kfp-operator/values.yaml
 ">Default &lt;code>values.yaml&lt;/code>&lt;/a> are:&lt;/p>