From eea7d483793c2b4f8c3aa20b1500535f7fbf50d4 Mon Sep 17 00:00:00 2001 From: Humair Khan Date: Tue, 30 Jul 2024 14:24:56 -0400 Subject: [PATCH] add dev docs for kfp/sdk (#11046) Signed-off-by: Humair Khan --- api/Makefile | 7 +++- kubernetes_platform/Makefile | 7 +++- sdk/CONTRIBUTING.md | 65 +++++++++++++++++++++++++++++++++--- 3 files changed, 73 insertions(+), 6 deletions(-) diff --git a/api/Makefile b/api/Makefile index 534a0e99c88..461250b5274 100644 --- a/api/Makefile +++ b/api/Makefile @@ -36,11 +36,16 @@ clean-go: rm -rf v2alpha1/go rm -f v2alpha1/google/rpc/status.proto -# Generate Python package. +# Build and locally install Python package. .PHONY: python python: v2alpha1/pipeline_spec.proto v2alpha1/google/rpc/status.proto python3 v2alpha1/python/generate_proto.py && cd v2alpha1/python && python3 setup.py bdist_wheel +# Build and locally install Python package using editable mode for development. +.PHONY: python-dev +python-dev: v2alpha1/pipeline_spec.proto v2alpha1/google/rpc/status.proto + python3 v2alpha1/python/generate_proto.py && cd v2alpha1/python && pip install -e . + # Delete all generated Python packages .PHONY: clean-python clean-python: diff --git a/kubernetes_platform/Makefile b/kubernetes_platform/Makefile index 4aa791039d9..c84a6400f93 100644 --- a/kubernetes_platform/Makefile +++ b/kubernetes_platform/Makefile @@ -35,11 +35,16 @@ clean-go: rm -rf kubernetes_platform/go -# Generate Python package. +# Build and locally install Python package. .PHONY: python python: proto/kubernetes_executor_config.proto python3 python/generate_proto.py && cd python && python3 setup.py bdist_wheel +# Build and locally install Python package using editable mode for development. +.PHONY: python-dev +python-dev: proto/kubernetes_executor_config.proto + python3 python/generate_proto.py && cd python && pip install -e .[dev] + # Delete all generated Python packages .PHONY: clean-python clean-python: diff --git a/sdk/CONTRIBUTING.md b/sdk/CONTRIBUTING.md index 116c7d79f26..01a523e7c8b 100644 --- a/sdk/CONTRIBUTING.md +++ b/sdk/CONTRIBUTING.md @@ -1,12 +1,69 @@ ## Contributing to the `kfp` SDK -KFP v2 for SDK is under development on the `master` branch. -To contribute to v1, please go to [sdk/release-1.8 branch](https://github.com/kubeflow/pipelines/tree/sdk/release-1.8) to submit your change. +For developing KFP v2 SDK, use the `master` branch. + +For developing KFP v1 SDK, use the [sdk/release-1.8 branch](https://github.com/kubeflow/pipelines/tree/sdk/release-1.8). For general contribution guidelines including pull request conventions, see [pipelines/CONTRIBUTING.md](https://github.com/kubeflow/pipelines/blob/master/CONTRIBUTING.md). -### Requirements -All development requirement versions are pinned in [requirements-dev.txt](https://github.com/kubeflow/pipelines/blob/master/sdk/python/requirements-dev.txt). +### Pre-requisites + +Clone the repo: + +```bash +git clone https://github.com/kubeflow/pipelines.git && cd pipelines +``` + +We suggest using a tool like [virtual env](https://docs.python.org/3/library/venv.html) or something similar for isolating +your environment and/or packages for you development environment. For this setup we'll stick with virtual env. + +For supported python versions, see the sdk [setup.py](https://github.com/kubeflow/pipelines/blob/master/sdk/python/setup.py). + +```bash +# optional, replace with your tool of choice +python -m venv env && source ./env/bin/activate +``` + +As with any Python package development, first ensure [wheels and setuptools](https://realpython.com/python-wheels/) are installed: + +```bash +python -m pip install -U pip wheel setuptools +``` + +All sdk development requirement versions are pinned in [requirements-dev.txt](https://github.com/kubeflow/pipelines/blob/master/sdk/python/requirements-dev.txt). + +```bash +pip install -r sdk/python/requirements-dev.txt +``` + +Install the SDK in development mode using the `--editable` or `-e` flag. This will allow you to implement and test changes iteratively. +Read more about it [here](https://setuptools.pypa.io/en/latest/userguide/development_mode.html). + +```bash +pip install -e sdk/python +``` + +The SDK also relies on a couple other python packages also found within KFP. +These consists of the [api proto package](https://github.com/kubeflow/pipelines/tree/master/api) and the kfp [kubernetes_platform](https://github.com/kubeflow/pipelines/tree/master/kubernetes_platform) package. + +For the proto code, we need protobuf-compiler to generate the python code. Read [here](../kubernetes_platform#dependencies) more about how to install this +dependency. + +You can install both packages either in development mode if you are planning to do active development on these, or simply do a regular install. + +To install the proto package: +```bash +pushd api +make generate python-dev # omit -dev for regular install +popd +``` + +To install the kubernetes_platform package: +```bash +pushd kubernetes_platform +make python-dev # omit -dev for regular install +popd +``` ### Testing We suggest running unit tests using [`pytest`](https://docs.pytest.org/en/7.1.x/). From the project root, the following runs all KFP SDK unit tests: