Skip to content

Commit 292e375

Browse files
ScottToddmarbre
ScottTodd
and
marbre
authored
Add scripting and documentation for pushing releases to PyPI. (#519)
Progress on #400. This scripting allows us to publish .whl files from https://github.com/nod-ai/SHARK-Platform/releases/tag/dev-wheels to [PyPI](https://pypi.org/). Here are the basic steps: 1. Download wheels for a specific pre-release (e.g. `2.9.1rc20241114`) 2. Edit the versions in the downloaded wheels to remove the `rcYYYYMMDD` suffix 3. Build the `shark-ai` meta package using the versions in those whls (NOTE: currently this uses the versions _in the source tree_, see below) 4. Upload all wheels to PyPI Logs of this running: https://gist.github.com/ScottTodd/9c7418d5bbbebc8aea72a39bc2ac37b0 (to push 2.9.1 to PyPI) The new `pypi_deploy.sh` script is based on the similar script we maintain in IREE: https://github.com/iree-org/iree/blob/main/build_tools/python_deploy/pypi_deploy.sh . The `README.md` file is also forked from IREE. Known sharp edges with the publishing process: * This currently mixes local information (versions from the source tree where this is running) with remote information (versions from the nightly release packages). Publishing nightly meta packages will let us simplify and make this safer to run from arbitrary working trees * The local build of the shark-ai meta package copies _all_ files in `shark-ai/build_tools/wheelhouse/` to the working directory that gets sent to `twine upload *`. If there are preexisting files in that directory they will be published. * The step that downloads releases from GitHub uses `pip download` so it can leverage PyPI's version resolution logic, but I couldn't figure out a way to download 3.13t wheels without running a 3.13t python interpreter. I happen to have 3.13t installed on my system, but we shouldn't require that in the scripting for all release engineers. We could try using the `gh` tool as in IREE, if we properly filter the download to just the version we want: https://github.com/iree-org/iree/blob/9eaa4ef7d6b439d8c444b533beddd82146578e25/build_tools/python_deploy/pypi_deploy.sh#L69-L72 (that has one release per nightly, while here we have a single release shared between all nightlies) --------- Co-authored-by: Marius Brehler <[email protected]>
1 parent 61a211f commit 292e375

8 files changed

+205
-2
lines changed

‎build_tools/python_deploy/README.md

+48
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Python Deployment
2+
3+
These scripts assist with building Python packages and pushing them to
4+
[PyPI (the Python Package Index)](https://pypi.org/). See also
5+
6+
* The Python Packaging User Guide: <https://packaging.python.org/en/latest/>
7+
8+
## Overview
9+
10+
See comments in scripts for canonical usage. This page includes additional
11+
notes.
12+
13+
### Package building
14+
15+
These scripts build packages:
16+
17+
* [`/shark-ai/build_tools/build_linux_package.sh`](/shark-ai/build_tools/build_linux_package.sh)
18+
* [`/sharktank/build_tools/build_linux_package.sh`](/sharktank/build_tools/build_linux_package.sh)
19+
* [`/shortfin/build_tools/build_linux_package.sh`](/shortfin/build_tools/build_linux_package.sh)
20+
21+
### Version management
22+
23+
These scripts handle versioning across packages, including considerations like
24+
major, minor, and patch levels (`X.Y.Z`), as well as suffixes like
25+
`rc20241107`:
26+
27+
* [`compute_common_version.py`](./compute_common_version.py)
28+
* [`compute_local_version.py`](./compute_local_version.py)
29+
* [`promote_whl_from_rc_to_final.py`](./promote_whl_from_rc_to_final.py)
30+
* [`write_requirements.py`](./write_requirements.py)
31+
32+
### PyPI deployment
33+
34+
These scripts handle promoting nightly releases packages to stable and pushing
35+
to PyPI:
36+
37+
* [`promote_whl_from_rc_to_final.py`](./promote_whl_from_rc_to_final.py)
38+
* [`pypi_deploy.sh`](./pypi_deploy.sh)
39+
40+
Both of these scripts expect to have the dependencies from
41+
[`requirements-pypi-deploy.txt`](./requirements-pypi-deploy.txt) installed.
42+
This can be easily managed by using a Python virtual environment:
43+
44+
```bash
45+
python -m venv .venv
46+
source .venv/bin/activate
47+
python -m pip install -r ./requirements-pypi-deploy.txt
48+
```

‎build_tools/python_deploy/compute_common_version.py

100644100755
+6-2
Original file line numberDiff line numberDiff line change
@@ -6,8 +6,12 @@
66
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
77

88
# This scripts grabs the `X.Y.Z[.dev]` version identifier from the
9-
# sharktank and shortfin version files and computes the version
10-
# for the meta package.
9+
# 'sharktank' and 'shortfin' version files and computes the version
10+
# for the meta 'shark-ai' package.
11+
#
12+
# Usage:
13+
# ./compute_common_version.py --stable-release --write-json
14+
# cat ../../shark-ai/version_local.json
1115

1216
import argparse
1317
from pathlib import Path

‎build_tools/python_deploy/compute_local_version.py

100644100755
File mode changed.

‎build_tools/promote_whl_from_rc_to_final.py ‎build_tools/python_deploy/promote_whl_from_rc_to_final.py

File renamed without changes.

‎build_tools/python_deploy/pypi_deploy.sh

+126
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
#!/bin/bash
2+
3+
# Copyright 2024 Advanced Micro Devices, Inc.
4+
#
5+
# Licensed under the Apache License v2.0 with LLVM Exceptions.
6+
# See https://llvm.org/LICENSE.txt for license information.
7+
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
8+
9+
# This script promotes Python packages from nightly releases to PyPI.
10+
#
11+
# Prerequisites:
12+
# * You will need to have PyPI credentials set up. See
13+
# https://packaging.python.org/en/latest/tutorials/packaging-projects/#uploading-the-distribution-archives
14+
# * Install requirements, e.g. in a Python virtual environment (venv):
15+
# `pip install -r requirements-pypi-deploy.txt`
16+
# * Install python3.13t and install pip. On Ubuntu:
17+
# ```bash
18+
# sudo add-apt-repository ppa:deadsnakes
19+
# sudo apt-get update
20+
# sudo apt-get install python3.13-nogil
21+
# python3.13t -m ensurepip --upgrade
22+
# ```
23+
# * Choose a release candidate to promote from
24+
# https://github.com/nod-ai/SHARK-Platform/releases/tag/dev-wheels
25+
#
26+
# Usage:
27+
# ./pypi_deploy.sh 2.9.0rc20241108
28+
29+
set -euo pipefail
30+
31+
RELEASE="$1"
32+
33+
SCRIPT_DIR="$(dirname -- "$( readlink -f -- "$0"; )")";
34+
REPO_ROOT="$(cd "$SCRIPT_DIR"/../../ && pwd)"
35+
TMPDIR="$(mktemp --directory --tmpdir shark_platform_pypi_wheels.XXXXX)"
36+
ASSETS_PAGE="https://github.com/nod-ai/SHARK-Platform/releases/expanded_assets/dev-wheels"
37+
38+
# TODO: rewrite in Python?
39+
40+
function download_wheels() {
41+
echo ""
42+
echo "Downloading wheels for '${RELEASE}'..."
43+
44+
# sharktank
45+
python -m pip download sharktank==${RELEASE} \
46+
--no-deps --python-version 3.11 -f ${ASSETS_PAGE}
47+
48+
# shortfin
49+
python -m pip download shortfin==${RELEASE} \
50+
--no-deps --python-version 3.11 -f ${ASSETS_PAGE}
51+
python -m pip download shortfin==${RELEASE} \
52+
--no-deps --python-version 3.12 -f ${ASSETS_PAGE}
53+
python -m pip download shortfin==${RELEASE} \
54+
--no-deps --python-version 3.13 -f ${ASSETS_PAGE}
55+
python -m pip download shortfin==${RELEASE} \
56+
--no-deps --python-version 3.13 -f ${ASSETS_PAGE}
57+
# TODO: fetch 3.13t using the same `python` somehow
58+
# * https://pip.pypa.io/en/stable/cli/pip_download/
59+
# * https://py-free-threading.github.io/installing_cpython/
60+
# * https://pip.pypa.io/en/stable/installation/
61+
python3.13t -m pip download shortfin==${RELEASE} --no-deps -f ${ASSETS_PAGE}
62+
63+
# TODO: shark-ai meta package when it is published to nightlies
64+
65+
echo ""
66+
echo "Downloaded wheels:"
67+
ls
68+
}
69+
70+
function edit_release_versions() {
71+
echo ""
72+
echo "Editing release versions..."
73+
for file in *
74+
do
75+
${SCRIPT_DIR}/promote_whl_from_rc_to_final.py ${file} --delete-old-wheel
76+
done
77+
78+
echo "Edited wheels:"
79+
ls
80+
}
81+
82+
function upload_wheels() {
83+
# TODO: list packages that would be uploaded, pause, prompt to continue
84+
echo ""
85+
echo "Uploading wheels:"
86+
ls
87+
twine upload --verbose *
88+
}
89+
90+
function build_shark_ai_meta_package() {
91+
# TODO: download meta package from nightly releases instead of this
92+
# Be aware that nightly releases pin other dependencies via the
93+
# generated `requirements.txt` compared to stable releases.
94+
echo ""
95+
96+
# TODO: rework `write_requirements.py` to use the versions from the downloaded whls?
97+
echo "Computing local versions for sharktank and shortfin..."
98+
${SCRIPT_DIR}/compute_local_version.py ${REPO_ROOT}/sharktank
99+
${SCRIPT_DIR}/compute_local_version.py ${REPO_ROOT}/shortfin
100+
101+
echo "Computing common version for shark-ai meta package..."
102+
${SCRIPT_DIR}/compute_common_version.py --stable-release --write-json
103+
104+
echo "Writing requirements for shark-ai meta package..."
105+
${SCRIPT_DIR}/write_requirements.py
106+
107+
echo "Building shark-ai meta package..."
108+
${REPO_ROOT}/shark-ai/build_tools/build_linux_package.sh
109+
110+
# TODO: This is error-prone. We only want to publish the whl for this release.
111+
# Copy instead? Specify exact file name? Clear directory before building?
112+
mv ${REPO_ROOT}/shark-ai/build_tools/wheelhouse/* .
113+
}
114+
115+
function main() {
116+
echo "Changing into ${TMPDIR}"
117+
cd "${TMPDIR}"
118+
# TODO: check_requirements (using pip)
119+
120+
download_wheels
121+
edit_release_versions
122+
build_shark_ai_meta_package
123+
upload_wheels
124+
}
125+
126+
main

‎build_tools/requirements-pypi-deploy.txt ‎build_tools/python_deploy/requirements-pypi-deploy.txt

File renamed without changes.

‎build_tools/python_deploy/write_requirements.py

100644100755
File mode changed.

‎shark-ai/build_tools/build_linux_package.sh

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
#!/bin/bash
2+
3+
# Copyright 2024 Advanced Micro Devices, Inc.
4+
#
5+
# Licensed under the Apache License v2.0 with LLVM Exceptions.
6+
# See https://llvm.org/LICENSE.txt for license information.
7+
# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
8+
9+
# build_linux_package.sh
10+
#
11+
# Builds shark-ai Python package for Linux.
12+
#
13+
# Usage:
14+
# ./build_tools/build_linux_package.sh
15+
16+
set -xeu -o errtrace
17+
18+
THIS_DIR="$(cd $(dirname $0) && pwd)"
19+
REPO_ROOT="$(cd "$THIS_DIR"/../../ && pwd)"
20+
OUTPUT_DIR="${OUTPUT_DIR:-${THIS_DIR}/wheelhouse}"
21+
22+
python -m pip wheel --disable-pip-version-check --no-deps -v -w "${OUTPUT_DIR}" "${REPO_ROOT}/shark-ai"
23+
24+
wheel_output="$(echo "${OUTPUT_DIR}/shark_ai-"*".whl")"
25+
ls "${wheel_output}"

0 commit comments

Comments
 (0)