Skip to content

Commit a10ab19

Browse files
dankingdanking
authored
fix: release-plz, uv, pyvortex; feat: ensure wheels build in CI (#1181)
Here's the exhaustive description of what went wrong for the sake of our future selves. Release-plz initially [failed](https://github.com/spiraldb/vortex/actions/runs/11600956759/job/32302591513) because I incorrectly elided some object_store features from the vortex-datafusion crate. We addressed that with #1168 / e5d1275 by pushing the depending code into pyvortex (where those features *are* enabled). I also ensured we also tested building with the default features (instead of `--all-features`). Release-plz [failed again](https://github.com/spiraldb/vortex/actions/runs/11615867277/job/32347476714) after merging PR #1168 because vortex-serde 0.13.0 was present in crates.io but unbuildable (due to the issue above). We yanked the all the 0.13.0 crates and released 0.13.1. Release-plz [failed again](https://github.com/spiraldb/vortex/actions/runs/11630848959/job/32391375914) because `pyarrow` was not installed in the environment in which we test the wheel. This is due to a difference in the behavior of `uv sync` and `rye sync`. In particular, when we transitioned from `rye` to `uv` we replaced ``` rye sync --no-lock ``` with: ``` uv sync --all-extras --dev -v ``` Rye sync recursively installs the dependencies of each workspace member. `uv sync` did successfully build the Rust component of pyvortex in order to [prepare the metadata of the vortex-array Python package](https://github.com/spiraldb/vortex/actions/runs/11630848959/job/32391375914#step:3:64), but it did not install any of the pyvortex dependencies. Every test file failed to [import the dev and normal dependencies](https://github.com/spiraldb/vortex/actions/runs/11630848959/job/32391375914#step:6:80). Moreover, the macOS build step failed because the maturin action tries to use the Homebrew Python to pip install `cffi`. Homebrew does not permit this. The Homebrew Python is "externally managed" just like a system Python on a Linux distro. --- While the 0.13.1 release was failing, another PR merged and kicked off a release-plz workflow. Unfortunately, [that second workflow](https://github.com/spiraldb/vortex/actions/runs/11630955081/job/32390985113) ran very quickly because the one which was failing had already built and published many of the packages. Basically this happened: - ... - Workflow for 0.13.1: vortex-fsst 0.13.1 is not published yet, I'll build it. - Workflow for f83a093: vortex-array 0.13.1 is published already, I'll do nothing. - Workflow for f83a093: ... does this very quickly for every package before vortex-fsst ... - Workflow for f83a093: vortex-fsst 0.13.1 is not published yet, I'll build it. - Workflow for 0.13.1: vortex-fsst is built. I will publish it now. Success. - Workflow for f83a093: vortex-fsst is built. I will publish it now. It already exists?!? PANIC I will fix this in a follow up PR. This PR only addresses the `uv` issues.
1 parent 9bf8cf2 commit a10ab19

13 files changed

+110
-423
lines changed

‎.github/workflows/ci.yml

+20-3
Original file line numberDiff line numberDiff line change
@@ -44,16 +44,33 @@ jobs:
4444

4545
- name: Doctest - PyVortex
4646
run: |
47-
make clean && make doctest
47+
uv run make clean && uv run make doctest
4848
working-directory: docs/
4949

5050
- name: Ensure docs build - PyVortex
51+
run: |
52+
uv run make html
53+
working-directory: docs/
54+
55+
- name: Ensure wheel and sdist can be built on Linux - PyVortex
56+
run: |
57+
# force linux compatibility to ensure there is exactly one wheel ergo no ambiguity in installation
58+
uv run maturin build --interpreter python3.10 --compatibility linux
59+
working-directory: pyvortex/
60+
61+
- name: Ensure wheel can run tests successfully
5162
run: |
5263
set -ex
5364
54-
(cd pyvortex && uv run maturin develop)
65+
echo creating a fresh venv which does *not* have the normal dependencies to ensure the wheel dependencies are correct
66+
67+
rm -rf .venv
68+
uv sync --only-dev
5569
56-
(cd docs && uv run make html)
70+
uv run pip install target/wheels/vortex_array-*.whl || { uv run pip debug --verbose ; exit 1 ; }
71+
uv run pip install pytest
72+
cd pyvortex/test
73+
uv run pytest
5774
5875
rust-docs:
5976
name: "Rust (docs)"

‎.github/workflows/docs.yml

+1-5
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,7 @@ jobs:
2020

2121
- name: build Python and Rust docs
2222
run: |
23-
set -ex
24-
25-
(cd pyvortex && uv run maturin develop)
26-
27-
(cd docs && uv run make full-html)
23+
uv run make -C docs full-html
2824
- name: commit python docs to gh-pages-bench
2925
run: |
3026
set -ex

‎.github/workflows/release-plz.yml

+14-11
Original file line numberDiff line numberDiff line change
@@ -75,32 +75,34 @@ jobs:
7575
rust-toolchain: ${{ steps.rust-toolchain.version }}
7676
working-directory: pyvortex
7777
target: x86_64
78-
args: --release --interpreter python3.11 --sdist
78+
args: --release --interpreter python3.10 --sdist
7979
- name: Build wheels - aarch64
8080
uses: PyO3/maturin-action@v1
8181
with:
8282
rust-toolchain: ${{ steps.rust-toolchain.version }}
8383
working-directory: pyvortex
8484
target: aarch64 # NB: aarch64 becomes arm64 in the wheel's platform tag.
85-
args: --release --interpreter python3.11
85+
args: --release --interpreter python3.10
8686
- name: Build wheels - universal2
8787
uses: PyO3/maturin-action@v1
8888
with:
8989
rust-toolchain: ${{ steps.rust-toolchain.version }}
9090
working-directory: pyvortex
9191
target: universal2-apple-darwin
92-
args: --release --interpreter python3.11
92+
args: --release --interpreter python3.10
9393
- name: test wheel
9494
run: |
9595
set -ex
96-
source .venv/bin/activate
96+
9797
ls -al target/wheels
9898
9999
echo removing linux_x86_64 if it exists because PyPI will reject the package if it is present
100100
rm -f target/wheels/*linux_x86_64.whl
101101
102-
pip install --no-deps --force-reinstall vortex-array --no-index --find-links target/wheels
103-
cd pyvortex/test && pytest
102+
uv run pip install --no-deps --force-reinstall vortex-array --no-index --find-links target/wheels
103+
104+
cd pyvortex/test
105+
uv run pytest
104106
- name: Upload wheels
105107
uses: actions/upload-artifact@v4
106108
with:
@@ -127,19 +129,20 @@ jobs:
127129
working-directory: pyvortex
128130
target: x86_64
129131
manylinux: auto
130-
args: --release --interpreter python3.11
132+
args: --release --interpreter python3.10
131133
- name: test wheel
132134
run: |
133135
set -ex
134-
source .venv/bin/activate
136+
135137
ls -al target/wheels
136138
137139
echo removing linux_x86_64 if it exists because PyPI will reject the package if it is present
138140
rm -f target/wheels/*linux_x86_64.whl
139141
140-
pip install --no-deps --force-reinstall vortex-array --no-index --find-links target/wheels
141-
pip install pytest
142-
cd pyvortex/test && pytest
142+
uv run pip install --no-deps --force-reinstall vortex-array --no-index --find-links target/wheels
143+
144+
cd pyvortex/test
145+
uv run pytest
143146
- name: Upload wheels
144147
uses: actions/upload-artifact@v4
145148
with:

‎.gitignore

+1-1
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ ipython_config.py
8888
# pyenv
8989
# For a library or package, you might want to ignore these files since the code is
9090
# intended to run in multiple environments; otherwise, check them in:
91-
.python-version
91+
# .python-version
9292

9393
# pipenv
9494
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.

‎.python-version

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
3.10

‎Cargo.toml

+1-1
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@ prettytable-rs = "0.10.0"
112112
prost = "0.13.0"
113113
prost-build = "0.13.0"
114114
prost-types = "0.13.0"
115-
pyo3 = { version = "0.22.2", features = ["extension-module", "abi3-py311"] }
115+
pyo3 = { version = "0.22.2", features = ["extension-module", "abi3-py310"] }
116116
pyo3-log = "0.11.0"
117117
rand = "0.8.5"
118118
rayon = "1.10.0"

‎docs/Makefile

+3-4
Original file line numberDiff line numberDiff line change
@@ -4,23 +4,22 @@
44
# You can set these variables from the command line, and also
55
# from the environment for the first two.
66
SPHINXOPTS ?= --fail-on-warning --keep-going
7-
SPHINXBUILD ?= uv run sphinx-build
7+
SPHINXBUILD ?= sphinx-build
88
SOURCEDIR = .
99
BUILDDIR = _build
1010

11+
.PHONY: Makefile help clean rust-html
12+
1113
# Put it first so that "make" without argument is like "make help".
1214
help:
1315
@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
1416

15-
.PHONY: help Makefile
16-
1717
# Catch-all target: route all unknown targets to Sphinx using the new
1818
# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS).
1919
%: Makefile
2020
@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
2121

2222
full-html: html rust-html
2323

24-
.PHONY: rust-html
2524
rust-html:
2625
cargo doc --no-deps --workspace --all-features --target-dir $(BUILDDIR)/html/rust

‎docs/README.md

+6
Original file line numberDiff line numberDiff line change
@@ -28,3 +28,9 @@ After building:
2828
```
2929
open pyvortex/_build/html/index.html
3030
```
31+
32+
## Python Doctests
33+
34+
```
35+
uv run make doctest
36+
```

‎docs/pyproject.toml

+4-1
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,14 @@ authors = []
66
dependencies = [
77
"pydata-sphinx-theme>=0.16.0",
88
"sphinx>=8.0.2",
9-
"pyvortex",
9+
"vortex-array",
1010
"sphinx-design>=0.6.1",
1111
]
1212
requires-python = ">= 3.10"
1313

1414
[tool.uv]
1515
managed = true
1616
package = false
17+
18+
[tool.uv.sources]
19+
vortex-array = { workspace = true }

‎pyproject.toml

+19-4
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ version = "0.1.0"
44
description = "Add your description here"
55
authors = [{ name = "Nicholas Gates", email = "[email protected]" }]
66
requires-python = ">= 3.10"
7+
dependencies = [
8+
"vortex-array",
9+
"docs",
10+
]
711

812
[build-system]
913
requires = ["hatchling"]
@@ -14,17 +18,28 @@ packages = ["dummy"] # Required for workspace project
1418

1519
[tool.uv]
1620
managed = true
21+
# Currently, all dev dependencies live in the root since uv doesn't have transitive dev dependencies.
22+
# See: https://github.com/astral-sh/uv/issues/7541
1723
dev-dependencies = [
18-
"pytest>=7.4.0",
19-
"pytest-benchmark>=4.0.0",
20-
"ruff>=0.1.11",
21-
"pip>=23.3.2",
24+
"duckdb>=1.1.2",
25+
"ipython>=8.26.0",
2226
"maturin>=1.7.2",
27+
"pandas>=2.2.3",
28+
"pip>=23.3.2",
29+
"polars>=1.9.0",
30+
"pyright>=1.1.385",
31+
"pytest-benchmark>=4.0.0",
32+
"pytest>=7.4.0",
33+
"ruff>=0.7.1",
2334
]
2435

2536
[tool.uv.workspace]
2637
members = ["pyvortex", "docs"]
2738

39+
[tool.uv.sources]
40+
vortex-array = { workspace = true }
41+
docs = { workspace = true }
42+
2843
[tool.ruff]
2944
line-length = 120
3045
extend-exclude = [".venv"]

‎pyvortex/pyproject.toml

-9
Original file line numberDiff line numberDiff line change
@@ -24,15 +24,6 @@ build-backend = "maturin"
2424

2525
[tool.uv]
2626
managed = true
27-
dev-dependencies = [
28-
"ipython>=8.26.0",
29-
"pandas>=2.2.3",
30-
"pip",
31-
"pyright>=1.1.385",
32-
"polars>=1.9.0",
33-
"duckdb>=1.1.2",
34-
"ruff>=0.7.1",
35-
]
3627

3728
[tool.maturin]
3829
python-source = "python"

‎pyvortex/python/vortex/encoding.py

+18-17
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,27 @@
11
from typing import TYPE_CHECKING, Any
22

3-
import pandas
43
import pyarrow
54

65
from ._lib import encoding as _encoding
76

8-
# HACK: monkey-patch a fixed implementation of the pd.ArrowDtype.type property accessor.
9-
# See https://github.com/pandas-dev/pandas/issues/60068 for more details
10-
11-
_old_ArrowDtype_type = pandas.ArrowDtype.type
12-
13-
14-
@property
15-
def __ArrowDtype_type_patched(self):
16-
if pyarrow.types.is_string_view(self.pyarrow_dtype):
17-
return str
18-
if pyarrow.types.is_binary_view(self.pyarrow_dtype):
19-
return bytes
20-
return _old_ArrowDtype_type(self)
21-
22-
23-
pandas.ArrowDtype.type = __ArrowDtype_type_patched
7+
try:
8+
import pandas
9+
except ImportError:
10+
pass
11+
else:
12+
# HACK: monkey-patch a fixed implementation of the pd.ArrowDtype.type property accessor.
13+
# See https://github.com/pandas-dev/pandas/issues/60068 for more details
14+
_old_ArrowDtype_type = pandas.ArrowDtype.type
15+
16+
@property
17+
def __ArrowDtype_type_patched(self):
18+
if pyarrow.types.is_string_view(self.pyarrow_dtype):
19+
return str
20+
if pyarrow.types.is_binary_view(self.pyarrow_dtype):
21+
return bytes
22+
return _old_ArrowDtype_type(self)
23+
24+
pandas.ArrowDtype.type = __ArrowDtype_type_patched
2425

2526

2627
if TYPE_CHECKING:

0 commit comments

Comments
 (0)