Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI: Add Windows wheels for the free-threaded build #60146

Merged
merged 4 commits into from
Nov 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -85,4 +85,5 @@ pandas/tests/io/parser/data export-ignore

# Include cibw script in sdist since it's needed for building wheels
scripts/cibw_before_build.sh -export-ignore
scripts/cibw_before_test.sh -export-ignore
scripts/cibw_before_build_windows.sh -export-ignore
scripts/cibw_before_test_windows.sh -export-ignore
4 changes: 2 additions & 2 deletions .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -387,8 +387,8 @@ jobs:
- name: Build Environment
run: |
python --version
python -m pip install --upgrade pip setuptools wheel meson[ninja]==1.2.1 meson-python==0.13.1
python -m pip install --pre --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy cython
python -m pip install --upgrade pip setuptools wheel numpy meson[ninja]==1.2.1 meson-python==0.13.1
python -m pip install --pre --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple cython
python -m pip install versioneer[toml]
python -m pip install python-dateutil pytz tzdata hypothesis>=6.84.0 pytest>=7.3.2 pytest-xdist>=3.4.0 pytest-cov
python -m pip install -ve . --no-build-isolation --no-index --no-deps -Csetup-args="--werror"
Expand Down
18 changes: 0 additions & 18 deletions .github/workflows/wheels.yml
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,6 @@ jobs:
- buildplat: [ubuntu-22.04, pyodide_wasm32]
python: ["cp312", "3.12"]
cibw_build_frontend: 'build'
# TODO: Build free-threaded wheels for Windows
exclude:
- buildplat: [windows-2022, win_amd64]
python: ["cp313t", "3.13"]

env:
IS_PUSH: ${{ github.event_name == 'push' && startsWith(github.ref, 'refs/tags/v') }}
Expand Down Expand Up @@ -181,20 +177,6 @@ jobs:
shell: bash -el {0}
run: for whl in $(ls wheelhouse); do wheel unpack wheelhouse/$whl -d /tmp; done

# Testing on windowsservercore instead of GHA runner to fail on missing DLLs
- name: Test Windows Wheels
if: ${{ matrix.buildplat[1] == 'win_amd64' }}
shell: pwsh
run: |
$TST_CMD = @"
python -m pip install hypothesis>=6.84.0 pytest>=7.3.2 pytest-xdist>=3.4.0;
python -m pip install `$(Get-Item pandas\wheelhouse\*.whl);
python -c `'import pandas as pd; pd.test(extra_args=[`\"--no-strict-data-files`\", `\"-m not clipboard and not single_cpu and not slow and not network and not db`\"])`';
"@
# add rc to the end of the image name if the Python version is unreleased
docker pull python:${{ matrix.python[1] == '3.13' && '3.13-rc' || format('{0}-windowsservercore', matrix.python[1]) }}
docker run --env PANDAS_CI='1' -v ${PWD}:C:\pandas python:${{ matrix.python[1] == '3.13' && '3.13-rc' || format('{0}-windowsservercore', matrix.python[1]) }} powershell -Command $TST_CMD

- uses: actions/upload-artifact@v4
with:
name: ${{ matrix.python[0] }}-${{ matrix.buildplat[1] }}
Expand Down
2 changes: 2 additions & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
Expand Up @@ -65,3 +65,5 @@ graft pandas/_libs/include

# Include cibw script in sdist since it's needed for building wheels
include scripts/cibw_before_build.sh
include scripts/cibw_before_build_windows.sh
include scripts/cibw_before_test_windows.sh
15 changes: 7 additions & 8 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,13 @@ free-threaded-support = true
before-build = "PACKAGE_DIR={package} bash {package}/scripts/cibw_before_build.sh"

[tool.cibuildwheel.windows]
before-build = "pip install delvewheel && bash {package}/scripts/cibw_before_build.sh"
before-build = "pip install delvewheel && bash {package}/scripts/cibw_before_build_windows.sh"
before-test = "bash {package}/scripts/cibw_before_test_windows.sh"
test-command = """
set PANDAS_CI='1' && \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could the PANDAS_CI be set in the before-test so that this section can re-use the test-command?

Copy link
Contributor Author

@lysnikolaou lysnikolaou Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could do that, yes, but all cibuildwheel custom configs include a different test-command. The Windows one has a slight difference as well. So I'm not sure if it's really worth it.

python -c "import pandas as pd; \
pd.test(extra_args=['--no-strict-data-files', '-m not clipboard and not single_cpu and not slow and not network and not db']);" \
"""
repair-wheel-command = "delvewheel repair -w {dest_dir} {wheel}"

[[tool.cibuildwheel.overrides]]
Expand All @@ -175,13 +181,6 @@ test-command = """
select = "*-musllinux*"
before-test = "apk update && apk add musl-locales"

[[tool.cibuildwheel.overrides]]
select = "*-win*"
# We test separately for Windows, since we use
# the windowsservercore docker image to check if any dlls are
# missing from the wheel
test-command = ""

[[tool.cibuildwheel.overrides]]
# Don't strip wheels on macOS.
# macOS doesn't support stripping wheels with linker
Expand Down
6 changes: 3 additions & 3 deletions scripts/cibw_before_build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ done

# TODO: Delete when there's a PyPI Cython release that supports free-threaded Python 3.13.
FREE_THREADED_BUILD="$(python -c"import sysconfig; print(bool(sysconfig.get_config_var('Py_GIL_DISABLED')))")"
if [[ $FREE_THREADED_BUILD == "True" ]]; then
if [[ $FREE_THREADED_BUILD == "True" ]]; then
python -m pip install -U pip
python -m pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy cython
python -m pip install ninja meson-python versioneer[toml]
python -m pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple cython
python -m pip install numpy ninja meson-python versioneer[toml]
fi
13 changes: 13 additions & 0 deletions scripts/cibw_before_build_windows.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Add 3rd party licenses, like numpy does
for file in $PACKAGE_DIR/LICENSES/*; do
cat $file >> $PACKAGE_DIR/LICENSE
done

# TODO: Delete when there's a PyPI Cython release that supports free-threaded Python 3.13
# and a NumPy Windows wheel for the free-threaded build on PyPI.
FREE_THREADED_BUILD="$(python -c"import sysconfig; print(bool(sysconfig.get_config_var('Py_GIL_DISABLED')))")"
if [[ $FREE_THREADED_BUILD == "True" ]]; then
python -m pip install -U pip
python -m pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy cython
python -m pip install ninja meson-python versioneer[toml]
fi
5 changes: 5 additions & 0 deletions scripts/cibw_before_test_windows.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# TODO: Delete when there's a NumPy Windows wheel for the free-threaded build on PyPI.
FREE_THREADED_BUILD="$(python -c"import sysconfig; print(bool(sysconfig.get_config_var('Py_GIL_DISABLED')))")"
if [[ $FREE_THREADED_BUILD == "True" ]]; then
python -m pip install -i https://pypi.anaconda.org/scientific-python-nightly-wheels/simple numpy
fi
Loading