Pyodide improvements: version setting, standalone environments

[agriyakhetarpal]

Description

This PR updates the Pyodide build procedure (see #1456) that is enabled with CIBW_PLATFORM: "pyodide" (or with the --platform pyodide CLI equivalent) post the changes in pyodide/pyodide#4882, where pyodide/pyodide-build was unvendored from the main Pyodide repository to accommodate faster updates and fixes.

This means that the Pyodide version and pyodide-build are not going to be in sync going forward, and that the Pyodide xbuildenv to install must be inferred by the versions available to install by pyodide-build through a recently added pyodide xbuildenv search command, which prints out this table:

Tap to expand table

┌────────────┬────────────┬────────────┬───────────────────────────┬────────────┐
│ Version    │ Python     │ Emscripten │ pyodide-build             │ Compatible │
├────────────┼────────────┼────────────┼───────────────────────────┼────────────┤
│ 0.27.0a2   │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.4     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.3     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.2     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.1     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.0     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
└────────────┴────────────┴────────────┴───────────────────────────┴────────────┘

Alternatively, one may use pyodide xbuildenv search --all to return both compatible and non-compatible versions. This would, however, be better received as JSON (please see pyodide/pyodide-build#28).

---

Additionally, in this PR, support has been added for installing arbitrary Pyodide versions, or, more specifically, arbitrary versions for "Pyodide cross-build environments (xbuildenvs)" – though, only the ones that are supported for a given pyodide-build version. This has been implemented through an environment variable CIBW_PYODIDE_VERSION and an associated configuration variable in the schema (through a table implemented via pyodide/pyodide-build#26).

This PR uses the https://github.com/astral-sh/python-build-standalone project to retrieve runnable Python builds, so that we don't fall into compatibility issues where pyodide-build doesn't support a particular Python version. We fetch the appropriate version of Python required by the xbuildenv and pyodide-build and recommend users to check it out using the pyodide xbuildenv search command as needed.

The rationale behind this is that WebAssembly/Pyodide builds are already experimental, and it would be useful not to tie the available Pyodide version to the cibuildwheel version – this would be helpful for downstream projects (statsmodels/statsmodels#9343, scikit-image/scikit-image#7525, scikit-learn/scikit-learn#29791, and so on) to allow testing against Pyodide's alpha releases and/or for the case of greater reproducibility against Pyodide's older releases.

cc: @hoodmane and @ryanking13 for visibility

---

Suggested CHANGELOG entry

- ✨ Allow the use of a custom Pyodide version to target by the use of the `CIBW_PYODIDE_VERSION` environment variable or the `pyodide-version` configuration option. This is an experimental feature and users are advised to look at the [compatible Pyodide versions](https://github.com/pyodide/pyodide/blob/main/pyodide-cross-build-environments.json) according to the [`pyodide-build`](https://github.com/pyodide/pyodide-build) version, or install `pyodide-build` and run the `pyodide xbuildenv search --all` command to view a compatibility table.

[agriyakhetarpal]

The Windows test failures are unrelated. I'll try to fix them later in the day, but happy to step back if someone else does it before me, or wishes to.

[agriyakhetarpal]

I'll mark this as a draft until pyodide/pyodide-build#203 and pyodide/pyodide#5642 are resolved.

[henryiii]

This is passing, what's the status? :)

[agriyakhetarpal]

This is passing, what's the status? :)

It's ready to go in now! I released pyodide/pyodide-build@0.30.4 (release) with the new cross-build environment metadata URL and updated the constraints.

[agriyakhetarpal]

The ubuntu-py3.14 failure with Pyodide is unrelated; it came from a GitHub rate limit affecting _ensure_virtualenv. :( We should still be good to go, I'll re-trigger CI. :)

[agriyakhetarpal]

Surfacing this conversation as there are a lot of them getting hidden in the UI: #2002 (comment)

[jezdez]

I don't agree with adopting python-build-standalone for creating pyodide wheels files like this was implemented, y'all. It essentially puts an Astral project, which was barely maintained until a few months ago, in the supply chain of Pyodide wheels files. There was no way to solve this differently?

[agriyakhetarpal]

Hi @jezdez! Let me try to answer your questions:

There was no way to solve this differently?

The other way to do this was through having a manually hardcoded list of what Python versions to set up in cibuildwheel's GitHub Action, which also doesn't solve the "Pyodide version being coupled with cibuildwheel version inside cibuildwheel's code" problem. The crux of the issue is that cross-compilation for Pyodide wheels has the same limitations as crossenv; the host and the build Python versions have to match, so building from Python 3.12 Linux/macOS to Python 3.13/emscripten is not possible.

It essentially puts an Astral project, which was barely maintained until a few months ago, in the supply chain of Pyodide wheels files.

It is also possible to build Pyodide wheels manually without any of this tooling, all of which is cbuildwheel-specific. Running pyodide build with the appropriate Python version will build the same wheels without python-build-standalone. I know that this doesn't necessarily help here, but I just wanted to point that out.

[jezdez]

Hi @jezdez! Let me try to answer your questions:

There was no way to solve this differently?

The other way to do this was through having a manually hardcoded list of what Python versions to set up in cibuildwheel's GitHub Action, which also doesn't solve the "Pyodide version being coupled with cibuildwheel version inside cibuildwheel's code" problem. The crux of the issue is that cross-compilation for Pyodide wheels has the same limitations as crossenv; the host and the build Python versions have to match, so building from Python 3.12 Linux/macOS to Python 3.13/emscripten is not possible.

Given that other versions of Python are hardcoded (right?), was that really needed, though? This PR seems to trade security for ease of use

It essentially puts an Astral project, which was barely maintained until a few months ago, in the supply chain of Pyodide wheels files.

It is also possible to build Pyodide wheels manually without any of this tooling, all of which is cbuildwheel-specific. Running pyodide build with the appropriate Python version will build the same wheels without python-build-standalone. I know that this doesn't necessarily help here, but I just wanted to point that out.

That's beside my point, and not sufficient to answer my concerns. What I worry about is that cibuildwheel's focus is to systematically build wheels for the community, end-users of CBW won't care to use a different Python build and choosing the default for them like in this PR, puts a heavily patched 3rd party Python build into the supply chain of the wheels created. That carries a non-trivial supply chain security risk that is not in line with projects that are supposed to benefit the community.

I understand that python-build-standalone solves a specific problem, of course, I honestly wish its functionality would be upstreamed into CPython proper.

[agriyakhetarpal]

There was no way to solve this differently?

The other way to do this was through having a manually hardcoded list of what Python versions to set up in cibuildwheel's GitHub Action, which also doesn't solve the "Pyodide version being coupled with cibuildwheel version inside cibuildwheel's code" problem. The crux of the issue is that cross-compilation for Pyodide wheels has the same limitations as crossenv; the host and the build Python versions have to match, so building from Python 3.12 Linux/macOS to Python 3.13/emscripten is not possible.

Given that other versions of Python are hardcoded (right?), was that really needed, though? This PR seems to trade security for ease of use

Yes, though the idea here is that people would be able to set a newer Pyodide version and build wheels for it, right? cibuildwheel did not allow building and testing against pre-releases of Pyodide (where we will freeze the ABI changes going forward) prior to this PR.

It essentially puts an Astral project, which was barely maintained until a few months ago, in the supply chain of Pyodide wheels files.

It is also possible to build Pyodide wheels manually without any of this tooling, all of which is cbuildwheel-specific. Running pyodide build with the appropriate Python version will build the same wheels without python-build-standalone. I know that this doesn't necessarily help here, but I just wanted to point that out.

That's beside my point, and not sufficient to answer my concerns. What I worry about is that cibuildwheel's focus is to systematically build wheels for the community, end-users of CBW won't care to use a different Python build and choosing the default for them like in this PR, puts a heavily patched 3rd party Python build into the supply chain of the wheels created. That carries a non-trivial supply chain security risk that is not in line with projects that are supposed to benefit the community.

Thanks, I didn't know about the security differences between PBS and CPython. I think it would be nice to find a way to do this, but having a hardcoded list of Python versions would only help with cibuildwheel's GitHub Action, and not with running cibuildwheel through other means such as a pip-installed cibuildwheel. I don't see an immediate alternative but I can take a look. Still, the best way would be able to compile from patched sysconfig data from the Python version that does not match the host Python, and I think that is non-trivial to do as well.

PBS is also being used for iOS and Android wheel support in cibuildwheel, xref #1960.

I understand that python-build-standalone solves a specific problem, of course, I honestly wish its functionality would be upstreamed into CPython proper.

I agree!

[joerick]

Hi Jezdez. I understand your concerns. We are very mindful of security when incorporating dependencies into the supply chain.

But, to be honest, I don't feel that we're making things any less secure with this decision. Pyodide already has rather a lot of build dependencies (~40) required to build wheels. If I was looking at supply chain security vectors for Pyodide builds, I'd suggest that python-build-standalone was a relatively small fraction. And, I'd bet that one or more of those packages use uv for their package management.

So as the Python ecosystem stands today, for better or worse, Astral joins companies such as Github, Anaconda, Quansight as being foundational.

Having said all that, I'd agree that if we could eliminate the dependency (such as if CPython supplied relocatable builds), we should.

It should be be cross-platform, though. I think the only other option today would be to use Docker/OCI containers for builds... I can't see how we could do it on Linux otherwise.

[agriyakhetarpal]

Yes, since Pyodide builds are only advertised (and work/are officially supported, more or less), on Linux and macOS, we could improve the security posture here through reducing convenience for users and asking them to set up the Python version (according to what their Pyodide version they require) themselves – either that, or build inside Docker for Linux, and download and use the Framework builds for macOS.

Another way could be to restrict PBS usage only when the experimental pyodide-version is enabled, and proceed with a pinned version (the earlier variant) otherwise.