Add Rust support to build-wheel

[mhsmith]

Several packages are blocked until we do this – see the cross-references below.

[mhsmith]

We have two examples so far of Python/Rust packages being successfully built for Android:

• The existing tokenizers recipe, which does it by inserting a large block into the setup.py script using a patch. This doesn't easily generalize to other packages, and it hasn't been updated since early 2022, so it will require updates to work with the current version of build-wheel.

• The new cryptography recipe by @emanuele-f (#827, #657), which requires a large number of manual actions, including rebuilding Python because the Python/Rust bridge PyO3 requires a separate sysconfigdata file for each ABI (see comment in unpackage-target.sh).

A proper solution would work as follows:

• Alter the build and packging scripts of Python itself (in the target directory) to preserve all the sysconfigdata files.

• Make build-wheel recognize a meta.yaml build requirement of rust, just as it currently recognizes fortran, and respond by setting up the Rust compiler and environment variables as necessary.

[divyanshluthra]

sounds like a pretty big task

[IsakTheHacker]

@mhsmith In som other issue, I think you mentioned that the whole package repository is being deprecated. Is Rust support for the build wheel tool (this issue) still a feature that you are willing to receive PRs for or is all work being put here redundant since Python is going to get official Android support anyway?

[mhsmith]

I would still accept a PR that fixed this in build-wheel, because the content of that solution would be useful for future work with the upstream Python/Rust ecosystem anyway.

However, it's probably better if you don't work on the "preserve all the sysconfigdata files" step above, because I'll be revisiting the Python build process soon as part of adding Python 3.13 support (#1210).

[mhsmith]

In #1257 I'll update the packaging script so that the stdlib artifact for Python 3.13 will include the sysconfigdata files from all ABIs. After that, the next steps would be:

• Update build-wheel to use this artifact and make the files available to Rust.
• Work out how to deal with Python 3.12 and older, where multiple sysconfigdata files can't currently coexist because they all have the same name.

[emanuele-f]

Hi @mhsmith , I have huge updates for the cryptography module, sysconfigdata is not necessary anymore for it and integration is much easier. I just need to clean things up, I will open a PR tomorrow

[emanuele-f]

See #1261

[mhsmith]

@emanuele-f: Thanks very much, this is a huge step forward.

If anyone wants to try building one of the other Rust-based packages, you can now use the cryptography recipe as an example. I'll leave this issue open until we've had a few more successes.