Remove build isolation in wheel builds

[jameslamb]

Description

Currently, RAPIDS projects build wheels with build isolation, like pip wheel ..

With build isolation, the build tool (in our case, pip wheel), creates a virtual environment and installs of the package's build-time dependencies into it. That virtual environment is created at a random location on each build, which means source files included from other wheels (like the rmm headers from the librmm wheel) are found at a different path on each build in CI.

Path changes lead to cache misses with sccache, and therefore longer-running and more memory-intensive builds.

As more of RAPIDS build-time dependencies are provided wheels (e.g. #33), this problem is going to get worse (see #33 (comment)).

This issue tracks the work of removing build isolation in RAPIDS wheel-building CI jobs.

Benefits of this work

Better sccache hit rate = faster, cheaper CI (contributes to #95).

Stronger guarantees that packages built in CI are actually installed at test time (i.e. would close #79).

Acceptance Criteria

• all RAPIDS C/C++ wheel builds in CI build with --no-build-isolation

Approach

Follow the approach from rapidsai/cuspatial#1473, and documented in #112.

• pure Python? build isolation
• Python + Cython? build isolation
• mostly C/C++? no build isolation

Notes

These can be done in any order.

Updates

Beta Give feedback

1. build-planning: document why we only use build isolation for non-Cython C++ builds (document why and when --no-build-isolation is used in wheel-building CI #112)
2. add notes about doing this to the other create-new-C++-wheels issues
3. cucim (build wheels without build isolation, print sccache stats in builds cucim#792)
4. cudf (build wheels without build isolation cudf#17088)
5. cugraph (nothing to do ... no non-Cython C/C++ deps delivered as wheels)
6. cuml (nothing to do ... no non-Cython C/C++ deps delivered as wheels)
7. cuopt (nothing to do ... no C/C++ build deps delivered as wheels)
8. cuspatial (build wheels without build isolation cuspatial#1473)
9. cuvs (nothing to do ... no non-Cython C/C++ deps delivered as wheels)
10. cuxfilter (remove unnecessary sccache configuration, reduce verbosity of wheel-building scripts cuxfilter#641)
11. dask-cuda (remove unnecessary cmake and sccache configuration dask-cuda#1400)
12. kvikio (nothing to do ... libkvikio does not have build deps delivered as wheels, kvikio is only Python + Cython)
13. pynvjitlink (nothing to do ... no C/C++ build deps delivered as wheels)
14. raft (nothing to do ... no non-Cython C/C++ deps delivered as wheels)
15. rmm (nothing to do ... librmm is header-only and rmm is only Python + Cython)
16. ucx-py (nothing to do ... only Python + Cython)
17. ucxx (build wheels without build isolation ucxx#301)

Do the GNN repos last, as some of the cugraph repos are currently being shuffled around. See e.g. rapidsai/cugraph-gnn#58 (comment).

GNN repos

Beta Give feedback

1. cugraph-gnn (nothing to do ... no non-Cython C/C++ deps delivered as wheels)
2. cugraph-ops (nothing to do ... no non-Cython C/C++ deps delivered as wheels)
3. wholegraph (nothing to do ... no non-Cython C/C++ deps delivered as wheels)

[jameslamb]

I've updated the description to reflect our decision from an offline conversation:

• pure Python? build isolation
• just Python + Cython? build isolation
• mostly C/C++? no build isolation

And proposed documenting that in the build-planning docs: #112

[vyasr]

We shouldn't need to do this for any of the packages that haven't switched to using C++ wheels from other libraries yet, i.e. anything that is still statically linking everything. cugraph, cuml, cuvs, and raft (that may be exhaustive, or I may have missed some) don't have relevant changes yet.

[jameslamb]

We shouldn't need to do this for ... cugraph, cuml, cuvs, and raft

Oh interesting. I'd expected that all of them were at least getting RMM headers from the librmm wheels, but looking around that appears not to be the case.

evidence of that (click me)

Looking at recent nightly wheel builds, it seems like all those projects you mentioned are getting rmm via CPM.

-- CPM: Adding package [email protected] (branch-24.12)

(pylibcugraph wheel build)

(cugraph wheel build)

(cuml wheel build)

(pylibraft wheel build)

cuvs is slightly different in that it gets it via CPM via raft's CMake, but the result is the same:

-- CPM: raft: Adding package [email protected] (branch-24.12)

(cuvs wheel build)

I'm comparing that to other projects like libcudf, which are using the librmm wheels and therefore have logs like this:

-- CPM: Using local package [email protected]

(cudf wheel build)

And any other build-time dependencies those projects are getting via wheels appear to only be reference in Cython code, which we've already said we don't want to use build isolation for (#108 (comment)).

I'll update the issue description here and add notes about using build isolation to the corresponding issues that track the work for delivering more C++ dependencies in those projects via wheels.

[vyasr]

I'd expected that all of them were at least getting RMM headers from the librmm wheels, but looking around that appears not to be the case.

Yeah we've been following a depth-first rather than breadth-first approach to migrating packages over to use C++ wheels i.e. instead of updating everything to use librmm wheels once they existed we switched kvikio to C++ wheels (using rmm), then cudf (using kvikio and rmm), etc. It makes more sense that way since there's very little benefit to doing this partially until you get to the leaf nodes in our tree anyway (cugraph/cuml using raft).

[jameslamb]

We shouldn't need to do this for any of the packages that haven't switched to using C++ wheels from other libraries yet

By this additional standard of "only use --no-build-isolation if at least 1 not-just-for-Cython C/C++ build-time dependency is delivered via wheels", we could skip all of these for the purpose of this issue, I think:

• cugraph (cugraph, pylibcugraph, nx-cugraph, cugraph-equivariant, cugraph-dgl, cugraph-pyg)
• cugraph-gnn (cugraph-pyg, cugraph-dgl, and pylibwholegraph)
• cugraph-ops (pylibcugraphops)
• cuml
• cuopt
• cuvs
• kvikio (libkvikio, kvikio)
• pynvjitlink
• raft (pylibraft, raft-dask)
• wholegraph (pylibwholegraph)

Some of those have e.g. rmm or pylibraft in build dependencies, but only for cimports in Cython code. None have build-time dependencies delivered via wheels that will otherwise show up on compile lines (e.g. libcudf, librmm).

Have I understood this correctly @vyasr ?

[vyasr]

Yup, that is correct. AFAIK right now we should really only need to make changes to cudf, cuspatial, and kvikio (rmm would need this if it had any dependencies, but since it's our root node it doesn't). For the other libraries we'll have to remember to set this up correctly when we get around to making the C++ wheels.

[jameslamb]

Great, thank you.

For the other libraries we'll have to remember to set this up correctly when we get around to making the C++ wheels.

Agreed. I'm going to fill out the remaining issues tracking each individual dynamic-wheels thing (e.g. https://github.com/rapidsai/rapids-wheels-planning/issues/66 for cugraph) with details about all the things that need to be done for those issues to be considered "done". Should be pretty straightforward and mostly copy-paste, and hopefully that'll make it easier for anyone to pick up the work.

AFAIK right now we should really only need to make changes to cudf, cuspatial, and kvikio

I think we want ucxx as well... libucxx wheels pull in librmm and libucx wheels at build time (pyproject.toml).

And I think we don't want kvikio, because it isn't actually pulling in any build dependencies via wheels except cmake and ninja (kvikio pyproject.toml).

[vyasr]

You're right about both ucxx and kvikio, thanks.

[jameslamb]

I've added notes about this to the "Approach" sections in all the rest of the C++ wheels issues, so I think this can now be closed.