--use-pep517 appears to be significanly slower for small projects

[zmanji]

Description

I have observed a performance regression in building wheels with --use-pep517 vs --no-use-pep517

Expected behavior

No significant performance change between --use-pep517 and --no-use-pep517

pip version

21.1.2

Python version

3.7.3

OS

Debian

How to Reproduce

To reproduce:

1. git clone [email protected]:zmanji/git-squash.git
2. Checkout e714352b532f2fdfc57b23df9b7cf8dc0fed34d6
3. Install the benchmarking tool hyperfine

Run

hyperfine -w2 'pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --no-use-pep517' 'pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --use-pep517'

Observe that the only difference between to the two pip wheel commands is --no-use-pep517 vs --use-pep517. Also observe the in tree build feature is enabled.

Feel free to change -w2 to -w3 or higher to increase warmups

Output

With pip 21.1.2 and wheel 0.36.2

I see the following results

on macOS 11.4 with Python 3.9.4

Benchmark #1: pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --no-use-pep517
  Time (mean ± σ):      1.220 s ±  0.039 s    [User: 917.3 ms, System: 200.4 ms]
  Range (min … max):    1.191 s …  1.325 s    10 runs
 
Benchmark #2: pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --use-pep517
  Time (mean ± σ):      4.390 s ±  0.018 s    [User: 3.563 s, System: 0.618 s]
  Range (min … max):    4.360 s …  4.416 s    10 runs
 
Summary
  'pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --no-use-pep517' ran
    3.60 ± 0.12 times faster than 'pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --use-pep517'

on Debian Linux with Python 3.7.3

Benchmark #1: pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --no-use-pep517
  Time (mean ± σ):      2.036 s ±  0.213 s    [User: 1.763 s, System: 0.216 s]
  Range (min … max):    1.871 s …  2.529 s    10 runs
 
Benchmark #2: pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --use-pep517
  Time (mean ± σ):      6.877 s ±  0.267 s    [User: 6.134 s, System: 0.658 s]
  Range (min … max):    6.458 s …  7.327 s    10 runs
 
Summary
  'pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --no-use-pep517' ran
    3.38 ± 0.38 times faster than 'pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --use-pep517'

I know it is a small project so this might be an extreme case, but this appears to be unexpected.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

This is probably caused by build isolation (which is used in PEP 517 builds but not in the "legacy" code path, if I remember correctly). Does performance improve if you use --no-build-isolation?

(Please understand that I'm asking for diagnosis purposes - you most definitely should not recommend using --no-build-isolation simply as a way of speeding up builds).

[zmanji]

Adding --no-build-isolation closes the gap:

Benchmark #1: pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --no-use-pep517
  Time (mean ± σ):      1.622 s ±  0.086 s    [User: 1.170 s, System: 0.331 s]
  Range (min … max):    1.522 s …  1.776 s    10 runs
 
Benchmark #2: pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --use-pep517 --no-build-isolation
  Time (mean ± σ):      1.620 s ±  0.049 s    [User: 1.166 s, System: 0.325 s]
  Range (min … max):    1.546 s …  1.702 s    10 runs
 
Summary
  'pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --use-pep517 --no-build-isolation' ran
    1.00 ± 0.06 times faster than 'pip wheel --use-feature=in-tree-build -w dist/ --only-binary ":all:" --no-deps -r requirements.txt . --no-use-pep517'

Is this expected?

[pfmoore]

Yes. When building using PEP 517/518, the build requirements (which are usually specified in pyproject.toml, but which will default to setuptools' "legacy" backend if not specified) may not be installed in the user's current environment. So pip installs them into a virtual environment and builds with that - that obviously takes some extra time. --no-build-isolation skips that virtual environment build step, but the cost is that the build will fail if you don't have all of the necessary dependencies installed (and in extreme cases that may not even be possible, if two requirements depend on, for example, different versions of setuptools to build).

While it would be nice to avoid the cost of setting up a build environment when it's not needed, it's frankly not been a significant issue for anyone until this point, and it would be hard to do so reliably. The legacy build process did none of this - it just tried the build in the environment the project was being installed into, and if that failed, tough. Of course, for simpler projects like your example, this worked fine. But for more complex projects it was an issue.

[pradyunsg]

And we do want to improve in this area -- there's other issues tracking that (like #7294).

[edmorley]

@zmanji Does #11257 (in pip 22.2) make much of a difference to the benchmark results?

[ichard26]

As discussed earlier, this is expected behaviour. I'm going to close this as #9081 and #7294 are already tracking potential avenues to reduce the build environment provisioning overhead.