From d37aa211b41f714bf587e8155cfda606b359d5c3 Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Sat, 11 Sep 2021 16:53:52 +0100 Subject: [PATCH 1/6] Add a dedicated "Build system interface" reference section This reworks a substantial portion of our existing build system related documentation, putting it all in a single location. There are certain areas that have been rephrased or rewritten: - The setup.py interface is now considered an implementation detail. - The setup.py interface is explicitly noted as a legacy interface, due for removal. - The pyproject.toml interface now has clearer documentation on how it is used and how it operates. - Clearer separation of `--use-pep-517` and `--no-build-isolation`, to make the difference between the two clearer. --- docs/html/cli/pip.rst | 169 +----------------- docs/html/cli/pip_install.rst | 80 +-------- docs/html/cli/pip_wheel.rst | 55 +----- docs/html/reference/build-system/index.md | 108 +++++++++++ .../reference/build-system/pyproject-toml.md | 127 +++++++++++++ docs/html/reference/build-system/setup-py.md | 134 ++++++++++++++ docs/html/reference/index.md | 1 + docs/html/topics/caching.md | 9 +- docs/html/topics/repeatable-installs.md | 2 +- 9 files changed, 386 insertions(+), 299 deletions(-) create mode 100644 docs/html/reference/build-system/index.md create mode 100644 docs/html/reference/build-system/pyproject-toml.md create mode 100644 docs/html/reference/build-system/setup-py.md diff --git a/docs/html/cli/pip.rst b/docs/html/cli/pip.rst index d5b6b034c80..77aac009fa1 100644 --- a/docs/html/cli/pip.rst +++ b/docs/html/cli/pip.rst @@ -74,178 +74,11 @@ when decision is needed. *(a)abort* Abort pip and return non-zero exit status. -.. _`build-interface`: - Build System Interface ====================== -pip builds packages by invoking the build system. By default, builds will use -``setuptools``, but if a project specifies a different build system using a -``pyproject.toml`` file, as per :pep:`517`, pip will use that instead. As well -as package building, the build system is also invoked to install packages -direct from source. This is handled by invoking the build system to build a -wheel, and then installing from that wheel. The built wheel is cached locally -by pip to avoid repeated identical builds. - -The current interface to the build system is via the ``setup.py`` command line -script - all build actions are defined in terms of the specific ``setup.py`` -command line that will be run to invoke the required action. - -Setuptools Injection -~~~~~~~~~~~~~~~~~~~~ - -When :pep:`517` is not used, the supported build system is ``setuptools``. -However, not all packages use ``setuptools`` in their build scripts. To support -projects that use "pure ``distutils``", pip injects ``setuptools`` into -``sys.modules`` before invoking ``setup.py``. The injection should be -transparent to ``distutils``-based projects, but 3rd party build tools wishing -to provide a ``setup.py`` emulating the commands pip requires may need to be -aware that it takes place. - -Projects using :pep:`517` *must* explicitly use setuptools - pip does not do -the above injection process in this case. - -Build System Output -~~~~~~~~~~~~~~~~~~~ - -Any output produced by the build system will be read by pip (for display to the -user if requested). In order to correctly read the build system output, pip -requires that the output is written in a well-defined encoding, specifically -the encoding the user has configured for text output (which can be obtained in -Python using ``locale.getpreferredencoding``). If the configured encoding is -ASCII, pip assumes UTF-8 (to account for the behaviour of some Unix systems). - -Build systems should ensure that any tools they invoke (compilers, etc) produce -output in the correct encoding. In practice - and in particular on Windows, -where tools are inconsistent in their use of the "OEM" and "ANSI" codepages - -this may not always be possible. pip will therefore attempt to recover cleanly -if presented with incorrectly encoded build tool output, by translating -unexpected byte sequences to Python-style hexadecimal escape sequences -(``"\x80\xff"``, etc). However, it is still possible for output to be displayed -using an incorrect encoding (mojibake). - -Under :pep:`517`, handling of build tool output is the backend's responsibility, -and pip simply displays the output produced by the backend. (Backends, however, -will likely still have to address the issues described above). - -PEP 517 and 518 Support -~~~~~~~~~~~~~~~~~~~~~~~ - -As of version 10.0, pip supports projects declaring dependencies that are -required at install time using a ``pyproject.toml`` file, in the form described -in :pep:`518`. When building a project, pip will install the required -dependencies locally, and make them available to the build process. -Furthermore, from version 19.0 onwards, pip supports projects specifying the -build backend they use in ``pyproject.toml``, in the form described in -:pep:`517`. - -When making build requirements available, pip does so in an *isolated -environment*. That is, pip does not install those requirements into the user's -``site-packages``, but rather installs them in a temporary directory which it -adds to the user's ``sys.path`` for the duration of the build. This ensures -that build requirements are handled independently of the user's runtime -environment. For example, a project that needs a recent version of setuptools -to build can still be installed, even if the user has an older version -installed (and without silently replacing that version). - -In certain cases, projects (or redistributors) may have workflows that -explicitly manage the build environment. For such workflows, build isolation -can be problematic. If this is the case, pip provides a -``--no-build-isolation`` flag to disable build isolation. Users supplying this -flag are responsible for ensuring the build environment is managed -appropriately (including ensuring that all required build dependencies are -installed). - -By default, pip will continue to use the legacy (direct ``setup.py`` execution -based) build processing for projects that do not have a ``pyproject.toml`` file. -Projects with a ``pyproject.toml`` file will use a :pep:`517` backend. Projects -with a ``pyproject.toml`` file, but which don't have a ``build-system`` section, -will be assumed to have the following backend settings:: - - [build-system] - requires = ["setuptools>=40.8.0", "wheel"] - build-backend = "setuptools.build_meta:__legacy__" - -.. note:: - - ``setuptools`` 40.8.0 is the first version of setuptools that offers a - :pep:`517` backend that closely mimics directly executing ``setup.py``. - -If a project has ``[build-system]``, but no ``build-backend``, pip will also use -``setuptools.build_meta:__legacy__``, but will expect the project requirements -to include ``setuptools`` and ``wheel`` (and will report an error if the -installed version of ``setuptools`` is not recent enough). - -If a user wants to explicitly request :pep:`517` handling even though a project -doesn't have a ``pyproject.toml`` file, this can be done using the -``--use-pep517`` command line option. Similarly, to request legacy processing -even though ``pyproject.toml`` is present, the ``--no-use-pep517`` option is -available (although obviously it is an error to choose ``--no-use-pep517`` if -the project has no ``setup.py``, or explicitly requests a build backend). As -with other command line flags, pip recognises the ``PIP_USE_PEP517`` -environment veriable and a ``use-pep517`` config file option (set to true or -false) to set this option globally. Note that overriding pip's choice of -whether to use :pep:`517` processing in this way does *not* affect whether pip -will use an isolated build environment (which is controlled via -``--no-build-isolation`` as noted above). - -Except in the case noted above (projects with no :pep:`518` ``[build-system]`` -section in ``pyproject.toml``), pip will never implicitly install a build -system. Projects **must** ensure that the correct build system is listed in -their ``requires`` list (this applies even if pip assumes that the -``setuptools`` backend is being used, as noted above). - -.. _pep-518-limitations: - -**Historical Limitations**: - -* ``pip<18.0``: only supports installing build requirements from wheels, and - does not support the use of environment markers and extras (only version - specifiers are respected). - -* ``pip<18.1``: build dependencies using .pth files are not properly supported; - as a result namespace packages do not work under Python 3.2 and earlier. - -Future Developments -~~~~~~~~~~~~~~~~~~~ - -:pep:`426` notes that the intention is to add hooks to project metadata in -version 2.1 of the metadata spec, to explicitly define how to build a project -from its source. Once this version of the metadata spec is final, pip will -migrate to using that interface. At that point, the ``setup.py`` interface -documented here will be retained solely for legacy purposes, until projects -have migrated. - -Specifically, applications should *not* expect to rely on there being any form -of backward compatibility guarantees around the ``setup.py`` interface. - - -Build Options -~~~~~~~~~~~~~ - -The ``--global-option`` and ``--build-option`` arguments to the ``pip install`` -and ``pip wheel`` inject additional arguments into the ``setup.py`` command -(``--build-option`` is only available in ``pip wheel``). These arguments are -included in the command as follows: - -.. tab:: Unix/macOS - - .. code-block:: console - - python setup.py BUILD COMMAND - -.. tab:: Windows - - .. code-block:: shell - - py setup.py BUILD COMMAND - -The options are passed unmodified, and presently offer direct access to the -distutils command line. Use of ``--global-option`` and ``--build-option`` -should be considered as build system dependent, and may not be supported in the -current form if support for alternative build systems is added to pip. - +This is now covered in :doc:`../reference/build-system/index`. .. _`General Options`: diff --git a/docs/html/cli/pip_install.rst b/docs/html/cli/pip_install.rst index 6b4d3b1a2ba..d9af1b1e8bd 100644 --- a/docs/html/cli/pip_install.rst +++ b/docs/html/cli/pip_install.rst @@ -419,7 +419,7 @@ with other repeatability strategies. (rare) packages that use it will cause those dependencies to be downloaded by setuptools directly, skipping pip's hash-checking. If you need to use such a package, see :ref:`Controlling - setup_requires`. + setup_requires `. .. warning:: @@ -537,83 +537,10 @@ the project path. This is one advantage over just using ``setup.py develop``, which creates the "egg-info" directly relative the current working directory. -.. _`controlling-setup-requires`: - -Controlling setup_requires --------------------------- - -Setuptools offers the ``setup_requires`` `setup() keyword -`_ -for specifying dependencies that need to be present in order for the -``setup.py`` script to run. Internally, Setuptools uses ``easy_install`` -to fulfill these dependencies. - -pip has no way to control how these dependencies are located. None of the -package index options have an effect. - -The solution is to configure a "system" or "personal" `Distutils configuration -file -`_ to -manage the fulfillment. - -For example, to have the dependency located at an alternate index, add this: - -:: - - [easy_install] - index_url = https://my.index-mirror.com - -To have the dependency located from a local directory and not crawl PyPI, add this: - -:: - - [easy_install] - allow_hosts = '' - find_links = file:///path/to/local/archives/ - - Build System Interface ---------------------- -In order for pip to install a package from source, ``setup.py`` must implement -the following commands:: - - setup.py egg_info [--egg-base XXX] - setup.py install --record XXX [--single-version-externally-managed] [--root XXX] [--compile|--no-compile] [--install-headers XXX] - -The ``egg_info`` command should create egg metadata for the package, as -described in the setuptools documentation at -https://setuptools.readthedocs.io/en/latest/userguide/commands.html#egg-info-create-egg-metadata-and-set-build-tags - -The ``install`` command should implement the complete process of installing the -package to the target directory XXX. - -To install a package in "editable" mode (``pip install -e``), ``setup.py`` must -implement the following command:: - - setup.py develop --no-deps - -This should implement the complete process of installing the package in -"editable" mode. - -All packages will be attempted to built into wheels:: - - setup.py bdist_wheel -d XXX - -One further ``setup.py`` command is invoked by ``pip install``:: - - setup.py clean - -This command is invoked to clean up temporary commands from the build. (TODO: -Investigate in more detail when this command is required). - -No other build system commands are invoked by the ``pip install`` command. - -Installing a package from a wheel does not invoke the build system at all. - -.. _PyPI: https://pypi.org/ -.. _setuptools extras: https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies - +This is now covered in :doc:`../reference/build-system/index`. .. _`pip install Options`: @@ -906,3 +833,6 @@ Examples .. [1] This is true with the exception that pip v7.0 and v7.0.1 required quotes around specifiers containing environment markers in requirement files. + +.. _setuptools extras: https://setuptools.readthedocs.io/en/latest/userguide/dependency_management.html#optional-dependencies +.. _PyPI: https://pypi.org/ diff --git a/docs/html/cli/pip_wheel.rst b/docs/html/cli/pip_wheel.rst index c2a9543fc99..bf371f28506 100644 --- a/docs/html/cli/pip_wheel.rst +++ b/docs/html/cli/pip_wheel.rst @@ -28,60 +28,7 @@ Description Build System Interface ---------------------- -In order for pip to build a wheel, ``setup.py`` must implement the -``bdist_wheel`` command with the following syntax: - -.. tab:: Unix/macOS - - .. code-block:: shell - - python setup.py bdist_wheel -d TARGET - -.. tab:: Windows - - .. code-block:: shell - - py setup.py bdist_wheel -d TARGET - - -This command must create a wheel compatible with the invoking Python -interpreter, and save that wheel in the directory TARGET. - -No other build system commands are invoked by the ``pip wheel`` command. - -Customising the build -^^^^^^^^^^^^^^^^^^^^^ - -It is possible using ``--global-option`` to include additional build commands -with their arguments in the ``setup.py`` command. This is currently the only -way to influence the building of C extensions from the command line. For -example: - -.. tab:: Unix/macOS - - .. code-block:: shell - - python -m pip wheel --global-option bdist_ext --global-option -DFOO wheel - -.. tab:: Windows - - .. code-block:: shell - - py -m pip wheel --global-option bdist_ext --global-option -DFOO wheel - - -will result in a build command of - -:: - - setup.py bdist_ext -DFOO bdist_wheel -d TARGET - -which passes a preprocessor symbol to the extension build. - -Such usage is considered highly build-system specific and more an accident of -the current implementation than a supported interface. - - +This is now covered in :doc:`../reference/build-system/index`. Options ======= diff --git a/docs/html/reference/build-system/index.md b/docs/html/reference/build-system/index.md new file mode 100644 index 00000000000..13e6e321e43 --- /dev/null +++ b/docs/html/reference/build-system/index.md @@ -0,0 +1,108 @@ +(build-interface)= + +# Build System Interface + +When dealing with installable source distributions of a package, pip does not +directly handle the build process for the package. This responsibility is +delegated to "build backends" -- also known as "build systems". This means +that pip needs an interface, to interact with these build backends. + +There are two main interfaces that pip uses for these interactions: + +```{toctree} +:hidden: + +pyproject-toml +setup-py +``` + + +[`pyproject.toml` based](pyproject-toml) +: Standards-backed interface, that has build isolation and explicit declaration + of build dependencies. + +[`setup.py` based](setup-py) +: Legacy interface, that we're working to migrate users away from. Has no build + isolation and no good mechanisms to declare build dependencies. + + +Details on the individual interfaces can be found on their dedicated pages, +linked above. This document covers the nuances around which build system +interface pip will use for a project, as well as details that apply all +the build system interfaces that pip may use. + +## Determining which build system interface is used + +Currently, pip uses the `pyproject.toml` based build system interface, if a +`pyproject.toml` file exists. If not, the legacy build system interface is used. +The intention is to switch to using the `pyproject.toml` build system interface +unconditionally and to drop support for the legacy build system interface at +some point in the future. + +When performing a build, pip will mention which build system interface it is +using. Typically, this will take the form of a message like: + +```none +Building wheel for pip (PEP 517)... done +``` + +```none +Building wheel for pip (setup.py)... done +``` + +Here, "PEP 517" refers to `pyproject.toml` based builds and "setup.py" refers +to `setup.py` based builds. + +## Controlling which build system interface is used + +It is possible to control which build system interface is used by pip, using +the [`--use-pep517`](install_--use-pep517) / `--no-use-pep517` flags (and +corresponding environment variable: `PIP_USE_PEP517`). + +(controlling-setup_requires)= + +## Controlling `setup_requires` + +```{hint} +This is only relevant for projects that use setuptools as the build backend, +and use the `setup_requires` keyword argument in their setup.py file. +``` + +The `setup_requires` argument in `setup.py` is used to specify build-time +dependencies for a package. This has been superceded by `build-system.requires` +key in `pyproject.toml` files (per {pep}`518`). However, there are situations +where you might encounter a package that uses that argument (eg: the package +has not been updated to use the newer approach yet!). + +Older versions of setuptools (`< 52.0`) use `easy_install` to try to fulfill +those dependencies, which can result in weird failures such as attempting to +install incompatible package versions or improper installation of such +dependencies. + +Usually, the best solution for dealing with packages with `setup_requires` is +to install the packages listed in `setup_requires` beforehand, using a prior +`pip install` command. This is because there is no way to control how these +dependencies are located by `easy_install`, or how setuptools will invoke `pip` +using pip's command line options -- which makes it tricky to get things working +appropriately. + +If you wish to ensure that `easy_install` invocations do not reach out to PyPI, +you will need to configure its behaviour using a +[`distutils` configuration file][distutils-config]. Here are some examples: + +- To have the dependency located at an alternate index with `easy_install` + + ```ini + [easy_install] + index_url = https://my.index-mirror.com + ``` + +- To have the dependency located from a local directory and not crawl PyPI, add this: + + ```ini + [easy_install] + allow_hosts = '' + find_links = file:///path/to/local/archives/ + ``` + +[distutils-config]: https://docs.python.org/3/install/index.html#distutils-configuration-files diff --git a/docs/html/reference/build-system/pyproject-toml.md b/docs/html/reference/build-system/pyproject-toml.md new file mode 100644 index 00000000000..1f32fb7de79 --- /dev/null +++ b/docs/html/reference/build-system/pyproject-toml.md @@ -0,0 +1,127 @@ +# `pyproject.toml` + +```{versionadded} 10.0 + +``` + +Modern Python packages can contain a `pyproject.toml` file, first introduced in +{pep}`518` and later expanded in {pep}`517`, {pep}`621` and {pep}`660`. +This file contains build system requirements and information, which is used by +pip to build the package. + +## Build process + +The overall process for building a package is: + +- Create an isolated build environment. +- Populate the build environment with build dependencies. +- Generate the package's metadata, if necessary and possible. +- Generate a wheel for the package. + +The wheel can then be used to perform an installation, if necessary. + +### Build Isolation + +For building packages using this interface, pip uses an _isolated environment_. +That is, pip will install build-time Python dependencies in a temporary +directory which will be added to `sys.path` for the build commands. This ensures +that build requirements are handled independently of the user's runtime +environment. + +For example, a project that needs an older version of setuptools to build can +still be installed, even if the user has an newer version installed (and +without silently replacing that version). + +### Build-time dependencies + +Introduced in {pep}`518`, the `build-system.requires` key in the +`pyproject.toml` file is a list of requirement specifiers for build-time +dependencies of a package. + +```toml +[build-system] +requires = ["setuptools ~= 58.0", "cython ~= 0.29.0"] +``` + +It is also possible for a build backend to provide dynamically calculated +build dependencies, using {pep}`517`'s `get_requires_for_build_wheel` hook. This +hook will be called by pip, and dependencies it describes will also be installed +in the build environment. + +### Metadata Generation + +```{versionadded} 19.0 + +``` + +Once the build environment has been created and populated with build-time +dependencies, `pip` will usually need metadata about a package (name, version, +dependencies, and more). + +If {pep}`517`'s `prepare_metadata_for_build_wheel` hook is provided by the +build backend, that will be used to generate the packages' metadata. Otherwise, +a wheel will be generated (as described below) and the metadata contained +within such a wheel will be used. + +### Wheel Generation + +```{versionadded} 19.0 + +``` + +For generating a wheel, pip uses the {pep}`517`'s `build_wheel` hook that needs +to be provided by the build backend. The build backend may compile C/C++ code +at this point, depending on the package. Wheels generated using this mechanism +can be [cached](wheel-caching) for reuse, to speed up future installations. + +### Editable Installation + +This is currently not supported. However, this will be supported in the near +future, once {pep}`660` is implemented in pip. + +## Build output + +It is the responsibility of the build backend to ensure that the output is +in the correct encoding, as described in {pep}`517`. These likely involve +the same challenges as pip has for legacy builds. + +## Fallback Behaviour + +If a project does not have a `pyproject.toml` file containing a `build-system` +section, it will be assumed to have the following backend settings: + +```toml +[build-system] +requires = ["setuptools>=40.8.0", "wheel"] +build-backend = "setuptools.build_meta:__legacy__" +``` + +If a project has a `build-system` section but no `build-backend`, then: + +- It is expected to include `setuptools` and `wheel` as build requirements. An + error is reported if the installed version of `setuptools` is not recent + enough. + +- The `setuptools.build_meta:__legacy__` build backend will be used. + +## Disabling build isolation + +This can be disabled using the `--no-build-isolation` flag -- users supplying +this flag are responsible for ensuring the build environment is managed +appropriately, including ensuring that all required build-time dependencies are +installed, since pip does not manage build-time dependencies when this flag is +passed. + +## Historical notes + +As this feature was incrementally rolled out, there have been various notable +changes and improvements in it. + +- setuptools 40.8.0 is the first version of setuptools that offers a + {pep}`517` backend that closely mimics directly executing `setup.py`. +- Prior to pip 18.0, pip only supports installing build requirements from + wheels, and does not support the use of environment markers and extras (only + version specifiers are respected). +- Prior to pip 18.1, build dependencies using `.pth` files are not properly + supported; as a result namespace packages do not work under Python 3.2 and + earlier. diff --git a/docs/html/reference/build-system/setup-py.md b/docs/html/reference/build-system/setup-py.md new file mode 100644 index 00000000000..dc117318125 --- /dev/null +++ b/docs/html/reference/build-system/setup-py.md @@ -0,0 +1,134 @@ +# `setup.py` (legacy) + +Prior to the introduction of pyproject.toml-based builds (in {pep}`517` and +{pep}`518`), pip had only supported installing packages using `setup.py` files +that were built using {pypi}`setuptools`. + +The interface documented here is retained currently solely for legacy purposes, +until the migration to `pyproject.toml`-based builds can be completed. + +```{caution} +The arguments and syntax of the various invocations of `setup.py` made by +pip, are considered an implementation detail that is strongly coupled with +{pypi}`setuptools`. This build system interface is not meant to be used by any +other build backend, which should be based on the {doc}`pyproject-toml` build +system interface instead. + +Further, projects should _not_ expect to rely on there being any form of +backward compatibility guarantees around the `setup.py` interface. +``` + +## Build process + +The overall process for building a package is: + +- Generate the package's metadata. +- Generate a wheel for the package. + - If this fails and we're trying to install the package, attempt a direct + installation. + +The wheel can then be used to perform an installation, if necessary. + +### Metadata Generation + +As a first step, `pip` needs to get metadata about a package (name, version, +dependencies, and more). It collects this by calling `setup.py egg_info`. + +The `egg_info` command generates the metadata for the package, which pip can +then consume and proceed to gather all the dependencies of the package. Once +the dependency resolution process is complete, pip will proceed to the next +stage of the build process for these packages. + +### Wheel Generation + +When provided with a {term}`pypug:source distribution (or "sdist")` for a +package, pip will attempt to build a {term}`pypug:wheel`. Since wheel +distributions can be [cached](wheel-caching), this can greatly speed up future +installations for the package. + +This is done by calling `setup.py bdist_wheel` which requires the {pypi}`wheel` +package to be installed. + +If this wheel generation is successful (this can include compiling C/C++ code, +depending on the package), the generated wheel is added to pip's wheel cache +and will be used for this installation. The built wheel is cached locally +by pip to avoid repeated identical builds. + +If this wheel generation fails, pip will attempt a direct installation instead. + +### Direct Installation + +When all else fails, pip will invoke `setup.py install` to install a package +using setuptools' mechanisms to perform the installation. This is currently the +last-resort fallback for projects that cannot be built into wheels, and may not +be supported in the future. + +### Editable Installation + +For installing packages in "editable" mode +({ref}`pip install --editable `), pip will invoke +`setup.py develop`, which will use setuptools' mechanisms to perform an +editable/development installation. + +### Cleanup + +After attempting installation, pip may run `setup.py clean` to clean up build +artifacts from that setuptools has generated. + +## Setuptools Injection + +To support projects that directly use `distutils`, pip injects `setuptools` into +`sys.modules` before invoking `setup.py`. This injection should be transparent +to `distutils`-based projects. + +## Customising the build + +The `--global-option` and `--build-option` arguments to the `pip install` +and `pip wheel` inject additional arguments into the `setup.py` command +(`--build-option` is only available in `pip wheel`). + +```{attention} +The use of `--global-option` and `--build-option` is highly build-system +specific, and is considered more an accident of the current implementation than +a supported interface. It is documented here for completeness. These flags will +not be supported, once this build system interface is dropped. +``` + +These arguments are included in the command as follows: + +``` +python setup.py BUILD COMMAND +``` + +The options are passed unmodified, and presently offer direct access to the +distutils command line. For example: + +```{pip-cli} +$ pip wheel --global-option bdist_ext --global-option -DFOO wheel +``` + +will result in pip invoking: + +``` +setup.py bdist_ext -DFOO bdist_wheel -d TARGET +``` + +This passes a preprocessor symbol to the extension build. + +## Build Output + +Any output produced by the build system will be read by pip (for display to the +user if requested). In order to correctly read the build system output, pip +requires that the output is written in a well-defined encoding, specifically +the encoding the user has configured for text output (which can be obtained in +Python using `locale.getpreferredencoding`). If the configured encoding is +ASCII, pip assumes UTF-8 (to account for the behaviour of some Unix systems). + +Build systems should ensure that any tools they invoke (compilers, etc) produce +output in the correct encoding. In practice - and in particular on Windows, +where tools are inconsistent in their use of the "OEM" and "ANSI" codepages - +this may not always be possible. pip will therefore attempt to recover cleanly +if presented with incorrectly encoded build tool output, by translating +unexpected byte sequences to Python-style hexadecimal escape sequences +(`"\x80\xff"`, etc). However, it is still possible for output to be displayed +using an incorrect encoding (mojibake). diff --git a/docs/html/reference/index.md b/docs/html/reference/index.md index 13e57b2a478..8d690188d7a 100644 --- a/docs/html/reference/index.md +++ b/docs/html/reference/index.md @@ -6,5 +6,6 @@ interoperability standards that pip utilises/implements. ```{toctree} :titlesonly: +build-system/index requirements-file-format ``` diff --git a/docs/html/topics/caching.md b/docs/html/topics/caching.md index 0f4dfe9b90b..d584859c334 100644 --- a/docs/html/topics/caching.md +++ b/docs/html/topics/caching.md @@ -1,6 +1,7 @@ # Caching ```{versionadded} 6.0 + ``` pip provides an on-by-default caching, designed to reduce the amount of time @@ -26,6 +27,8 @@ While this cache attempts to minimize network activity, it does not prevent network access altogether. If you want a local install solution that circumvents accessing PyPI, see {ref}`Installing from local packages`. +(wheel-caching)= + ### Locally built wheels pip attempts to use wheels from its local wheel cache whenever possible. @@ -38,6 +41,10 @@ wheel using the package's build system. If the build is successful, this wheel is added to the cache and used in subsequent installs for the same package version. +Wheels built from source distributions provided to pip as a direct path (such +as `pip install .`) are not cached across runs, though they may be reused within +the same `pip` execution. + ```{versionchanged} 20.0 pip now caches wheels when building from an immutable Git reference (i.e. a commit hash). @@ -79,7 +86,7 @@ implementation detail and may change between any two versions of pip. ## Disabling caching -pip's caching behaviour is disabled by passing the ``--no-cache-dir`` option. +pip's caching behaviour is disabled by passing the `--no-cache-dir` option. It is, however, recommended to **NOT** disable pip's caching. Doing so can significantly slow down pip (due to repeated operations and package builds) diff --git a/docs/html/topics/repeatable-installs.md b/docs/html/topics/repeatable-installs.md index eca633d4ade..c6e8f9689e4 100644 --- a/docs/html/topics/repeatable-installs.md +++ b/docs/html/topics/repeatable-installs.md @@ -94,5 +94,5 @@ identical packages. Beware of the `setup_requires` keyword arg in {file}`setup.py`. The (rare) packages that use it will cause those dependencies to be downloaded by setuptools directly, skipping pip's protections. If you need to use such a -package, see {ref}`Controlling setup_requires `. +package, see {ref}`Controlling setup_requires `. ``` From 61b1e6905e146cd51b799160c01b66d346db2cdf Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Wed, 22 Sep 2021 06:58:29 +0100 Subject: [PATCH 2/6] :newspaper: --- news/10497.doc.rst | 1 + 1 file changed, 1 insertion(+) create mode 100644 news/10497.doc.rst diff --git a/news/10497.doc.rst b/news/10497.doc.rst new file mode 100644 index 00000000000..8c77c32ac66 --- /dev/null +++ b/news/10497.doc.rst @@ -0,0 +1 @@ +Create a "Build System Interface" reference section, for documenting how pip interacts with build systems. From 229808807b7f225da23935f96451005de15edc3e Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Wed, 29 Sep 2021 19:10:03 +0100 Subject: [PATCH 3/6] Update according to the PEP 660 implementation --- docs/html/reference/build-system/index.md | 11 ++++++++--- .../reference/build-system/pyproject-toml.md | 19 +++++++++++++++++-- 2 files changed, 25 insertions(+), 5 deletions(-) diff --git a/docs/html/reference/build-system/index.md b/docs/html/reference/build-system/index.md index 13e6e321e43..341f806d39b 100644 --- a/docs/html/reference/build-system/index.md +++ b/docs/html/reference/build-system/index.md @@ -43,15 +43,20 @@ When performing a build, pip will mention which build system interface it is using. Typically, this will take the form of a message like: ```none -Building wheel for pip (PEP 517)... done +Building wheel for pip (pyproject.toml)... done ``` ```none Building wheel for pip (setup.py)... done ``` -Here, "PEP 517" refers to `pyproject.toml` based builds and "setup.py" refers -to `setup.py` based builds. +The content in the brackets, refers to which build system interface is being +used. + +```{versionchanged} 21.3 +The output uses "pyproject.toml" instead of "PEP 517" to refer to be +`pyproject.toml` based build system interface. +``` ## Controlling which build system interface is used diff --git a/docs/html/reference/build-system/pyproject-toml.md b/docs/html/reference/build-system/pyproject-toml.md index 1f32fb7de79..326e188ed8a 100644 --- a/docs/html/reference/build-system/pyproject-toml.md +++ b/docs/html/reference/build-system/pyproject-toml.md @@ -76,8 +76,23 @@ can be [cached](wheel-caching) for reuse, to speed up future installations. ### Editable Installation -This is currently not supported. However, this will be supported in the near -future, once {pep}`660` is implemented in pip. +```{versionadded} 21.3 + +``` + +For performing editable installs, pip will use {pep}`660`'s +`build_wheel_for_editable` hook that needs to be provided by the build backend. +The wheels generated using this mechanism are not cached. + +```{admonition} Compatibility fallback +If this hook is missing on the build backend _and_ there's a `setup.py` file +in the project, pip will fallback to the legacy setup.py-based editable +installation. + +This is considered a stopgap solution until setuptools adds support for +{pep}`660`, at which point this functionality will be removed; following pip's +regular {ref}`deprecation policy `. +``` ## Build output From 25a31d49e5f53e19b0f7b5037893b6878a99861a Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Wed, 29 Sep 2021 19:57:43 +0100 Subject: [PATCH 4/6] Improve build system interface docs based on feedback Co-authored-by: Paul Moore --- docs/html/reference/build-system/index.md | 62 ++++++++++++------- .../reference/build-system/pyproject-toml.md | 26 ++++---- docs/html/reference/build-system/setup-py.md | 4 +- 3 files changed, 56 insertions(+), 36 deletions(-) diff --git a/docs/html/reference/build-system/index.md b/docs/html/reference/build-system/index.md index 341f806d39b..423fbaa2189 100644 --- a/docs/html/reference/build-system/index.md +++ b/docs/html/reference/build-system/index.md @@ -18,17 +18,17 @@ setup-py [`pyproject.toml` based](pyproject-toml) -: Standards-backed interface, that has build isolation and explicit declaration - of build dependencies. +: Standards-backed interface, that has explicit declaration and management of + build dependencies. [`setup.py` based](setup-py) -: Legacy interface, that we're working to migrate users away from. Has no build - isolation and no good mechanisms to declare build dependencies. +: Legacy interface, that we're working to migrate users away from. Has no good + mechanisms to declare build dependencies. Details on the individual interfaces can be found on their dedicated pages, linked above. This document covers the nuances around which build system -interface pip will use for a project, as well as details that apply all +interface pip will use for a project, as well as details that apply to all the build system interfaces that pip may use. ## Determining which build system interface is used @@ -60,9 +60,10 @@ The output uses "pyproject.toml" instead of "PEP 517" to refer to be ## Controlling which build system interface is used -It is possible to control which build system interface is used by pip, using -the [`--use-pep517`](install_--use-pep517) / `--no-use-pep517` flags (and -corresponding environment variable: `PIP_USE_PEP517`). +The [`--use-pep517`](install_--use-pep517) flag (and corresponding environment +variable: `PIP_USE_PEP517`) can be used to force all packages to build using +the `pyproject.toml` based build system interface. There is no way to force +the use of the legacy build system interface. (controlling-setup_requires)= @@ -74,22 +75,22 @@ and use the `setup_requires` keyword argument in their setup.py file. ``` The `setup_requires` argument in `setup.py` is used to specify build-time -dependencies for a package. This has been superceded by `build-system.requires` -key in `pyproject.toml` files (per {pep}`518`). However, there are situations -where you might encounter a package that uses that argument (eg: the package -has not been updated to use the newer approach yet!). - -Older versions of setuptools (`< 52.0`) use `easy_install` to try to fulfill -those dependencies, which can result in weird failures such as attempting to -install incompatible package versions or improper installation of such -dependencies. - -Usually, the best solution for dealing with packages with `setup_requires` is -to install the packages listed in `setup_requires` beforehand, using a prior -`pip install` command. This is because there is no way to control how these -dependencies are located by `easy_install`, or how setuptools will invoke `pip` -using pip's command line options -- which makes it tricky to get things working -appropriately. +dependencies for a package. This has been superceded by the +`build-system.requires` key in `pyproject.toml` files (per {pep}`518`). +However, there are situations where you might encounter a package that uses +`setup_requires` (eg: the package has not been updated to use the newer +approach yet!). + +If you control the package, consider adding a `pyproject.toml` file to utilise +the modern build system interface. That avoids invoking the problematic +behaviour by deferring to pip for the installations. + +For the end users, the best solution for dealing with packages with +`setup_requires` is to install the packages listed in `setup_requires` +beforehand, using a prior `pip install` command. This is because there is no +way to control how these dependencies are located by `easy_install`, or how +setuptools will invoke `pip` using pip's command line options -- which makes it +tricky to get things working appropriately. If you wish to ensure that `easy_install` invocations do not reach out to PyPI, you will need to configure its behaviour using a @@ -110,4 +111,17 @@ you will need to configure its behaviour using a find_links = file:///path/to/local/archives/ ``` +```{admonition} Historical context +`setuptools < 52.0` will use `easy_install` to try to fulfill those +dependencies, which can result in weird failures -- it does not understand +many of the modern Python packaging standards, and will usually attempt to +install incompatible package versions or to build packages incorrectly. It also +generates improper script wrappers, which don't do the right thing in many +situations. + +Newer versions of `setuptools` will use `pip` for these installations, but have +limited ability to pass through any command line arguments. This can also result +in weird failures and subtly-incorrect behaviour. +``` + [distutils-config]: https://docs.python.org/3/install/index.html#distutils-configuration-files diff --git a/docs/html/reference/build-system/pyproject-toml.md b/docs/html/reference/build-system/pyproject-toml.md index 326e188ed8a..ee93df034a7 100644 --- a/docs/html/reference/build-system/pyproject-toml.md +++ b/docs/html/reference/build-system/pyproject-toml.md @@ -6,7 +6,7 @@ Modern Python packages can contain a `pyproject.toml` file, first introduced in {pep}`518` and later expanded in {pep}`517`, {pep}`621` and {pep}`660`. -This file contains build system requirements and information, which is used by +This file contains build system requirements and information, which are used by pip to build the package. ## Build process @@ -46,7 +46,8 @@ requires = ["setuptools ~= 58.0", "cython ~= 0.29.0"] It is also possible for a build backend to provide dynamically calculated build dependencies, using {pep}`517`'s `get_requires_for_build_wheel` hook. This hook will be called by pip, and dependencies it describes will also be installed -in the build environment. +in the build environment. For example, newer versions of setuptools expose the +contents of `setup_requires` to pip via this hook. ### Metadata Generation @@ -69,10 +70,13 @@ within such a wheel will be used. ``` -For generating a wheel, pip uses the {pep}`517`'s `build_wheel` hook that needs -to be provided by the build backend. The build backend may compile C/C++ code -at this point, depending on the package. Wheels generated using this mechanism -can be [cached](wheel-caching) for reuse, to speed up future installations. +For generating a wheel, pip uses the {pep}`517` `build_wheel` hook that has +to be provided by the build backend. The build backend will generate a wheel, +which may involve compiling extension code written in C/C++ (or other +languages). + +Wheels generated using this mechanism can be [cached](wheel-caching) for reuse, +to speed up future installations. ### Editable Installation @@ -80,8 +84,8 @@ can be [cached](wheel-caching) for reuse, to speed up future installations. ``` -For performing editable installs, pip will use {pep}`660`'s -`build_wheel_for_editable` hook that needs to be provided by the build backend. +For performing editable installs, pip will use {pep}`660` +`build_wheel_for_editable` hook that has to be provided by the build backend. The wheels generated using this mechanism are not cached. ```{admonition} Compatibility fallback @@ -97,8 +101,8 @@ regular {ref}`deprecation policy `. ## Build output It is the responsibility of the build backend to ensure that the output is -in the correct encoding, as described in {pep}`517`. These likely involve -the same challenges as pip has for legacy builds. +in the correct encoding, as described in {pep}`517`. This likely involves +dealing with [the same challenges as pip has for legacy builds](build-output). ## Fallback Behaviour @@ -114,7 +118,7 @@ build-backend = "setuptools.build_meta:__legacy__" If a project has a `build-system` section but no `build-backend`, then: - It is expected to include `setuptools` and `wheel` as build requirements. An - error is reported if the installed version of `setuptools` is not recent + error is reported if the available version of `setuptools` is not recent enough. - The `setuptools.build_meta:__legacy__` build backend will be used. diff --git a/docs/html/reference/build-system/setup-py.md b/docs/html/reference/build-system/setup-py.md index dc117318125..722a05fa9ae 100644 --- a/docs/html/reference/build-system/setup-py.md +++ b/docs/html/reference/build-system/setup-py.md @@ -88,7 +88,7 @@ and `pip wheel` inject additional arguments into the `setup.py` command (`--build-option` is only available in `pip wheel`). ```{attention} -The use of `--global-option` and `--build-option` is highly build-system +The use of `--global-option` and `--build-option` is highly setuptools specific, and is considered more an accident of the current implementation than a supported interface. It is documented here for completeness. These flags will not be supported, once this build system interface is dropped. @@ -115,6 +115,8 @@ setup.py bdist_ext -DFOO bdist_wheel -d TARGET This passes a preprocessor symbol to the extension build. +(build-output)= + ## Build Output Any output produced by the build system will be read by pip (for display to the From d6d72172e8078f0b6efee6060947d27ba49b8978 Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Wed, 29 Sep 2021 20:02:07 +0100 Subject: [PATCH 5/6] Rephase historical note about `easy_install`/`setup_requires` It wasn't clear what "those dependencies" refers to. --- docs/html/reference/build-system/index.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/html/reference/build-system/index.md b/docs/html/reference/build-system/index.md index 423fbaa2189..354e79a3ff3 100644 --- a/docs/html/reference/build-system/index.md +++ b/docs/html/reference/build-system/index.md @@ -112,12 +112,12 @@ you will need to configure its behaviour using a ``` ```{admonition} Historical context -`setuptools < 52.0` will use `easy_install` to try to fulfill those -dependencies, which can result in weird failures -- it does not understand -many of the modern Python packaging standards, and will usually attempt to -install incompatible package versions or to build packages incorrectly. It also -generates improper script wrappers, which don't do the right thing in many -situations. +`setuptools < 52.0` will use `easy_install` to try to fulfill `setup_requires` +dependencies, which can result in weird failures -- `easy_install` does not +understand many of the modern Python packaging standards, and will usually +attempt to install incompatible package versions or to build packages +incorrectly. It also generates improper script wrappers, which don't do the +right thing in many situations. Newer versions of `setuptools` will use `pip` for these installations, but have limited ability to pass through any command line arguments. This can also result From 4d7d38a3c060cf05d7061d36f0b892f2fa0741ce Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Thu, 30 Sep 2021 06:07:24 +0100 Subject: [PATCH 6/6] Addresss spelling nit Co-authored-by: Paul Moore --- docs/html/reference/build-system/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/html/reference/build-system/index.md b/docs/html/reference/build-system/index.md index 354e79a3ff3..ed43fec3778 100644 --- a/docs/html/reference/build-system/index.md +++ b/docs/html/reference/build-system/index.md @@ -75,7 +75,7 @@ and use the `setup_requires` keyword argument in their setup.py file. ``` The `setup_requires` argument in `setup.py` is used to specify build-time -dependencies for a package. This has been superceded by the +dependencies for a package. This has been superseded by the `build-system.requires` key in `pyproject.toml` files (per {pep}`518`). However, there are situations where you might encounter a package that uses `setup_requires` (eg: the package has not been updated to use the newer