From 78ab4c4a2c3cef66fa6c3578fa27a507f2f45c5b Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 13:48:37 -0700 Subject: [PATCH 01/35] PEP 759, External Wheel Hosting --- peps/pep-0759.rst | 472 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 472 insertions(+) create mode 100644 peps/pep-0759.rst diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst new file mode 100644 index 00000000000..958175c6c69 --- /dev/null +++ b/peps/pep-0759.rst @@ -0,0 +1,472 @@ +PEP: 759 +Title: External Wheel Hosting +Author: Barry Warsaw , + Ethan Smith +BDFL-Delegate: Donald Stufft +Status: Draft +Type: Standards Track +Topic: Packaging +Created: 19-Sep-2024 +Post-History: + +Abstract +======== + +This PEP proposes a mechanism by which projects hosted on `pypi.org +`__ can safely host wheel artifacts on external sites other +than PyPI. This PEP explicitly does *not* propose external hosting of +projects, packages, or their metadata. That functionality is already available +by externally hosting independent package indexes. Because this PEP only +provides a mechanism for projects to customize the download URL for specific +released wheel artifacts, dependency resolution as already implemented by +common installer tools such as `pip `__, and +`uv `__ does not need to change. + +This PEP defines what it means to be "safe" in this context, along with a new +package upload file format called a ``.rim`` file. It defines how ``.rim`` +files affect the metadata returned for a package's `Simple Repository API +`__ +in both HTML and JSON format, and how traditional wheels can easily be turned +into ``.rim`` files. + +Rationale +========= + +The Python Package Index, hosted at pypi.org imposes `default limits +`__ on upload artifact file size (100 MiB) and total +project size (10.0 GiB). Most projects can comfortably fit within these limits +during the lifetime of the project, through years of uploads. A few projects +have encountered these limits, and have been granted both `file size +exceptions `__ and `project size +exceptions `__, allowing them to +continue uploading new releases without having to take more drastic measures, +such as removing files which may potentially still be in use by consumers +(e.g. through version pins). + +A related workaround is the `"wheel stub" +`__ approach, which provides an +indirect link between PyPI and an external third party package index, where +such limitations can be avoided. Wheel stubs are `source distributions +`__, +a.k.a. "sdists" which utilize a :pep:`517` build backend that, instead of turning +source code into a binary wheel, performs some logic to calculate the URL for +an existing, externally hosted wheel to download. Technically speaking, this +satisfies the build backend contract, but the link between PyPI, the sdist, +and the externally hosted wheel isn't evident, easily discoverable, or +auditable. + +Historical context +================== + +In 2013, :pep:`438` proposed a "backward-compatible two-phase transition +process" to modify several aspects of release file hosting on PyPI. As this +PEP describes, PyPI originally supported only project and release +*registration* without also allowing for artifact file hosting. As such, most +projects hosted release file artifacts elsewhere. Artifact hosting was later +added, but the mix of externally and PyPI-hosted files lead to a wide range of +usability and potential security-related problems. PEP 438 was an attempt to +provide several facilities to allow external hosting while promoting a +PyPI-first hosting preference. + +PEP 438 was complex, with three different "hosting modes", ``rel`` metadata in +the simple HTML index pages to signify hosting locations, and a two-phase +transition plan affecting PyPI and installer tools. PEP 438 was ultimately +retracted in 2015 by :pep:`470`. In one respect, PEP 438 did succeed in: + + bringing about more people to utilize PyPI's repository features, an + altogether good thing given the global CDN powering PyPI providing speed + ups for a lot of people[...] + +Instead of external hosting, PEP 470 promoted the use of explicit multiple +repositories, providing full package indexing and artifact hosting, and +enabled through installer tool support, such as ``pip install +--extra-index-url`` allowing ``pip`` to essentially treat multiple +repositories as `one single global repository +`__ +for package installation resolution. Because this has been the blessed norm +for so many years, all Python package installation tools support querying +multiple indexes for dependency resolution. + +The problem with multiple indexes +================================= + +Why then does this PEP propose to allow a more limited form of external +hosting, and how does this proposal avoid the problems documented in PEP 470? + +One well-known problem that consolidating multiple indexes enables is +`dependency confusion attacks +`__ of +which Python *can* be particularly vulnerable, due to the algorithm that ``pip +install`` uses for resolving package dependencies and preferred versions. The +``uv`` tool addresses this by supporting an additional `index strategy +`__ option, +whereby users can select between, e.g. a ``pip``-compatible strategy, and a +more limited strategy that prevents such dependency confusion attacks. + +:pep:`708` provides additional background about dependency confusion attacks, +and takes a different approach to preventing them. At its core, PEP 708 allows +repository owners to indicate that projects track across different +repositories, which allows installers to determine how to treat the global +package namespace when combined across multiple repositories. PEP 708 has been +provisionally accepted, pending several required conditions as outlined in PEP +708, some of which may have an indeterminate future. As PEP 708 itself says, +this won't by itself solve dependency confusion attacks, but is one way to +provide enough information to installers to help minimize these attacks. + +While there can still be valid use cases for standing up a totally independent +package index (such as providing richer platform support for GPUs until a +fully formed `variant proposal +`__ +is accepted), this PEP takes a different, simpler approach and doesn't replace +any of the existing, proposed, or approved package index cooperation +specifications. + +This PEP also preserves the core purpose of PyPI, and allows it to +remain the traditional, canonical, centralized index of all Python +packages. + +It's not just about indexes +=========================== + +The problem *this* PEP solves is not just reduction or elimination of +dependency confusion attacks. There is another class of problem that a safe +external wheel hosting solution can mitigate: artifact size limits. + +PyPI imposes a `default artifact size limit `__ of +100MiB and a default overall `project size limit +`__ of 10GiB. Most packages and artifacts can +easily fit in these limits, even for packages containing binary extension modules for a +variety of platforms. A small, but important class of packages routinely exceed these +limits, requiring them to submit PyPI support tickets `requesting higher limits`_ It's not +necessarily difficult to get resolution on such exceptions, but it is a special process +that can take some time to resolve, and the criteria for granting such exceptions isn't +well documented. + +Reduce operational complexity +============================= + +Standing up and maintaining an entire package index can be a complex +operational solution, both time and resource intensive. This is especially +true if the main purpose of such an index is just to avoid file size +limitations. The external index approach also imposes a tricky UX on consumers +of projects on the external index, requiring them to understand how CLI +options such as ``--external-index-url`` work, along with the security +implications of such flags. It would be much easier for both producers and +consumers of large wheel packages to just stand up and maintain a simple web +server, capable of serving individual files with no more complex API than +``HTTP GET``. Such an interface is also easily cacheable or placed behind a +`CDN `__. Simple HTTP +servers are also much easier to audit for security purposes, easier to proxy, +and usually take much less resources to run, support, and maintain. Even +something like `Amazon S3 `__ could be used to +host external wheels. + +This PEP proposes an approach that favors such operational simplicity. + +Specification +============= + +A new type of uploadable file is defined, called a "RIM" (i.e. ``.rim``) file, so named to +evoke the image of a wheel with the tire removed, and backronym'd to "Remote Installable +Metadata" files. ``.rim`` files are easily derivable from wheels, and the process of +turning a wheel into a ``.rim`` is outlined below. The file name format exactly matches +the `wheel file naming format`_ specification, except that RIM files use the suffix +``.rim``. This means that all the tags used to discriminate ``.whl`` files also +distinguish between different ``.rim`` files, and thus can be used during dependency +resolution steps, exactly as ``.whl`` files are today. In this respect, ``.whl`` and +``.rim`` files are interchangeable. + +The content of a ``.rim`` file is *nearly* identical to ``.whl`` files, however ``.rim`` +files **MUST** contain only the ``.dist-info`` directory from a wheel. No other top-level +file or directory is allowed in the ``.rim`` zip file. The ``.dist-info`` directory +**MUST** contain a single additional file in addition to those `allowed`_ in a ``.whl`` +file's ``.dist-info`` directory: a file called ``EXTERNAL-HOSTING.json``. + +.. _file-format: + +This is a JSON file contains containing the following keys: + +``version`` + This is the file format version, which for this PEP **MUST** be ``1.0``. +``owner`` + This **MUST** name the PyPI organization owner of this externally hosted file, for + reasons which will be described in detail below. +``uri`` + This is a single URL naming the location of the physical ``.whl`` file hosted on an + external site. This URL **MUST** use the ``https`` scheme. +``size`` + This is an integer value describing the size in bytes of the physical ``.whl`` file. +``hashes`` + This is a dictionary of the format described in :pep:`694`, used to capture both the + `hash algorithm and hex encoded digests`_ of the physical ``.whl`` file, with the same + constraints as proposed in that PEP. Since these hashes are immutable once uploaded + to PyPI, they serve as a critical validation that the externally hosted wheel hasn't + been corrupted or compromised. + +The only effect of a ``.rim`` file is to change the download URL for the wheel artifact in +both the HTML and JSON interfaces in the `simple repository API`_. In the HTML page for a +package release, the ``href`` attribute **MUST** be the value of the ``uri`` key, +including a ``#=`` fragment. this hash fragment **MUST** be in +exactly the same format as described the :pep:`376` originated `signed wheel file format`_ +in the ``.dist-info/RECORD`` file. The exact same rules for selection of hash algorithm +and encoding is used here. + +Similarly in the `JSON response`_ the ``url`` key pointing to the download file must be +the value of the :ref:`uri ` key, and the ``hashes`` dictionary **MUST** be +included with values populated from the ``hashes`` dictionary provided above. + +In all other respects, a compliant package index should treat ``.rim`` files the same as +``.whl`` files, with some other minor exceptions as outlined below. For example, ``.rim`` +files can be `deleted `__ and yanked (:pep:`592`) just +like any ``.whl`` file, with the exact same semantics (i.e. deletions are permanent). When +a ``.rim`` is deleted, an index **MUST NOT** allow a matching ``.whl`` or ``.rim`` file to +be (re-)uploaded. + +Externally hosted wheels **MUST** be available before the corresponding ``.rim`` file is +uploaded to PyPI, otherwise a publishing race condition is introduced, although this +requirement **MAY** be relaxed for ``.rim`` files uploaded to a :pep:`694` staged release. + +Wheels can override RIMs +======================== + +Indexes **MUST** reject ``.rim`` files if a matching ``.whl`` file already exists with the +exact same file name tags. However, indexes **MAY** accept a ``.whl`` file if a matching +``.rim`` file exists, as long as that ``.rim`` file hasn't been deleted or yanked. This +allows uploaders to replace an externally hosted wheel file with an index hosted wheel +file, but the converse is prohibited. Since the default is to host wheels on the same +package index that contains the package metadata, it is not allowed to "downgrade" an +existing wheel file once uploaded. When a ``.whl`` replaces a ``.rim``, the index **MUST** +provide download URLs for the package using its own hosted file service. When uploading +the overriding ``.whl`` file, the package index **MUST** validate the hash from the +existing ``.rim`` file, and these hashes must match or the overriding upload **MUST** be +rejected. + +PyPI API bump unnecessary +========================= + +It's likely that the changes are backward compatible enough that a bump in the `PyPI +repository version`_ is not necessary. Since ``.rim`` files are essentially changes only +to the upload API, package resolvers and package installers can continue to function with +the APIs they've always supported. + +External hosting resiliency +=========================== + +One of the key concerns leading to PEP 438's revocation in PEP 470 was +potential user confusion when an external index disappeared. From PEP 470: + + This confusion comes down to end users of projects not realizing if a + project is hosted on PyPI or if it relies on an external service. This + often manifests itself when the external service is down but PyPI is + not. People will see that PyPI works, and other projects works, but this + one specific one does not. They oftentimes do not realize who they need to + contact in order to get this fixed or what their remediation steps are. + +While the problem of external wheel hosting service going down is not directly +solved by this PEP, several safeguards are in place to greatly reduce the +potential burden on PyPI administrators. + +This PEP thus proposes that: + +- External wheel hosting is only allowed for packages which are owned by + `organization accounts `__. + External hosting is an organization-wide setting. +- Organization accounts do not automatically gain the ability to externally + host wheels; this feature MUST be explicitly enabled by PyPI admins at their discretion. Since + this will not be a common request, we don't expect the overhead to be nearly + as burdensome as :pep:`541` resolutions, account recovery requests, or even + file/project size increase requests. External hosting requests would be + handled in the same manner as those requests, i.e. via the `PyPI GitHub + support tracker `__. +- Organization accounts requesting external wheel hosting **MUST** register their own + support contact URI, be it a ``mailto`` URI for a contact email address, or the URL to + the organization's support tracker. Such a contact URI is optional for organizations + which do not avail themselves of external wheel file hosting. + +Combined with the ``EXTERNAL-HOSTING.json`` file's ``owner`` key, this allows for +installer tools to unambiguously redirect any download errors away from the PyPI support +admins and squarely to the organization's support admins. + +While the exact mechanics of storing and retrieving this organization support +URL will be defined separately, for the sake of example, let's say a package +``foo`` externally hosts wheel files on ```https://foo.example.com`` +`__ and that host becomes unreachable. When an +installer tool tries to download and install the package ``foo`` wheel, the +download step will fail. The installer would then be able to query PyPI to +provide a useful error message to the end user: + +- The installer downloads the `.rim` file and reads the ``owner`` key from the + ``EXTERNAL-HOSTING.json`` file inside the ``.rim`` zip file. +- The installer queries PyPI for the support URI for the organization + owner of the externally hosted wheel. +- An informative error message would then be displayed, e.g.: + + The externally hosted wheel file ``foo-....whl`` could not be + downloaded. Please contact support@foo.example.com for help. Do not report + this to the PyPI administrators. + +Dismounting wheels +================== + +It is generally very easy to produce a ``.rim`` file from an existing ``.whl`` +file. This could be done efficiently by a :pep:`518` build backend with an +additional command line option, or a separate tool which takes a ``.whl`` file +as input and creates the associated ``.rim`` file. To complete the analogy, +the act of turning a ``.whl`` into a ``.rim`` is called "dismounting". The +steps such a tool would take are: + +- Accept as input the source ``.whl`` file, the organization owner of the + package, and URL at which the ``.whl`` will be hosted, and the support URI + to report download problems from. These could in fact be captured in the + ``pyproject.toml`` file, but that specification is out of scope for this + PEP. +- Unzip the ``.whl`` and create the ``.rim`` zip archive. +- Omit from the ``.rim`` file any path in the ``.whl`` that **isn't** rooted + at the ``.dist-info`` directory. +- Calculate the hash of the source ``.whl`` file. +- Add the ``EXTERNAL-HOSTING.json`` file containing the JSON keys and values as described + above, to the ``.rim`` archive. + +Changes to tools +================ + +Theoretically, installer tools shouldn't need any changes, since when they +have identified the wheel to download and install, they simply consult the +download URLs returned by PyPI's Simple API. In practice though, tools such as +``pip`` and ``uv`` may have constrained lists of hosts they will allow +downloads from, such as PyPI's own ``pythonhosted.org`` domain. + +In this case, such tools will need to relax those constraints, but the exact policy for +this is left to the installer tools themselves. Any number of approaches could be +implemented, such as downloading the ``.rim`` file and verifying the +``EXTERNAL-HOSTING.json`` metadata, or simply trusting the external downloads for any +wheel with a matching checksum. They could also query PyPI for the project's organization +owner and support URI before trusting the download. They could warn the user when +externally hosted wheel files are encountered, and/or require the use of a command line +option to enable additional download hosts. Any of these verification policies could be +chosen in configuration files. + +Installer tools should also probably provide better error messages when +externally hosted wheels cannot be downloaded, e.g. because a host is +unreachable. As described above, such tools could query enough metadata from +PyPI to provide clear and distinct error messages pointing users to the +package's external hosting support email or issue tracker. + +Constraints for external hosting services +========================================= + +The following constraints lead to reliable and compatible external wheel hosting services: + +- External wheels **MUST** be served over HTTPS, with a certificate signed by + `Mozilla's root certificate store `__. This ensures + compatibility with `pip `__ + and `uv + `__. At + the time of this writing, ``pip`` 24.2 on Python 3.10 or newer uses the system + certificate store in addition to the Mozilla store provided by the third party `certifi + `__ Python package. ``uv`` uses the Mozilla store + provided by the `webpki-roots `__ crate, but not + the system store unless the ``--native-tls`` flag is given [#fn1]_. *The PyPI + administrators may modify this requirement in the future, but compatibility with popular + installers will not be compromised.* +- External wheel hosts **SHOULD** use a content delivery network (`CDN + `__), just as PyPI does. +- External wheel hosts **MUST** commit to a stable URL for all wheels they host. +- Externally hosted wheels **MUST NOT** be removed from an external wheel host unless the + corresponding ``.rim`` file is deleted from PyPI first, and **MUST NOT** remove external + wheels for yanked releases. +- External wheel hosts **MUST** support `HTTP range requests`_. +- External wheel hosts **SHOULD** support the `HTTP/2`_ protocol. + +Security +======== + +Several factors as described in this proposal should mitigate security +concerns with externally hosted wheels, such as: + +- Wheel file checksums **MUST** be included in ``.rim`` files, and once uploaded cannot be + changed. Since the checksum stored on PyPI is immutable and required, it is not possible + to spoof an external wheel file, even if the owning organization lost control of their + hosting domain. +- Externally hosted wheels **MUST** be served over HTTPS. +- In order to serve externally hosted wheels, organizations **MUST** be approved by the + PyPI admins. + +When users identify malware or vulnerabilities in PyPI-hosted projects, they can now +report this using the `malware reporting facilities `__ on +PyPI, as also described in this `blog post`_. The same process can be used to report +security issues in externally hosted wheels, and the same remediation process should be +used. In addition, since organizations with external hosting enabled MUST provide a +support contact URI, that URI can be used in some cases to report the security issue to +the hosting organization. Such organization reporting won't make sense for malware, but +could indeed be a very useful way to report security vulnerabilities in externally hosted +wheels. + +Rejected ideas +============== + +Several ideas were considered and rejected. + +- Requiring digital signatures on externally hosted wheel files, either in + addition to or other than hashes. We deem this unnecessary since the + checksum requirement should be enough to validate that the metadata on PyPI + for a wheel exactly matches the downloaded wheel. The added complexity of + key management outweighs any additional benefit such digital signatures + might convey. +- Hash verification on ``.rim`` file uploads. PyPI *could* verify that the + hash in the uploaded ``.rim`` file matches the externally hosted wheel + before it accepts the upload, but this requires downloading the external + wheel and performing the checksum. This would impose an ordering restriction + on uploads, such that the external wheel file is uploaded and publicly + available *before* uploading the ``.rim`` file to PyPI. While this might be + a good idea to avoid any race condition, it should not be a + requirement. This also means that the upload of the ``.rim`` file cannot be + accepted until this external ``.whl`` file is downloaded and verified. This + increases PyPI bandwidth and slows down the upload query, although + :pep:`694` draft uploads could potentially mitigate these concerns. Still, + the benefit is not likely worth the additional complexity. +- Periodic verification of the download URLs by the index. PyPI could try to periodically + ensure that the external wheel host or the external ``.whl`` file itself is still + available, e.g. via an ``HTTP HEAD`` request. This is likely overkill and without also + providing the file's checksum in the response [#fn2]_, may not provide much additional + benefit. +- This PEP could allow for an organization to provide fallback download hosts, + such that a secondary is available if the primary goes down. We believe + that DNS-based replication is a much better, well-known technique, and + probably much more resilient anyway. +- ``.rim`` file replacement. While it is allowed for ``.whl`` files to replace + existing ``.rim`` files, as long as a) the ``.rim`` file hasn't been deleted + or yanked, b) the checksums match, we do not allow replacing ``.whl`` files + with ``.rim`` files, nor do we allow a ``.rim`` file to overwrite an + existing ``.rim`` file. This latter could be a technique to change the + hosting URL for an externally hosted ``.whl``, however we do not think this + is a good idea. There are other ways to "fix" an external host URL as + described above, and we do not want to encourage mass re-uploads of existing + ``.rim`` files. + +Footnotes +========= +.. [#fn1] The ``uv --native-tls`` flag `replaces + `__ + the ``webpki-roots`` store. +.. [#fn2] There being no standard way to return the file's checksum in response to an + ``HTTP HEAD`` request. + +Copyright +========= + +This document is placed in the public domain or under the +CC0-1.0-Universal license, whichever is more permissive. + +.. _`requesting higher limits`: https://github.com/pypi/support/issues?q=is%3Aissue+is%3Aclosed+file+limit+request +.. _`wheel file naming format`: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-format +.. _`allowed`: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-dist-info-directory +.. _`signed wheel file format`: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#signed-wheel-files +.. _`simple repository API`: https://packaging.python.org/en/latest/specifications/simple-repository-api/# +.. _`JSON response`: https://packaging.python.org/en/latest/specifications/simple-repository-api/#json-based-simple-api-for-python-package-indexes +.. _`PyPI repository version`: https://packaging.python.org/en/latest/specifications/simple-repository-api/#versioning-pypi-s-simple-api +.. _`blog post`: https://blog.pypi.org/posts/2024-03-06-malware-reporting-evolved/ +.. _`HTTP HEAD`: https://datatracker.ietf.org/doc/html/rfc9110#section-9.3.2 +.. _`hash algorithm and hex encoded digests`: https://peps.python.org/pep-0694/#upload-each-file +.. _`HTTP range requests`: https://http.dev/range-request +.. _`HTTP/2`: https://http.dev/2 From f1978588d5e66c5dd18e9d6e33fe131694ba6ffb Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 13:53:52 -0700 Subject: [PATCH 02/35] Fix some formatting --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 958175c6c69..1be020d94f5 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -295,7 +295,7 @@ installer tool tries to download and install the package ``foo`` wheel, the download step will fail. The installer would then be able to query PyPI to provide a useful error message to the end user: -- The installer downloads the `.rim` file and reads the ``owner`` key from the +- The installer downloads the ``.rim`` file and reads the ``owner`` key from the ``EXTERNAL-HOSTING.json`` file inside the ``.rim`` zip file. - The installer queries PyPI for the support URI for the organization owner of the externally hosted wheel. From f1ad9f6175f39309092039945ae760ac99f093ed Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 13:59:19 -0700 Subject: [PATCH 03/35] /peps/numerical.rst is generated --- .gitignore | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 6004c5e61f7..6e056e9f68d 100644 --- a/.gitignore +++ b/.gitignore @@ -24,4 +24,5 @@ coverage.xml /venv # Builds -/sphinx-warnings.txt \ No newline at end of file +/sphinx-warnings.txt +/peps/numerical.rst From ffa8a6aa604c820bec7cab25d02b0e737e6ff495 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 14:16:24 -0700 Subject: [PATCH 04/35] Update peps/pep-0759.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 1be020d94f5..4632f85c78a 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -19,7 +19,7 @@ projects, packages, or their metadata. That functionality is already available by externally hosting independent package indexes. Because this PEP only provides a mechanism for projects to customize the download URL for specific released wheel artifacts, dependency resolution as already implemented by -common installer tools such as `pip `__, and +common installer tools such as `pip `__ and `uv `__ does not need to change. This PEP defines what it means to be "safe" in this context, along with a new From 599d2e1a040b2df68245c9ce99b3c44153ca0f2f Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 14:17:27 -0700 Subject: [PATCH 05/35] Update peps/pep-0759.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 4632f85c78a..7ad5569b6df 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -32,7 +32,7 @@ into ``.rim`` files. Rationale ========= -The Python Package Index, hosted at pypi.org imposes `default limits +The Python Package Index, hosted at https://pypi.org, imposes `default limits `__ on upload artifact file size (100 MiB) and total project size (10.0 GiB). Most projects can comfortably fit within these limits during the lifetime of the project, through years of uploads. A few projects From 056aa9f56ddf9b6119bde19d1da5942f3b31593c Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 14:17:36 -0700 Subject: [PATCH 06/35] Update peps/pep-0759.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 7ad5569b6df..9b1e3ae57b5 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -63,7 +63,7 @@ process" to modify several aspects of release file hosting on PyPI. As this PEP describes, PyPI originally supported only project and release *registration* without also allowing for artifact file hosting. As such, most projects hosted release file artifacts elsewhere. Artifact hosting was later -added, but the mix of externally and PyPI-hosted files lead to a wide range of +added, but the mix of externally and PyPI-hosted files led to a wide range of usability and potential security-related problems. PEP 438 was an attempt to provide several facilities to allow external hosting while promoting a PyPI-first hosting preference. From 5e6707acc8a00f48eeee3f32780fc7a72521bad6 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 14:18:26 -0700 Subject: [PATCH 07/35] Update peps/pep-0759.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 9b1e3ae57b5..d98fe807440 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -95,7 +95,7 @@ hosting, and how does this proposal avoid the problems documented in PEP 470? One well-known problem that consolidating multiple indexes enables is `dependency confusion attacks -`__ of +`__, to which Python *can* be particularly vulnerable, due to the algorithm that ``pip install`` uses for resolving package dependencies and preferred versions. The ``uv`` tool addresses this by supporting an additional `index strategy From 4c64f22538c7f46e2a4cd69bcf6297cccc6abad1 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 14:18:38 -0700 Subject: [PATCH 08/35] Update peps/pep-0759.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index d98fe807440..6eb41302f27 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -137,7 +137,7 @@ PyPI imposes a `default artifact size limit `__ of 10GiB. Most packages and artifacts can easily fit in these limits, even for packages containing binary extension modules for a variety of platforms. A small, but important class of packages routinely exceed these -limits, requiring them to submit PyPI support tickets `requesting higher limits`_ It's not +limits, requiring them to submit PyPI support tickets `requesting higher limits`_. It's not necessarily difficult to get resolution on such exceptions, but it is a special process that can take some time to resolve, and the criteria for granting such exceptions isn't well documented. From fbe93a3caccdbf224c891bcec8c55dbb39a7a959 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 14:19:02 -0700 Subject: [PATCH 09/35] Update peps/pep-0759.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 6eb41302f27..75a4fd48e2d 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -139,7 +139,7 @@ easily fit in these limits, even for packages containing binary extension module variety of platforms. A small, but important class of packages routinely exceed these limits, requiring them to submit PyPI support tickets `requesting higher limits`_. It's not necessarily difficult to get resolution on such exceptions, but it is a special process -that can take some time to resolve, and the criteria for granting such exceptions isn't +that can take some time to resolve, and the criteria for granting such exceptions aren't well documented. Reduce operational complexity From 156870151b4531a8fc1485b3f2a30a6be0e0ec05 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 14:19:14 -0700 Subject: [PATCH 10/35] Update peps/pep-0759.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 75a4fd48e2d..17f7558780c 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -145,7 +145,7 @@ well documented. Reduce operational complexity ============================= -Standing up and maintaining an entire package index can be a complex +Setting up and maintaining an entire package index can be a complex operational solution, both time and resource intensive. This is especially true if the main purpose of such an index is just to avoid file size limitations. The external index approach also imposes a tricky UX on consumers From 0c90cb48a0fc415ad2f30a34c1f288f9740a3963 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Tue, 1 Oct 2024 14:20:17 -0700 Subject: [PATCH 11/35] Update peps/pep-0759.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 17f7558780c..0ae3ce0cf25 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -152,7 +152,7 @@ limitations. The external index approach also imposes a tricky UX on consumers of projects on the external index, requiring them to understand how CLI options such as ``--external-index-url`` work, along with the security implications of such flags. It would be much easier for both producers and -consumers of large wheel packages to just stand up and maintain a simple web +consumers of large wheel packages to just set up and maintain a simple web server, capable of serving individual files with no more complex API than ``HTTP GET``. Such an interface is also easily cacheable or placed behind a `CDN `__. Simple HTTP From f88c18ff6aa70c82c98fec8440484a1c1ae37cc6 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:33:38 -0700 Subject: [PATCH 12/35] Update peps/pep-0759.rst Co-authored-by: Jelle Zijlstra --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 0ae3ce0cf25..316075dcb4c 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -439,7 +439,7 @@ Several ideas were considered and rejected. or yanked, b) the checksums match, we do not allow replacing ``.whl`` files with ``.rim`` files, nor do we allow a ``.rim`` file to overwrite an existing ``.rim`` file. This latter could be a technique to change the - hosting URL for an externally hosted ``.whl``, however we do not think this + hosting URL for an externally hosted ``.whl``; however, we do not think this is a good idea. There are other ways to "fix" an external host URL as described above, and we do not want to encourage mass re-uploads of existing ``.rim`` files. From 6c86b80d19d77929bfdc31dc0151b09a38f729e8 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:33:51 -0700 Subject: [PATCH 13/35] Update peps/pep-0759.rst Co-authored-by: Ethan Smith --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 316075dcb4c..d3bbcd362ca 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -450,7 +450,7 @@ Footnotes `__ the ``webpki-roots`` store. .. [#fn2] There being no standard way to return the file's checksum in response to an - ``HTTP HEAD`` request. + :rfc:`9110#section-9.3.2` request. Copyright ========= From 85ec40e59b6c571b89bc175f4cd88daac873e05a Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:34:08 -0700 Subject: [PATCH 14/35] Update peps/pep-0759.rst Co-authored-by: Ethan Smith --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index d3bbcd362ca..93a0c529e76 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -427,7 +427,7 @@ Several ideas were considered and rejected. the benefit is not likely worth the additional complexity. - Periodic verification of the download URLs by the index. PyPI could try to periodically ensure that the external wheel host or the external ``.whl`` file itself is still - available, e.g. via an ``HTTP HEAD`` request. This is likely overkill and without also + available, e.g. via an :rfc:`9110#section-9.3.2` request. This is likely overkill and without also providing the file's checksum in the response [#fn2]_, may not provide much additional benefit. - This PEP could allow for an organization to provide fallback download hosts, From aba3c5029b2aa4233bc9329272dbcf554719dac6 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:35:05 -0700 Subject: [PATCH 15/35] Update peps/pep-0759.rst Co-authored-by: Ethan Smith --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 93a0c529e76..61fb79f3286 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -198,7 +198,7 @@ This is a JSON file contains containing the following keys: This is an integer value describing the size in bytes of the physical ``.whl`` file. ``hashes`` This is a dictionary of the format described in :pep:`694`, used to capture both the - `hash algorithm and hex encoded digests`_ of the physical ``.whl`` file, with the same + :pep:`694#upload-each-file` of the physical ``.whl`` file, with the same constraints as proposed in that PEP. Since these hashes are immutable once uploaded to PyPI, they serve as a critical validation that the externally hosted wheel hasn't been corrupted or compromised. From dc17cd053b8b671fbdb143914898d9bc704e6890 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:35:16 -0700 Subject: [PATCH 16/35] Update peps/pep-0759.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 61fb79f3286..2ab02295ab1 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -13,7 +13,7 @@ Abstract ======== This PEP proposes a mechanism by which projects hosted on `pypi.org -`__ can safely host wheel artifacts on external sites other +`__ can safely host wheel artifacts on external sites other than PyPI. This PEP explicitly does *not* propose external hosting of projects, packages, or their metadata. That functionality is already available by externally hosting independent package indexes. Because this PEP only From d76f91558cdc123af720565b1c456a606da6ad67 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:35:26 -0700 Subject: [PATCH 17/35] Update peps/pep-0759.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 2ab02295ab1..08ba60fa0cd 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -26,7 +26,7 @@ This PEP defines what it means to be "safe" in this context, along with a new package upload file format called a ``.rim`` file. It defines how ``.rim`` files affect the metadata returned for a package's `Simple Repository API `__ -in both HTML and JSON format, and how traditional wheels can easily be turned +in both HTML and JSON formats, and how traditional wheels can easily be turned into ``.rim`` files. Rationale From 9eb22a5e22d26d6deb25ecbfabfe1b5087065eec Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:36:23 -0700 Subject: [PATCH 18/35] Update peps/pep-0759.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 08ba60fa0cd..78115b0ccbb 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -2,7 +2,7 @@ PEP: 759 Title: External Wheel Hosting Author: Barry Warsaw , Ethan Smith -BDFL-Delegate: Donald Stufft +PEP-Delegate: Donald Stufft Status: Draft Type: Standards Track Topic: Packaging From d04b6376ac2385df9d5d697554b35a3d7850cfc7 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:36:32 -0700 Subject: [PATCH 19/35] Update peps/pep-0759.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 78115b0ccbb..0452293239b 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -134,7 +134,7 @@ external wheel hosting solution can mitigate: artifact size limits. PyPI imposes a `default artifact size limit `__ of 100MiB and a default overall `project size limit -`__ of 10GiB. Most packages and artifacts can +`__ of 10 GiB. Most packages and artifacts can easily fit in these limits, even for packages containing binary extension modules for a variety of platforms. A small, but important class of packages routinely exceed these limits, requiring them to submit PyPI support tickets `requesting higher limits`_. It's not From 207637fce581a3bd202f1a30eca8e33f21cbfd94 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:36:40 -0700 Subject: [PATCH 20/35] Update peps/pep-0759.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 0452293239b..e8c20690e69 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -167,7 +167,7 @@ Specification ============= A new type of uploadable file is defined, called a "RIM" (i.e. ``.rim``) file, so named to -evoke the image of a wheel with the tire removed, and backronym'd to "Remote Installable +evoke the image of a wheel with the tire removed, and backronymed to "Remote Installable Metadata" files. ``.rim`` files are easily derivable from wheels, and the process of turning a wheel into a ``.rim`` is outlined below. The file name format exactly matches the `wheel file naming format`_ specification, except that RIM files use the suffix From 8753c5d064fe2d1c031c49b9ef5bfe9c7cb7db6f Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:36:50 -0700 Subject: [PATCH 21/35] Update peps/pep-0759.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index e8c20690e69..f269d17b53e 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -34,7 +34,7 @@ Rationale The Python Package Index, hosted at https://pypi.org, imposes `default limits `__ on upload artifact file size (100 MiB) and total -project size (10.0 GiB). Most projects can comfortably fit within these limits +project size (10 GiB). Most projects can comfortably fit within these limits during the lifetime of the project, through years of uploads. A few projects have encountered these limits, and have been granted both `file size exceptions `__ and `project size From 442fc27dd2a129188f910eaabaae43bb1b971a2c Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:37:01 -0700 Subject: [PATCH 22/35] Update peps/pep-0759.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index f269d17b53e..163c6b8006b 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -133,7 +133,7 @@ dependency confusion attacks. There is another class of problem that a safe external wheel hosting solution can mitigate: artifact size limits. PyPI imposes a `default artifact size limit `__ of -100MiB and a default overall `project size limit +100 MiB and a default overall `project size limit `__ of 10 GiB. Most packages and artifacts can easily fit in these limits, even for packages containing binary extension modules for a variety of platforms. A small, but important class of packages routinely exceed these From 5be072b5b82747c0f97ac8506ec1be05e84c8ad7 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:41:19 -0700 Subject: [PATCH 23/35] Update CODEOWNERS for PEP 759 --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 36b5fbed6d0..d5f6c8482c1 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -638,6 +638,7 @@ peps/pep-0753.rst @warsaw # ... peps/pep-0756.rst @vstinner peps/pep-0757.rst @vstinner +peps/pep-0759.rst @warsaw @ethanhs @ethanhs-nv peps/pep-0789.rst @njsmith # ... peps/pep-0801.rst @warsaw From dac3a68ce20fe552f241b7450789c616b5c896f0 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:43:47 -0700 Subject: [PATCH 24/35] Update peps/pep-0759.rst Co-authored-by: Ethan Smith --- peps/pep-0759.rst | 2 -- 1 file changed, 2 deletions(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 163c6b8006b..c3a37bbafae 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -466,7 +466,5 @@ CC0-1.0-Universal license, whichever is more permissive. .. _`JSON response`: https://packaging.python.org/en/latest/specifications/simple-repository-api/#json-based-simple-api-for-python-package-indexes .. _`PyPI repository version`: https://packaging.python.org/en/latest/specifications/simple-repository-api/#versioning-pypi-s-simple-api .. _`blog post`: https://blog.pypi.org/posts/2024-03-06-malware-reporting-evolved/ -.. _`HTTP HEAD`: https://datatracker.ietf.org/doc/html/rfc9110#section-9.3.2 -.. _`hash algorithm and hex encoded digests`: https://peps.python.org/pep-0694/#upload-each-file .. _`HTTP range requests`: https://http.dev/range-request .. _`HTTP/2`: https://http.dev/2 From b88ec81fed14cdcec1b5e4438fc96f3bf4ba4f60 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 11:44:28 -0700 Subject: [PATCH 25/35] Update peps/pep-0759.rst Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com> --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index c3a37bbafae..4685a03555b 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -6,7 +6,7 @@ PEP-Delegate: Donald Stufft Status: Draft Type: Standards Track Topic: Packaging -Created: 19-Sep-2024 +Created: 1-Oct-2024 Post-History: Abstract From 3e15a58874435c35dcadb3fa49ae39c5cd4e41d5 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 12:26:36 -0700 Subject: [PATCH 26/35] Header dates must have two digit days --- peps/pep-0759.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 4685a03555b..ef8d1714b86 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -6,7 +6,7 @@ PEP-Delegate: Donald Stufft Status: Draft Type: Standards Track Topic: Packaging -Created: 1-Oct-2024 +Created: 01-Oct-2024 Post-History: Abstract From 87d27903681d62be4ec8705e83898d3a1d5aa0b2 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 12:29:32 -0700 Subject: [PATCH 27/35] Use `HTTP HEAD` in RFC text --- peps/pep-0759.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index ef8d1714b86..fdde895606c 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -427,7 +427,7 @@ Several ideas were considered and rejected. the benefit is not likely worth the additional complexity. - Periodic verification of the download URLs by the index. PyPI could try to periodically ensure that the external wheel host or the external ``.whl`` file itself is still - available, e.g. via an :rfc:`9110#section-9.3.2` request. This is likely overkill and without also + available, e.g. via an :rfc:`HTTP HEAD <9110#section-9.3.2>` request. This is likely overkill and without also providing the file's checksum in the response [#fn2]_, may not provide much additional benefit. - This PEP could allow for an organization to provide fallback download hosts, @@ -450,7 +450,7 @@ Footnotes `__ the ``webpki-roots`` store. .. [#fn2] There being no standard way to return the file's checksum in response to an - :rfc:`9110#section-9.3.2` request. + :rfc:`HTTP HEAD <9110#section-9.3.2>` request. Copyright ========= From 77344baf93041d25b1dc2cdda0491f775a0888b5 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 13:48:13 -0700 Subject: [PATCH 28/35] Update .github/CODEOWNERS Co-authored-by: Ethan Smith --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index d5f6c8482c1..3bc05f47b5d 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -638,7 +638,7 @@ peps/pep-0753.rst @warsaw # ... peps/pep-0756.rst @vstinner peps/pep-0757.rst @vstinner -peps/pep-0759.rst @warsaw @ethanhs @ethanhs-nv +peps/pep-0759.rst @warsaw peps/pep-0789.rst @njsmith # ... peps/pep-0801.rst @warsaw From 2b15c38e1c940c357f934ebd3bc62d2416e85d8f Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 13:59:25 -0700 Subject: [PATCH 29/35] Improve the wording around PEP 470 and PEP 438 --- peps/pep-0759.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index fdde895606c..fc128029974 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -70,8 +70,9 @@ PyPI-first hosting preference. PEP 438 was complex, with three different "hosting modes", ``rel`` metadata in the simple HTML index pages to signify hosting locations, and a two-phase -transition plan affecting PyPI and installer tools. PEP 438 was ultimately -retracted in 2015 by :pep:`470`. In one respect, PEP 438 did succeed in: +transition plan affecting PyPI and installer tools. PEP 438 was ultimately +retracted in 2015 by :pep:`470`, which acknowledges that PEP 438 did succeed +in... bringing about more people to utilize PyPI's repository features, an altogether good thing given the global CDN powering PyPI providing speed From 3226b987a431e1a2a714b39397874746733919e2 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 14:30:48 -0700 Subject: [PATCH 30/35] Responding to PR comments * Move some sections to subsections of Specifications * In Rejected Ideas, remove a comment that referred to a previous non-requirement --- peps/pep-0759.rst | 33 +++++++++++++++++++-------------- 1 file changed, 19 insertions(+), 14 deletions(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index fc128029974..796258c1039 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -204,6 +204,9 @@ This is a JSON file contains containing the following keys: to PyPI, they serve as a critical validation that the externally hosted wheel hasn't been corrupted or compromised. +Effects of the RIM file +----------------------- + The only effect of a ``.rim`` file is to change the download URL for the wheel artifact in both the HTML and JSON interfaces in the `simple repository API`_. In the HTML page for a package release, the ``href`` attribute **MUST** be the value of the ``uri`` key, @@ -223,12 +226,15 @@ like any ``.whl`` file, with the exact same semantics (i.e. deletions are perman a ``.rim`` is deleted, an index **MUST NOT** allow a matching ``.whl`` or ``.rim`` file to be (re-)uploaded. +Availability order +------------------ + Externally hosted wheels **MUST** be available before the corresponding ``.rim`` file is uploaded to PyPI, otherwise a publishing race condition is introduced, although this requirement **MAY** be relaxed for ``.rim`` files uploaded to a :pep:`694` staged release. Wheels can override RIMs -======================== +------------------------ Indexes **MUST** reject ``.rim`` files if a matching ``.whl`` file already exists with the exact same file name tags. However, indexes **MAY** accept a ``.whl`` file if a matching @@ -243,7 +249,7 @@ existing ``.rim`` file, and these hashes must match or the overriding upload **M rejected. PyPI API bump unnecessary -========================= +------------------------- It's likely that the changes are backward compatible enough that a bump in the `PyPI repository version`_ is not necessary. Since ``.rim`` files are essentially changes only @@ -414,27 +420,26 @@ Several ideas were considered and rejected. for a wheel exactly matches the downloaded wheel. The added complexity of key management outweighs any additional benefit such digital signatures might convey. -- Hash verification on ``.rim`` file uploads. PyPI *could* verify that the - hash in the uploaded ``.rim`` file matches the externally hosted wheel - before it accepts the upload, but this requires downloading the external - wheel and performing the checksum. This would impose an ordering restriction - on uploads, such that the external wheel file is uploaded and publicly - available *before* uploading the ``.rim`` file to PyPI. While this might be - a good idea to avoid any race condition, it should not be a - requirement. This also means that the upload of the ``.rim`` file cannot be - accepted until this external ``.whl`` file is downloaded and verified. This - increases PyPI bandwidth and slows down the upload query, although - :pep:`694` draft uploads could potentially mitigate these concerns. Still, - the benefit is not likely worth the additional complexity. + +- Hash verification on ``.rim`` file uploads. PyPI *could* verify that the hash in the + uploaded ``.rim`` file matches the externally hosted wheel before it accepts the upload, + but this requires downloading the external wheel and performing the checksum, which also + implies that the upload of the ``.rim`` file cannot be accepted until this external + ``.whl`` file is downloaded and verified. This increases PyPI bandwidth and slows down + the upload query, although :pep:`694` draft uploads could potentially mitigate these + concerns. Still, the benefit is not likely worth the additional complexity. + - Periodic verification of the download URLs by the index. PyPI could try to periodically ensure that the external wheel host or the external ``.whl`` file itself is still available, e.g. via an :rfc:`HTTP HEAD <9110#section-9.3.2>` request. This is likely overkill and without also providing the file's checksum in the response [#fn2]_, may not provide much additional benefit. + - This PEP could allow for an organization to provide fallback download hosts, such that a secondary is available if the primary goes down. We believe that DNS-based replication is a much better, well-known technique, and probably much more resilient anyway. + - ``.rim`` file replacement. While it is allowed for ``.whl`` files to replace existing ``.rim`` files, as long as a) the ``.rim`` file hasn't been deleted or yanked, b) the checksums match, we do not allow replacing ``.whl`` files From a78fb6992d5bf16f5ed4eebb9e4a780163dfc15a Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 14:39:09 -0700 Subject: [PATCH 31/35] Update peps/pep-0759.rst Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com> --- peps/pep-0759.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 796258c1039..c7a7026eba0 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -24,8 +24,8 @@ common installer tools such as `pip `__ and This PEP defines what it means to be "safe" in this context, along with a new package upload file format called a ``.rim`` file. It defines how ``.rim`` -files affect the metadata returned for a package's `Simple Repository API -`__ +files affect the metadata returned for a package's :ref:`Simple Repository API +` in both HTML and JSON formats, and how traditional wheels can easily be turned into ``.rim`` files. From bba566c01897d0b4078bf33ad392aa841560e5c4 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 15:24:04 -0700 Subject: [PATCH 32/35] Respond to the feedback on wheel stubs in the Rationale section --- peps/pep-0759.rst | 37 ++++++++++++++++--------------------- 1 file changed, 16 insertions(+), 21 deletions(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index c7a7026eba0..4c9088ffd5b 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -33,27 +33,22 @@ Rationale ========= The Python Package Index, hosted at https://pypi.org, imposes `default limits -`__ on upload artifact file size (100 MiB) and total -project size (10 GiB). Most projects can comfortably fit within these limits -during the lifetime of the project, through years of uploads. A few projects -have encountered these limits, and have been granted both `file size -exceptions `__ and `project size -exceptions `__, allowing them to -continue uploading new releases without having to take more drastic measures, -such as removing files which may potentially still be in use by consumers -(e.g. through version pins). - -A related workaround is the `"wheel stub" -`__ approach, which provides an -indirect link between PyPI and an external third party package index, where -such limitations can be avoided. Wheel stubs are `source distributions -`__, -a.k.a. "sdists" which utilize a :pep:`517` build backend that, instead of turning -source code into a binary wheel, performs some logic to calculate the URL for -an existing, externally hosted wheel to download. Technically speaking, this -satisfies the build backend contract, but the link between PyPI, the sdist, -and the externally hosted wheel isn't evident, easily discoverable, or -auditable. +`__ on upload artifact file size (100 MiB) and total project size +(10 GiB). Most projects can comfortably fit within these limits during the lifetime of the +project, through years of uploads. A few projects have encountered these limits, and have +been granted both file size and project size exceptions, allowing them to continue +uploading new releases without having to take more drastic measures, such as removing +files which may potentially still be in use by consumers (e.g. through version pins). + +A related workaround is the `"wheel stub" `__ +approach, which provides an indirect link between PyPI and an external third party package +index, where such limitations can be avoided. Wheel stubs are :ref:`source distributions +` (a.k.a. "sdists") which utilize a :pep:`517` build +backend that, instead of turning source code into a binary wheel, performs some logic to +calculate the URL for an existing, externally hosted wheel to download and install. This +approach works, but it obscures the connection between PyPI, the sdist, and the externally +hosted wheel, since there is no way to present this information to ``pip`` or other such +tools. Historical context ================== From 5a815bd10e9642742f73ec01b102c044597ae039 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 15:30:28 -0700 Subject: [PATCH 33/35] Improve a heading and relabel some sections as subsections --- peps/pep-0759.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 4c9088ffd5b..991b923c198 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -51,7 +51,7 @@ hosted wheel, since there is no way to present this information to ``pip`` or ot tools. Historical context -================== +------------------ In 2013, :pep:`438` proposed a "backward-compatible two-phase transition process" to modify several aspects of release file hosting on PyPI. As this @@ -84,7 +84,7 @@ for so many years, all Python package installation tools support querying multiple indexes for dependency resolution. The problem with multiple indexes -================================= +--------------------------------- Why then does this PEP propose to allow a more limited form of external hosting, and how does this proposal avoid the problems documented in PEP 470? @@ -121,8 +121,8 @@ This PEP also preserves the core purpose of PyPI, and allows it to remain the traditional, canonical, centralized index of all Python packages. -It's not just about indexes -=========================== +Addressing PyPI limits +---------------------- The problem *this* PEP solves is not just reduction or elimination of dependency confusion attacks. There is another class of problem that a safe @@ -138,8 +138,8 @@ necessarily difficult to get resolution on such exceptions, but it is a special that can take some time to resolve, and the criteria for granting such exceptions aren't well documented. -Reduce operational complexity -============================= +Reducing operational complexity +------------------------------- Setting up and maintaining an entire package index can be a complex operational solution, both time and resource intensive. This is especially From b12bc7e5bf661a9ce3a306dded7f670caf72ec1a Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 15:58:58 -0700 Subject: [PATCH 34/35] Reword the PyPI limits rationale, based on feedback --- peps/pep-0759.rst | 24 ++++++++++-------------- 1 file changed, 10 insertions(+), 14 deletions(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 991b923c198..7fea965deb1 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -124,19 +124,15 @@ packages. Addressing PyPI limits ---------------------- -The problem *this* PEP solves is not just reduction or elimination of -dependency confusion attacks. There is another class of problem that a safe -external wheel hosting solution can mitigate: artifact size limits. - -PyPI imposes a `default artifact size limit `__ of -100 MiB and a default overall `project size limit -`__ of 10 GiB. Most packages and artifacts can -easily fit in these limits, even for packages containing binary extension modules for a -variety of platforms. A small, but important class of packages routinely exceed these -limits, requiring them to submit PyPI support tickets `requesting higher limits`_. It's not -necessarily difficult to get resolution on such exceptions, but it is a special process -that can take some time to resolve, and the criteria for granting such exceptions aren't -well documented. +This proposal also addresses the problem of size limits imposed by PyPI, where there is a +`default artifact size limit `__ of 100 MiB and a +default overall `project size limit `__ of 10 +GiB. Most packages and artifacts can easily fit in these limits, even for packages +containing binary extension modules for a variety of platforms. A small, but important +class of packages routinely exceed these limits, requiring them to submit PyPI `exception +request support tickets`_. It's not necessarily difficult to get resolution on such +exceptions, but it is a special process that can take some time to resolve, and the +criteria for granting such exceptions aren't well documented. Reducing operational complexity ------------------------------- @@ -459,7 +455,7 @@ Copyright This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. -.. _`requesting higher limits`: https://github.com/pypi/support/issues?q=is%3Aissue+is%3Aclosed+file+limit+request +.. _`exception request support tickets`: https://github.com/pypi/support/issues?q=is%3Aissue+is%3Aclosed+file+limit+request .. _`wheel file naming format`: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-format .. _`allowed`: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-dist-info-directory .. _`signed wheel file format`: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#signed-wheel-files From 6f692c00c1cb96e166f104d2657b630a1579b7e6 Mon Sep 17 00:00:00 2001 From: Barry Warsaw Date: Wed, 2 Oct 2024 16:47:38 -0700 Subject: [PATCH 35/35] Reword the intro to the specifications section, and other improvements --- peps/pep-0759.rst | 21 +++++++++++++-------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/peps/pep-0759.rst b/peps/pep-0759.rst index 7fea965deb1..810f4d19247 100644 --- a/peps/pep-0759.rst +++ b/peps/pep-0759.rst @@ -158,11 +158,12 @@ This PEP proposes an approach that favors such operational simplicity. Specification ============= -A new type of uploadable file is defined, called a "RIM" (i.e. ``.rim``) file, so named to -evoke the image of a wheel with the tire removed, and backronymed to "Remote Installable -Metadata" files. ``.rim`` files are easily derivable from wheels, and the process of -turning a wheel into a ``.rim`` is outlined below. The file name format exactly matches -the `wheel file naming format`_ specification, except that RIM files use the suffix +A new type of uploadable file is defined, called a "RIM" (i.e. ``.rim``), or "Remote +Installable Metadata" file. The name evokes the image of a wheel with the tire removed, +and emphasizes that ``.rim`` files are easily derived from ``.whl`` files. The process of +turning a ``.whl`` into a ``.rim`` is :ref:`outlined below `. The file name +format exactly matches the :ref:`wheel file naming format +` specification, except that RIM files use the suffix ``.rim``. This means that all the tags used to discriminate ``.whl`` files also distinguish between different ``.rim`` files, and thus can be used during dependency resolution steps, exactly as ``.whl`` files are today. In this respect, ``.whl`` and @@ -182,12 +183,13 @@ This is a JSON file contains containing the following keys: This is the file format version, which for this PEP **MUST** be ``1.0``. ``owner`` This **MUST** name the PyPI organization owner of this externally hosted file, for - reasons which will be described in detail below. + reasons which will be described in :ref:`detail below `. ``uri`` This is a single URL naming the location of the physical ``.whl`` file hosted on an external site. This URL **MUST** use the ``https`` scheme. ``size`` - This is an integer value describing the size in bytes of the physical ``.whl`` file. + This is an integer value describing the size in bytes of the physical ``.whl`` file on + the remote host. ``hashes`` This is a dictionary of the format described in :pep:`694`, used to capture both the :pep:`694#upload-each-file` of the physical ``.whl`` file, with the same @@ -247,6 +249,8 @@ repository version`_ is not necessary. Since ``.rim`` files are essentially chan to the upload API, package resolvers and package installers can continue to function with the APIs they've always supported. +.. _resiliency: + External hosting resiliency =========================== @@ -303,6 +307,8 @@ provide a useful error message to the end user: downloaded. Please contact support@foo.example.com for help. Do not report this to the PyPI administrators. +.. _dismounting: + Dismounting wheels ================== @@ -456,7 +462,6 @@ This document is placed in the public domain or under the CC0-1.0-Universal license, whichever is more permissive. .. _`exception request support tickets`: https://github.com/pypi/support/issues?q=is%3Aissue+is%3Aclosed+file+limit+request -.. _`wheel file naming format`: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#file-format .. _`allowed`: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#the-dist-info-directory .. _`signed wheel file format`: https://packaging.python.org/en/latest/specifications/binary-distribution-format/#signed-wheel-files .. _`simple repository API`: https://packaging.python.org/en/latest/specifications/simple-repository-api/#