From 97bab9f23f7b337bf03715e1365e18d5040b7401 Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Fri, 11 Jun 2021 16:05:58 +0100 Subject: [PATCH 1/5] Add topic guide: Dependency Resolution --- docs/html/topics/dependency-resolution.md | 107 ++++++++++++++++++++++ docs/html/topics/index.md | 1 + 2 files changed, 108 insertions(+) create mode 100644 docs/html/topics/dependency-resolution.md diff --git a/docs/html/topics/dependency-resolution.md b/docs/html/topics/dependency-resolution.md new file mode 100644 index 00000000000..f4b4d283298 --- /dev/null +++ b/docs/html/topics/dependency-resolution.md @@ -0,0 +1,107 @@ +# Dependency Resolution + +pip is capable of determining and installing the dependencies of packages. The +process of determining which version of a dependency to install is known as +dependency resolution. This behaviour can be disabled by passing +{any}`--no-deps` to {any}`pip install`. + +## How it works + +When a user does a `pip install` (e.g. `pip install tea`), pip needs to work +out the package's dependencies (e.g. `spoon`, `hot-water`, `tea-leaves` etc.) +and what the versions of each of those dependencies it should install. + +pip does not have all the information it needs to work out the dependencies of +the requested packages. Over the course of the dependency resolution process, +pip needs to download distributions of the package, which can then provide +the list of dependencies that the package requires. + +## Backtracking + +```{versionchanged} 20.3 +pip's dependency resolver is now capable of backtracking. +``` + +During dependency resolution, pip needs to make assumptions about the package +versions it needs to install and, later, check these assumptions were not +incorrect. When pip finds that an assumption it made earlier is incorrect, it +has to backtrack, which means also discarding some of the work that has already +been done, and going back to choose another path. + +This can look like pip downloading multiple versions of the same package, +since pip explicitly presents each download to the user. The backtracking of +choices made during is not unexpected behaviour or a bug. It is part of how +dependency resolution for Python packages works. + +````{admonition} Example +The user requests `pip install tea`. The package `tea` declares a dependency on +`hot-water`, `spoon`, `cup`, amongst others. + +pip starts by picking the most recent version of `tea` and get the list of +dependencies of that version of `tea`. It will then repeat the process for +those packages, picking the most recent version of `spoon` and then `cup`. Now, +pip notices that the version of `cup` is has chosen is not compatible with the +version of `spoon` it has chosen. Thus, pip will "go back" (backtrack) and try +to use another version of `cup`. If it is successful, it will continue onto the +next package (like `sugar`). Otherwise, it will continue to backtrack on `cup` +until it finds a version of `cup` that is compatible with all the other +packages. + +This can look like: + +``` +$ pip install tea +Collecting tea + Downloading tea-1.9.8-py2.py3-none-any.whl (346 kB) + |████████████████████████████████| 346 kB 10.4 MB/s +Collecting spoon==2.27.0 + Downloading spoon-2.27.0-py2.py3-none-any.whl (312 kB) + |████████████████████████████████| 312 kB 19.2 MB/s +Collecting cup>=1.6.0 + Downloading cup-3.22.0-py2.py3-none-any.whl (397 kB) + |████████████████████████████████| 397 kB 28.2 MB/s +INFO: pip is looking at multiple versions of this package to determine +which version is compatible with other requirements. +This could take a while. + Downloading cup-3.21.0-py2.py3-none-any.whl (395 kB) + |████████████████████████████████| 395 kB 27.0 MB/s + Downloading cup-3.20.0-py2.py3-none-any.whl (394 kB) + |████████████████████████████████| 394 kB 24.4 MB/s + Downloading cup-3.19.1-py2.py3-none-any.whl (394 kB) + |████████████████████████████████| 394 kB 21.3 MB/s + Downloading cup-3.19.0-py2.py3-none-any.whl (394 kB) + |████████████████████████████████| 394 kB 26.2 MB/s + Downloading cup-3.18.0-py2.py3-none-any.whl (393 kB) + |████████████████████████████████| 393 kB 22.1 MB/s + Downloading cup-3.17.0-py2.py3-none-any.whl (382 kB) + |████████████████████████████████| 382 kB 23.8 MB/s + Downloading cup-3.16.0-py2.py3-none-any.whl (376 kB) + |████████████████████████████████| 376 kB 27.5 MB/s + Downloading cup-3.15.1-py2.py3-none-any.whl (385 kB) + |████████████████████████████████| 385 kB 30.4 MB/s +INFO: pip is looking at multiple versions of this package to determine +which version is compatible with other requirements. +This could take a while. + Downloading cup-3.15.0-py2.py3-none-any.whl (378 kB) + |████████████████████████████████| 378 kB 21.4 MB/s + Downloading cup-3.14.0-py2.py3-none-any.whl (372 kB) + |████████████████████████████████| 372 kB 21.1 MB/s +``` + +These multiple `Downloading cup-{version}` lines show that pip is backtracking +choices it is making during dependency resolution. +```` + +If pip starts backtracking choices during dependency resolution, it does not +know how many choices it will backtrack, and how much computation would be +needed. + +For the user, this means it can take a long time to complete when pip starts +backtracking. In the case where a package has a lot of versions, arriving at a +good candidate can take a lot of time. The amount of time depends on the +package size, the number of versions pip must try, and various other factors. + +Backtracking reduces the risk that installing a new package will accidentally +break an existing installed package, and so reduces the risk that your +environment gets messed up. To do this, pip has to do more work, to find out +which version of a package is a good candidate to install. diff --git a/docs/html/topics/index.md b/docs/html/topics/index.md index 478aacf2aa6..19b4d06c098 100644 --- a/docs/html/topics/index.md +++ b/docs/html/topics/index.md @@ -13,4 +13,5 @@ This section of the documentation is currently being fleshed out. See authentication caching configuration +dependency-resolution ``` From 520aed8b066312344abf669d4ee94a42856f52b8 Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Fri, 2 Jul 2021 12:18:53 +0100 Subject: [PATCH 2/5] Stylise console output with `console` highlighting Co-authored-by: Tzu-ping Chung --- docs/html/topics/dependency-resolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/html/topics/dependency-resolution.md b/docs/html/topics/dependency-resolution.md index f4b4d283298..a2300d56ba8 100644 --- a/docs/html/topics/dependency-resolution.md +++ b/docs/html/topics/dependency-resolution.md @@ -49,7 +49,7 @@ packages. This can look like: -``` +```console $ pip install tea Collecting tea Downloading tea-1.9.8-py2.py3-none-any.whl (346 kB) From 414da9b0296d3aa74ffedee6fbde4aebb0e8f133 Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Fri, 9 Jul 2021 10:55:25 +0100 Subject: [PATCH 3/5] Reword pip-starts-backtracking sentence This reads clearer than the initial iteration. Co-authored-by: Tzu-ping Chung Co-authored-by: Paul Moore --- docs/html/topics/dependency-resolution.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/html/topics/dependency-resolution.md b/docs/html/topics/dependency-resolution.md index a2300d56ba8..8cb9e3734a8 100644 --- a/docs/html/topics/dependency-resolution.md +++ b/docs/html/topics/dependency-resolution.md @@ -92,9 +92,8 @@ These multiple `Downloading cup-{version}` lines show that pip is backtracking choices it is making during dependency resolution. ```` -If pip starts backtracking choices during dependency resolution, it does not -know how many choices it will backtrack, and how much computation would be -needed. +If pip starts backtracking during dependency resolution, it does not know how +many choices it will reconsider, and how much computation would be needed. For the user, this means it can take a long time to complete when pip starts backtracking. In the case where a package has a lot of versions, arriving at a From 67da5a6b88bf604ada3ae3f324f613921a5c27ae Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Sun, 11 Jul 2021 09:27:26 +0100 Subject: [PATCH 4/5] Fix a typo Co-authored-by: Tzu-ping Chung --- docs/html/topics/dependency-resolution.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/html/topics/dependency-resolution.md b/docs/html/topics/dependency-resolution.md index 8cb9e3734a8..057976181bb 100644 --- a/docs/html/topics/dependency-resolution.md +++ b/docs/html/topics/dependency-resolution.md @@ -40,7 +40,7 @@ The user requests `pip install tea`. The package `tea` declares a dependency on pip starts by picking the most recent version of `tea` and get the list of dependencies of that version of `tea`. It will then repeat the process for those packages, picking the most recent version of `spoon` and then `cup`. Now, -pip notices that the version of `cup` is has chosen is not compatible with the +pip notices that the version of `cup` it has chosen is not compatible with the version of `spoon` it has chosen. Thus, pip will "go back" (backtrack) and try to use another version of `cup`. If it is successful, it will continue onto the next package (like `sugar`). Otherwise, it will continue to backtrack on `cup` From 14e48b5f286f9e471005163e320e048bac5a8719 Mon Sep 17 00:00:00 2001 From: Pradyun Gedam Date: Tue, 13 Jul 2021 23:10:19 +0100 Subject: [PATCH 5/5] Rephrase paragraph describing how dependency resolution works - Clarify that pip is incrementally building up this information. - Clarify that pip has to download files, which contain the dependency information. - Hint at the fact that this is a recursive process. --- docs/html/topics/dependency-resolution.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/docs/html/topics/dependency-resolution.md b/docs/html/topics/dependency-resolution.md index 057976181bb..0394e1a1d5c 100644 --- a/docs/html/topics/dependency-resolution.md +++ b/docs/html/topics/dependency-resolution.md @@ -11,10 +11,12 @@ When a user does a `pip install` (e.g. `pip install tea`), pip needs to work out the package's dependencies (e.g. `spoon`, `hot-water`, `tea-leaves` etc.) and what the versions of each of those dependencies it should install. -pip does not have all the information it needs to work out the dependencies of -the requested packages. Over the course of the dependency resolution process, -pip needs to download distributions of the package, which can then provide -the list of dependencies that the package requires. +At the start of a `pip install` run, pip does not have all the dependency +information of the requested packages. It needs to work out the dependencies +of the requested packages, the dependencies of those dependencies, and so on. +Over the course of the dependency resolution process, pip will need to download +distribution files of the packages which are used to get the dependencies of a +package. ## Backtracking