Lock markers and groups #9427

radoering · 2024-05-20T13:23:39Z

Pull Request Check List

Requires: #9345
Requires: python-poetry/poetry-core#733

Added tests for changed code.
Updated documentation for changed code.

First commit

Write information about the resulting groups that require a locked package and the resulting marker to the lock file. (Information is only written, not used yet.)

Second commit

Introduce a new config option installer.re-resolve (default true); when set to false the marker information from the lock file is used, there is no re-resolving anymore when installing from the lock file.

Motivation

What triggered me was the discussion at https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists and especially https://github.com/brettcannon/peps/blob/lock-file/peps/pep-9999.rst, which might become a draft for a general lock file. To support this lock file format, we have to lock the resulting markers for packages so that re-resolving is not required for installation.

Further

We can simplifiy the dependency walker in poetry-plugin-export and get rid of dependency walk failed issues.
make use of locked markers in lock file version 2.1 and thereby avoid dependency walk failed poetry-plugin-export#286
We can do other stuff like creating an extra with pinned depenencies (not sure if that is something we want or if it is more something for a plugin).
PoC: automatically create a pinned extra with dependencies pinned from the lock file #9428

Performance

We have to distinguish between locking and installing.

Locking

There is an additional step to calculate the resulting markers so this change can only make things slower. However, I could not measure a significant increase in time required for locking for several projects. I suppose that's because what we are doing in the additional step are mostly marker operations, which have probably been cached during dependency resolution before.

Installing

Evaluating markers instead of re-resolving could be faster or slower. However, I also could not measure a significant effect here. I suppose determining the exact packages to install is just not a bottleneck compared to the actual installation. Even when all packages were already installed (and installation is a noop), I could not measure a difference. However, there is a side effect:

Side effect on installation process

With re-resolving we are installing batches of packages in parallel. The batches are determined by the depth in the dependency graph (same depth -> same batch). Since we do not lock the depth, without re-resolving all packages are in the same batch.

On the one hand, this can have a positive effect on performance because we do not have to wait for the last package of a batch.
(I could observe the complete installation of the dependencies of a largish project with all packages in the cache going down from 70s to 60s.)

On the other hand, this could have a negative effect on robustness (due to race conditions). I suppose, creating batches of packages with the same depth results in only installing packages in parallel that are (more or less) independent of each other.
(If a package depends on another package, they will normally have a different depth.) Parallel installation might be an issue if two packages write to the same directory (name space packages) and try to create it at the same time.

Conclusion

Locking the resulting groups and markers opens up some new possibilities, especially for poetry-plugin-export. Installation without re-resolving is not a game changer on its own. Only the side effect will probably decrease the time for installation in some cases.

For now, I decided to make installation without re-resolving opt-in. We can also make it opt-out if we think it is safe enough and want to get more feedback.

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

github-actions · 2024-06-16T11:43:38Z

Deploy preview for website ready!

✅ Preview
https://website-26789y56u-python-poetry.vercel.app

Built with commit 713e672.
This pull request is being automatically deployed with vercel-action

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

Secrus

Not my area of expertise, so not 100% sure, but looks pretty good to me, minor comments

Secrus · 2024-10-07T21:38:45Z

src/poetry/installation/installer.py

@@ -249,6 +258,13 @@ def _do_install(self) -> int:
                    "pyproject.toml changed significantly since poetry.lock was last"
                    " generated. Run `poetry lock [--no-update]` to fix the lock file."
                )
+            if not reresolve and not self._locker.is_locked_groups_and_markers():


Suggested change

if not reresolve and not self._locker.is_locked_groups_and_markers():

if not (reresolve or self._locker.is_locked_groups_and_markers()):

src/poetry/installation/installer.py

Secrus · 2024-10-29T11:40:34Z

src/poetry/installation/installer.py

@@ -236,10 +233,18 @@ def _do_install(self) -> int:
            with solver.provider.use_source_root(
                source_root=self._env.path.joinpath("src")
            ):
-                ops = solver.solve(use_latest=self._whitelist).calculate_operations()
+                solution = solver.solve(use_latest=self._whitelist)
+                solved_packages = solution.get_solved_packages()


Any reason for making this into 2 variables? solution doesn't seem to be used anywhere past this point.

One more variable but one less line. No strong opinion so I changed it.

…etry#9427)

…ting locked markers) (python-poetry#9427) - introduce "installer.re-resolve" config option (default: True) - if the config option is set to False and the lock file is at least version 2.1, the installer will not re-resolve but evaluate locked markers

radoering changed the title ~~Lock markers and groups3a~~ Lock markers and groups May 20, 2024

radoering mentioned this pull request May 27, 2024

2.0.0 Release #9448

Open

radoering added a commit to radoering/poetry that referenced this pull request Jun 2, 2024

locker: lock transitive marker and groups for each package (python-po…

7f49ddc

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from ec11430 to 49f69c9 Compare June 2, 2024 19:17

radoering added a commit to radoering/poetry that referenced this pull request Jun 16, 2024

locker: lock transitive marker and groups for each package (python-po…

cb60735

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from 49f69c9 to 4d01327 Compare June 16, 2024 11:32

radoering force-pushed the lock-markers-and-groups3a branch from 4d01327 to 241ecfe Compare June 16, 2024 11:38

radoering added the impact/docs Contains or requires documentation changes label Jun 16, 2024

edgarrmondragon mentioned this pull request Jul 2, 2024

packaging: Bump poetry so urllib3 constraint can be removed meltano/sdk#2514

Closed

radoering added a commit to radoering/poetry that referenced this pull request Jul 14, 2024

locker: lock transitive marker and groups for each package (python-po…

e83c36d

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from 241ecfe to b56b49a Compare July 14, 2024 05:36

radoering mentioned this pull request Jul 14, 2024

poc: isolated project plugins & tools #5740

Draft

radoering added a commit to radoering/poetry that referenced this pull request Jul 29, 2024

locker: lock transitive marker and groups for each package (python-po…

2814922

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from b56b49a to 50e93f7 Compare July 29, 2024 04:43

radoering mentioned this pull request Jul 29, 2024

Build requirements issue: PEP517 build from sdist fails rapidfuzz/RapidFuzz#396

Closed

radoering mentioned this pull request Aug 16, 2024

Poetry does not install a non-optional package from lock file #9636

Closed

radoering added a commit to radoering/poetry that referenced this pull request Aug 24, 2024

locker: lock transitive marker and groups for each package (python-po…

f35dd26

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from 50e93f7 to a65c879 Compare August 24, 2024 12:43

radoering mentioned this pull request Aug 25, 2024

poetry install --no-root with only poetry.lock #1301

Open

2 tasks

radoering force-pushed the lock-markers-and-groups3a branch from a65c879 to 623602d Compare September 4, 2024 03:49

radoering added a commit to radoering/poetry that referenced this pull request Sep 15, 2024

locker: lock transitive marker and groups for each package (python-po…

b4b4ccb

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from 623602d to 3e3c070 Compare September 15, 2024 12:04

radoering marked this pull request as ready for review September 15, 2024 12:37

radoering requested a review from a team September 15, 2024 12:37

radoering added a commit to radoering/poetry that referenced this pull request Sep 29, 2024

locker: lock transitive marker and groups for each package (python-po…

99dd799

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from 3e3c070 to 46a8221 Compare September 29, 2024 16:59

radoering added a commit to radoering/poetry that referenced this pull request Oct 4, 2024

locker: lock transitive marker and groups for each package (python-po…

31db080

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from 46a8221 to 5db16dd Compare October 4, 2024 11:28

Secrus added this to the Poetry 2.0 milestone Oct 6, 2024

radoering added a commit to radoering/poetry that referenced this pull request Oct 19, 2024

locker: lock transitive marker and groups for each package (python-po…

ad6e2ef

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from 5db16dd to 517080a Compare October 19, 2024 17:47

radoering added a commit to radoering/poetry that referenced this pull request Oct 20, 2024

locker: lock transitive marker and groups for each package (python-po…

cce52a9

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from 517080a to bdf5dfb Compare October 20, 2024 12:19

radoering added a commit to radoering/poetry that referenced this pull request Oct 20, 2024

locker: lock transitive marker and groups for each package (python-po…

a6deee1

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from bdf5dfb to 1e0300b Compare October 20, 2024 12:41

radoering added a commit to radoering/poetry that referenced this pull request Oct 29, 2024

locker: lock transitive marker and groups for each package (python-po…

f064512

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from 1e0300b to 1244d21 Compare October 29, 2024 04:33

christophe-david mentioned this pull request Oct 29, 2024

Added compatibility with Python 3.13 fast-aircraft-design/FAST-OAD#575

Draft

Secrus reviewed Nov 4, 2024

View reviewed changes

radoering added 2 commits November 5, 2024 06:44

locker: lock transitive marker and groups for each package (python-po…

6a6d6d6

…etry#9427)

radoering force-pushed the lock-markers-and-groups3a branch from 1244d21 to 713e672 Compare November 5, 2024 05:46

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Lock markers and groups #9427

Lock markers and groups #9427

radoering commented May 20, 2024 •

edited

Loading

github-actions bot commented Jun 16, 2024 •

edited

Loading

Secrus left a comment

Secrus Oct 7, 2024

radoering Nov 5, 2024

Secrus Oct 29, 2024

radoering Nov 5, 2024

	if not reresolve and not self._locker.is_locked_groups_and_markers():
	if not (reresolve or self._locker.is_locked_groups_and_markers()):

Lock markers and groups #9427

Are you sure you want to change the base?

Lock markers and groups #9427

Conversation

radoering commented May 20, 2024 • edited Loading

Pull Request Check List

First commit

Second commit

Motivation

Performance

Locking

Installing

Side effect on installation process

Conclusion

github-actions bot commented Jun 16, 2024 • edited Loading

Secrus left a comment

Choose a reason for hiding this comment

Secrus Oct 7, 2024

Choose a reason for hiding this comment

radoering Nov 5, 2024

Choose a reason for hiding this comment

Secrus Oct 29, 2024

Choose a reason for hiding this comment

radoering Nov 5, 2024

Choose a reason for hiding this comment

radoering commented May 20, 2024 •

edited

Loading

github-actions bot commented Jun 16, 2024 •

edited

Loading