[Feature] Poetry needs a reproducible install for itself

[awilkins]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the documentation and believe that my question is not covered.

Feature Request

When installed, Poetry installs the latest available versions of packages in it's dependency graph.

This has lead to multiple occasions where even when installing a specific version of the Poetry client, differences in the installed libraries have lead to an inconsistent experience.

• e.g. Poetry refuses to install package with correct hash #4523
  • which was caused by ongoing releases in poetry-core deprecating support for md5 hashes
• e.g. "Unable to find installation candidates for" packages cached from a LegacyRepository (/simple) #4688
  • which was caused by an update to cachecontrol

It's quite ironic for a tool which promotes reproducible builds through the use of a lockfile that it should suffer from breakages caused by installing itself via pip.

Given the purpose of Poetry (and poetry-core), it's very likely to be installed and used in a CI pipeline, where stability and reliability are strong concerns. I humbly submit that installing Poetry and poetry-core should be a reproducible act for a given version number.

Suggest that this may be achievable by (preferring the first)

• Having a poetry build mode that outputs a setup.py with frozen dependency specifications based on the lock file
  • and using this mode for subsequent releases of poetry and poetry-core
• Using the output of pip freeze for install-poetry.py instead of

        if self._git:
            specification = "git+" + version
        elif self._path:
            specification = version
        else:
            specification = f"poetry=={version}"

        subprocess.run(
            [str(python), "-m", "pip", "install", specification],

[finswimmer]

Hello @awilkins,

the recommended installation method for poetry is the install-poetry.py or get-poetry.py install script. Both make use of the lock file.

fin swimmer

[awilkins]

Both make use of the lock file.

I pulled install-poetry.py today. I think this is the relevant function.

    def install_poetry(self, version: str, env_path: Path) -> None:
        self._overwrite(
            "Installing {} ({}): {}".format(
                colorize("info", "Poetry"),
                colorize("b", version),
                colorize("comment", "Installing Poetry"),
            )
        )

        if WINDOWS:
            python = env_path.joinpath("Scripts/python.exe")
        else:
            python = env_path.joinpath("bin/python")

        if self._git:
            specification = "git+" + version
        elif self._path:
            specification = version
        else:
            specification = f"poetry=={version}"

        subprocess.run(
            [str(python), "-m", "pip", "install", specification],
            stdout=subprocess.PIPE,
            stderr=subprocess.STDOUT,
            check=True,
        )

Could you point out where it makes use of the lockfile here, because I'm not seeing it? Is this the wrong routine?

[finswimmer]

Ah, sorry. Looks like a was to fast with my answer (shouldn't answer during my vacation 😄)

Pinning of the dependency was done by vendoring when using the get-poetry. py script.

With the new installer script we do not vendor the dependencies anymore. Taking the lock file into account is also not possible during installation, because this would require to have poetry installed.

[tom-bowles]

We deploy our poetry-developed applications by first installing a requirements.txt that we generate from the lock file using poetry export. Then we install the wheel. This gives us dependencies that match the lock file without requiring poetry on the deployment machine.

I am in the process of writing a custom install script for the users I support that does the same thing for poetry itself, because we have no choice but to pin both poetry and poetry-core due to a series of releases that are severely broken on Windows (please could somebody give my one-line PR #4682 some love?). Until I complete this custom script, our poetry installation process is to:

• Use install-poetry.py with the --version parameter to install version 1.1.6
• Use pip3.exe from in poetry's venv to --force-reinstall poetry-core to version 1.0.4

Requirements.txt based pinning of dependencies in the installer would be awesome. An alternative (not saying if it's necessarily better) would be to implement the feature discussed in #2778 and use it for poetry itself.

[abn]

At the moment, I'd be hesitent to recommend that we use the poetry export output for anything as it is not yet a 100% reliable and needs a lot more attention. Additionally, since we are bootstrapping from a wheel any pip freeze, poetry export etc. is not necessarily a viable option. Since poetry.lock is Poetry specific, it is unlikely that pip will understand it. If, let's say we are able to interrogate dynamic metadata prior to pip install and somehow passin a runtime option like --use-lock and poetry-core understands it - we could probably do this in the install-poetry.py script. But that does not seem like a near term solution and will require changes outside of Poetry too.

Alternative options are;

1. Distribute poetry as a "dependency pinned" wheel (Build packages with pinned dependencies from .lock file #2778). This is not really diserable as this will mean the most common use case (pip installing to default site) will not longer work. We could still make this available for the adventurous.
2. Distribute poetry as a zipapp. This will still have issues.
3. Use a standalone executable per supported platform (this is painful) Experimental beeware based cross-platform build #2848.

It's quite ironic for a tool which promotes reproducible builds through the use of a lockfile that it should suffer from breakages caused by installing itself via pip.

I would argue that this is not one of the objectives for Poetry. The poetry.lock file is by design poetry specific for the environments managed by poetry. For python applications (yes such as Poetry), this is useful and there are a lot of downstream benefits for this. #2778, https://github.com/nix-community/poetry2nix etc.

Poetry, as per best practice, defines the acceptable dependency version versions as package metadata that gets used by pip etc. In the end, Poetry is just another python package.

[awilkins]

So I went with "what if the install script contained a full frozen install spec".

Sadly it wasn't entirely 100% simple - mostly because of packages with complicated constraint history

However...

Here's what I arrived at : install-poetry-f.sh

The process was basically ...

• Clone poetry at the tag for 1.1.11
• Fix the lock file (which is out of sync with the project)
  poetry lock --no-update
• Do
  poetry export
• Embed the output in the script as a multiline string
• Hack the install_poetry routine around to write the content to a temp file to pass to pip
• Hack the export output around until it worked, this was mostly messing with
  • secretstorage via keyring, which needed constraints adding to stop it trying to install both 2.3.1 and 3.3.1
  • six, for which I removed the bonkers complicated constraint altogether because it was trying to install 0.9 from another transitive dependency
  • cachecontrol which needed the [filecache] extra identifier adding to it, or it wasn't matching up
  • Last but not least, add poetry and it's hashes

The result is a script that installs a working version of Poetry that doesn't have the bugged version of cachecontrol. Doesn't look nice, can only do v1.1.11, but reliably installs the version locked when the branch was tagged, which is the important part - I can embed this script in my CI pipeline and not worry about whether the version of Poetry it installs is going to break in a few weeks.

[tom-bowles]

Re poetry export - if it's not in a usable state it would be good to have a mention of that in the docs. What kinds of problems might be expected? I've been using it with --without-hashes without any issues so far (before I added --without-hashes there were some obscure errors when pip installing from the generated file that I haven't had a chance to look into yet). I'm not using the new poetry export plugin but rather the command that's built into poetry (for now).

I agree that using the wheel's version constraints for locked dependency information would not be the right thing. The dependency constraints and the fixed versions in the lock file have different purposes and the distinction shouldn't be muddied. Putting the concrete dependencies in the wheel would be a hack that would probably be regretted later.

Let me state the requirement here in a way that I think/hope nobody disagrees with:

The lock file contains a set of concrete dependencies that satisfy the logical dependency constraints. It is clearly desirable to use this fixed set of dependencies during development and testing, and this is a (if not the) primary purpose of poetry.

In general, when installing a package from pip it is not possible to require that the exact dependency versions in the lock file are used. If all packages were to do that, it would be practically impossible to install any combination of packages into the same venv, due to version conflicts.

In some deployment scenarios, though, you can fix all dependency versions because you know that no other conflicting packages are going to be installed. This is true in at least the following cases:

1. The package is being installed in a virtualenv that has been created solely for the purpose of installing this package and its dependencies and will never be used for anything else.

2. The package is being installed without a venv but it is a special case and will be the only package installed in this way.

(The vast majority of cases are 1 but, for example, we do 2 for poetry itself in CI where we pip install poetry separately into each python version we have available on an agent.)

(Note that the use of poetry-managed venvs during development/CI is an example of 1, but far from the only one.)

In deployment scenarios like this, it is generally desirable to install fixed versions of dependencies that match those used during development/testing.

I'll also state another thing that I think/hope is uncontroversial:

Poetry is a development tool. It belongs on development and CI machines but should not be required on production machines. It needs to be possible to install without poetry.

If these things are agreed, then it seems like generating a requirements.txt is a very natural way of achieving the goal. It captures the fixed dependencies separately from the version constraints, and does so in a way that can be installed without poetry.

The only problem with it, really, is that it requires everybody that does this to solve some problems in their own way. If poetry were to use this approach itself in install-poetry.py, for example, the poetry project would need to establish a way to make the requirements.txt corresponding to each poetry version available to the install script, and the install script would need special code to obtain and make use of the requirements.txt. This would all be out-of-band from pypi and wouldn't be reusable for other projects.

I think doing better than this, however, is probably not a challenge poetry can solve on its own. Ideally, the locked dependency information would be published along with the package through the usual channels. Probably the easiest way to achieve that would be to embed the information in the package, but separately from the version constraints. There would need to be options at installation time to choose whether these "preferred" dependency versions should be used. There are probably a load of subtleties to consider.

I think something like this could be a great solution, and if the underlying mechanisms existed it would clearly be desirable for poetry to support them. However, they don't and I suspect anything poetry does on its own to address this requirement (beyond fully supporting poetry export, ideally as a native feature rather than plugin) would be a hack.

[tom-bowles]

With the above said (about the general topic of fixing dependencies for installations of applications developed with poetry), I think poetry itself is facing the same challenge as other projects here and, like them, should ideally do the best it can with the mechanisms that are currently available.

Poetry is a fast moving project and is heavily depended upon as a key piece of development/CI infrastructure. Users need to be able to pin the installed version as that is frequently the only short-term remediation for problems introduced in new releases. They also need to be able to pin third party dependencies to avoid incompatibilities/bugs introduced into those. The facts are:

1. The --version parameter on install-poetry.py is insufficient for this as it only affects the version of poetry itself. Much of the implementation is in poetry-core and --version doesn't even allow you to use an old version of that, let alone any of the third-party dependencies.

2. The most common (and recommended) installation approach is to use install-poetry.py. This creates a venv just for poetry, so it is clearly possible for it to pin dependencies.

My suggestion would be to commit a requirements-x.y.z.txt file for each released version of poetry into a folder at the root of the repo, so it can be fetched with a raw.githubusercontent.com in the same way as people are invited to fetch install-poetry.py itself. install-poetry.py could then use the most recent of these (or a specific version if --version is specified) when it does its pip install.

[awilkins]

What kinds of problems might be expected?

I ran into a few when doing the script linked above (which includes the hashes) ; I think they might be quite difficult to solve though (and a lot of that is down to a lack of rigour in published package constraints).

e.g. keyring wants secretstorage, but only on Linux. There are three possible versions of keyring that poetry can use, depending on your python version ; 2.7 uses 18.0.1, 3.5 uses 20.0.1 and 3.6 and up use 21.8.0. Older versions of keyring were less explicit about the markers, which leads to the export having two versions of secretstorage in it.

There seems to be a bit of confusion with the boolean logic surrounding the sys_platform marker in the second line, but the main problem here is that the first line has no python version marker - if you're installing on Python 3.6 or above, both mutually exclusive versions of the package are selected, which means there is no install solution and "strict" pip will fail.

secretstorage==2.3.1; sys_platform == "linux"
secretstorage==3.3.1; python_version >= "3.6" and python_version < "4.0" and sys_platform == "linux" or sys_platform == "linux"

Another one was that the cachecontrol entry is lacking it's extras, which leads to it being identified as a different package to cachecontrol[filecache] ; when that turns up elsewhere in the dependency graph, pip assigns a different version to it, and thus can't install again. Adding the [filecache] to the exported line fixes this.

cachecontrol==0.12.6; (python_version >= "2.7" and python_full_version < "3.0.0") or (python_full_version >= "3.4.0")

---

In some deployment scenarios, though, you can fix all dependency versions

As you note, the Poetry installer itself is one of these scenarios, because it takes pains to create it's own cloistered venv, specifically to avoid the problem of other package installs interfering with it. It's slightly more complex, because dependency versions depend on your environment.. unless you know up front what your environment will be and can rely on it not changing, which makes the solution simpler.

make the requirements.txt corresponding to each poetry version available to the install script .. make use of the requirements.txt

This in itself is not a terribly tough problem : the gist linked above does this (by just embedding it in a variable). The fiddle is overcoming the exceptions, which wasn't terribly time consuming but did involve some manual guesses. In an ideal world, the script would then enter a test matrix of the supported install platforms to check it worked.

commit a requirements-x.y.z.txt file for each released version of poetry

Not keen on this for the same reason I'm not keen on curling a shell script from the master branch of a public git repo in my build pipeline ; nothing stops a bad actor with commit access changing that file. I've written the exported requirements (with hashes) directly into the script, which means I can copy it and be sure that what it installs will never change and can't be amended by a third party. The requirements doubled the size of script, which is to say they add about 25kB per version.

[tom-bowles]

If you embed the requirements information in the installation script you will need to ensure that old versions of the installation script can be obtained. I'm not sure I like the fact that that links the requirements information to the installation script implementation such that you can't install older versions with the updated installation logic (problematic if bugs are found in the installation script).

I understand the desire not to curl install scripts to bash, but the install script has to be published some way. If you want to review the contents of the script and then be sure it won't change when you invoke it repeatedly you need to put a copy of it somewhere you control. The same is true of the requirements.txt (whether it's in the script or not). It should actually be a bit easier if the requirements and script are separate as that way there won't be a whole new script to review each time you adopt a new version, just a new requirements.txt.

Anyway, like you I am setting up a custom installation script based on requirements.txt (but not embedded). Something along these lines is clearly needed.

[awilkins]

If you embed the requirements information in the installation script you will need to ensure that old versions of the installation script can be obtained.

Or do as I've done and embed the requirements in a map of version : requirements: I've only done 1.1.11, but you could in principal go back and do all the old releases too and have versions that were not broken by the releases of later deps like poetry==1.1.8 which has poetry-core==1.0.4 in it's lockfile but was then broken by the release of poetry-core==1.0.5.

Yes, this makes a giant ugly script with a big map of text data embedded in it. But also a single thing you can copy, hash, sign, whatever, for your build process rather than 2 things. By all means keep all the data in separate files and maybe have a process that squishes them onto the end of the install script for a release.

[tom-bowles]

Sure, I don't really mind as long as I can install old versions.

[novas0x2a]

This has been dormant for a while; did anyone make any progress on a methodology to install poetry with pinned, hashed transitive deps? It does sort of seem like things got worse, running poetry export on 1.4.0 unfortunately fails.

[Kache]

Related to this, I've always wished poetry would store the version of itself in the lockfile it generated, similar to how Ruby's bundler does. It helps esp when jumping between projects that may use different poetry versions that behave differently.

Poetry wouldn't have to be really strict about using a non-matching version, but it provides info for the user to act on. e.g. I've got multiple poetry installations via pipx.