poetry install with namespace packages has race condition leading to broken files
#9158
Comments
StefanBRas
added
kind/bug
if you have two packages that are trying to install different files in the same place then this is doomed anyway: which even should win? An approach that downloaded to a temporary file and then went |
|
Since poetry gives a warning when it overwrites a file, I'd say that the expected behavior is that the last file "wins" - knowing that this is not optimal. Does it handle differently if the files are identical? Ideally the files are the same yes - in my specific case they only differ in comments. |
|
I'd say the meaning of the warning is roughly "this is surprising, if something goes wrong you'll know where to look". In modern namespace packaging the "duplicate" Anyway I've suggested a fix that you might try: do that, and if it works out, submit a merge request! |
|
Would it not be the same if I just changed it to not overwrite but just ignore the file? Then it would be "first file wins" and since the order (seems) random that's effectively the same as "last file wins" but just with less of a hassle. |
|
I reckon I would buy that |
|
Thank you for raising this issue, Stefan. I understand from the PR discussions that a fix is not trivial. Meanwhile does anyone know what would be the workaround? postinstall step is mentioned above, but how would that look? I am considering setting parallel to false or modern installation to false, but I don't know if that would shortcircut the problematic vode path 🤔 Could someone confirm? Lower performance is better than undeterministic results |
|
Update: skipping the modern installation works, though unclear when will it be removed #8987 |
How to skip the modern installation? Currently, my workaround is to retry. Usually, it works the second time. |
|
https://python-poetry.org/docs/configuration/#installermodern-installation Set it to false. Has worked 100% since activating it in May. The strategy to try one more time on the other hand failed often in the CI. |
Description
If multiple packages uses the same namespace and have files with the same name, they will overwrite each other.
This is by design, as seen in this line.
However sometimes it seems to write to the file at the same time, leading to a file that is a mix of multiple files and often not valid python. My guess is that there needs to be a file lock here
Workarounds
Use a post install script to fix the files. But this is very manual
Poetry Installation Method
pip
Operating System
macOS 14.4
Poetry Version
1.8.2
Poetry Configuration
Python Sysconfig
Example pyproject.toml
If you keep running
It will eventually fail.
Poetry Runtime Logs
The text was updated successfully, but these errors were encountered: