requires-plugins breaks poetry installation on Windows

[pseusys]

Description

I have a poetry project, which I install and test on Windows machine in GitHub Actions.
Recently, I tried using the new requires-plugins feature for poethepoet and poetry-plugin-export plugins installation.
The pyproject.toml file looks somewhat like this:

...
[tool.poetry.requires-plugins]
poethepoet = "^0.32.1"
poetry-plugin-export = "^1.9.0"
...

However, instead of expected installation, what I receive is an error:

Ensuring that the Poetry plugins required by the project are available...
The following Poetry plugins are required by the project but are not installed in Poetry's environment:
  - poethepoet (>=0.32.1,<0.33.0)
  - poetry-plugin-export (>=1.9.0,<2.0.0)
Installing Poetry plugins only for the current project...
Updating dependencies
Resolving dependencies...

Package operations: 4 installs, 0 updates, 0 removals

  - Installing pastel (0.2.1)
  - Installing pyyaml (6.0.2)
  - Installing poethepoet (0.32.1)
  - Installing poetry-plugin-export (1.9.0)

  ValueError

  path is on mount 'C:', start on mount 'D:'

  at C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\ntpath.py:747 in relpath
      743|         path_abs = abspath(normpath(path))
      744|         start_drive, start_rest = splitdrive(start_abs)
      745|         path_drive, path_rest = splitdrive(path_abs)
      746|         if normcase(start_drive) != normcase(path_drive):
    > 747|             raise ValueError("path is on mount %r, start on mount %r" % (
      748|                 path_drive, start_drive))
      749| 
      750|         start_list = [x for x in start_rest.split(sep) if x]
      751|         path_list = [x for x in path_rest.split(sep) if x]

Cannot install poethepoet.


Failed to install required Poetry plugins

Workarounds

The plugins still can be installed traditionally, using poetry self add ....
However, I still would like to try the new feature... it works on Linux so nice!

Poetry Installation Method

pip

Operating System

windiws-latest (that is Windows Server 2022 OS Version: 10.0.20348 Build 2966)

Poetry Version

Poetry (version 2.0.1)

Poetry Configuration

cache-dir = "C:\\Users\\runneradmin\\AppData\\Local\\pypoetry\\Cache"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = true
keyring.enabled = true
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}\\virtualenvs"  # C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false

Python Sysconfig

sysconfig.log

Platform: "win-amd64"
Python version: "3.10"
Current installation scheme: "nt"
Paths:
data = "C:\hostedtoolcache\windows\Python\3.10.11\x64"
include = "C:\hostedtoolcache\windows\Python\3.10.11\x64\Include"
platinclude = "C:\hostedtoolcache\windows\Python\3.10.11\x64\Include"
platlib = "C:\hostedtoolcache\windows\Python\3.10.11\x64\Lib\site-packages"
platstdlib = "C:\hostedtoolcache\windows\Python\3.10.11\x64\Lib"
purelib = "C:\hostedtoolcache\windows\Python\3.10.11\x64\Lib\site-packages"
scripts = "C:\hostedtoolcache\windows\Python\3.10.11\x64\Scripts"
stdlib = "C:\hostedtoolcache\windows\Python\3.10.11\x64\Lib"
Variables:
BINDIR = "C:\hostedtoolcache\windows\Python\3.10.11\x64"
BINLIBDEST = "C:\hostedtoolcache\windows\Python\3.10.11\x64\Lib"
EXE = ".exe"
EXT_SUFFIX = ".cp310-win_amd64.pyd"
INCLUDEPY = "C:\hostedtoolcache\windows\Python\3.10.11\x64\Include"
LIBDEST = "C:\hostedtoolcache\windows\Python\3.10.11\x64\Lib"
SO = ".cp310-win_amd64.pyd"
TZPATH = ""
VERSION = "310"
abiflags = ""
base = "C:\hostedtoolcache\windows\Python\3.10.11\x64"
exec_prefix = "C:\hostedtoolcache\windows\Python\3.10.11\x64"
installed_base = "C:\hostedtoolcache\windows\Python\3.10.11\x64"
installed_platbase = "C:\hostedtoolcache\windows\Python\3.10.11\x64"
platbase = "C:\hostedtoolcache\windows\Python\3.10.11\x64"
platlibdir = "lib"
prefix = "C:\hostedtoolcache\windows\Python\3.10.11\x64"
projectbase = "C:\hostedtoolcache\windows\Python\3.10.11\x64"
py_version = "3.10.11"
py_version_nodot = "310"
py_version_nodot_plat = "310"
py_version_short = "3.10"
srcdir = "C:\hostedtoolcache\windows\Python\3.10.11\x64"
userbase = "C:\Users\runneradmin\AppData\Roaming\Python"

Paste the output of 'python -m sysconfig', over this line.

Example pyproject.toml

[tool.poetry.requires-plugins]
poethepoet = "^0.32.1"
poetry-plugin-export = "^1.9.0"

Poetry Runtime Logs

poetry-runtime.log

output.txt

Paste the output of 'poetry -vvv <command>', over this line.

[abn]

This is likely a Windows file path issue like when the drive is a link and relpath is called.

Can you provide full logs with -vvv please?

[j-adamczyk]

I face the same problems on GitHub Actions workflow (where I don't fully control the environment).

I tried to add poetry-dynamic-versioning plugin:

[tool.poetry.requires-plugins]
poetry-dynamic-versioning = { version = ">=1.0.0,<2.0.0", extras = ["plugin"] }

My error is basically the same:

Run poetry sync --with test
Creating virtualenv skfp in D:\a\scikit-fingerprints\scikit-fingerprints\.venv
Ensuring that the Poetry plugins required by the project are available...
The following Poetry plugins are required by the project but are not installed in Poetry's environment:
  - poetry-dynamic-versioning[plugin] (>=1.0.0,<2.0.0)
Installing Poetry plugins only for the current project...
Updating dependencies
Resolving dependencies...

Package operations: 4 installs, 0 updates, 0 removals

  - Installing markupsafe (3.0.2)
  - Installing dunamai (1.23.0)
  - Installing jinja2 (3.1.5)

  ValueError

  path is on mount 'C:', start on mount 'D:'

  at <frozen ntpath>:766 in relpath

Cannot install dunamai.


Failed to install required Poetry plugins
Error: Process completed with exit code 1.

Workflow definition: https://github.com/scikit-fingerprints/scikit-fingerprints/blob/eaa1499a12db49e128808f920df61d8004ef396e/.github/workflows/python-publish.yml.

Full stacktrace with -vvv option: https://github.com/scikit-fingerprints/scikit-fingerprints/actions/runs/12743574960/job/35513793864?pr=346

[abn]

Thank you @j-adamczyk this is helpful. I can confirm the issue is as I suspected.

https://github.com/scikit-fingerprints/scikit-fingerprints/actions/runs/12743574960/job/35513793864?pr=346#step:7:1250

I am not entirely sure if this is something Poetry can fix as the failure is due to the way action runner sets up the mounts (for the user data directory). Might be able to make the code resilient to that, lets see.

[mtkennerly]

Could you use abspath as a fallback when relpath fails? I found a similar case here: https://github.com/okken/pytest-check/pull/87/files

[abn]

Probably, will have to check the impacted code first, I also would like to know if there are other places where we might be exposed to this issue.

[BrianPugh]

Here's a full stack trace from a recent GA run on windows:

9  ~\poetry\venv\lib\site-packages\poetry\installation\executor.py:274 in _execute_operation
      272| 
      273|             try:
    > 274|                 result = self._do_execute_operation(operation)
      275|             except EnvCommandError as e:
      276|                 if e.e.returncode == -2:

  8  ~\poetry\venv\lib\site-packages\poetry\installation\executor.py:385 in _do_execute_operation
      383|             return 0
      384| 
    > 385|         result: int = getattr(self, f"_execute_{method}")(operation)
      386| 
      387|         if result != 0:

  7  ~\poetry\venv\lib\site-packages\poetry\installation\executor.py:509 in _execute_install
      507| 
      508|     def _execute_install(self, operation: Install | Update) -> int:
    > 509|         status_code = self._install(operation)
      510| 
      511|         self._save_url_reference(operation)

  6  ~\poetry\venv\lib\site-packages\poetry\installation\executor.py:562 in _install
      560|                 self._remove(operation.initial_package)
      561| 
    > 562|             self._wheel_installer.install(archive)
      563|         finally:
      564|             if cleanup_archive:

  5  ~\poetry\venv\lib\site-packages\poetry\installation\wheel_installer.py:109 in install
      107|             )
      108| 
    > 109|             install(
      110|                 source=source,
      111|                 destination=destination,

  4  ~\poetry\venv\lib\site-packages\installer\_core.py:131 in install
      129| 
      130|     written_records.append((root_scheme, RecordEntry(record_file_path, None, None)))
    > 131|     destination.finalize_installation(
      132|         scheme=root_scheme,
      133|         record_file_path=record_file_path,

  3  ~\poetry\venv\lib\site-packages\installer\destinations.py:278 in finalize_installation
      276| 
      277|         record_list = list(records)
    > 278|         with construct_record_file(record_list, prefix_for_scheme) as record_stream:
      279|             self.write_to_fs(
      280|                 scheme, record_file_path, record_stream, is_executable=False

  2  ~\poetry\venv\lib\site-packages\installer\utils.py:211 in construct_record_file
      209|     writer = csv.writer(stream, delimiter=",", quotechar='"', lineterminator="\n")
      210|     for scheme, record in records:
    > 211|         writer.writerow(record.to_row(prefix_for_scheme(scheme)))
      212|     stream.seek(0)
      213|     return stream.detach()

  1  ~\poetry\venv\lib\site-packages\installer\destinations.py:271 in prefix_for_scheme
      269|             if file_scheme == scheme:
      270|                 return None
    > 271|             path = os.path.relpath(
      272|                 self.scheme_dict[file_scheme],
      273|                 start=self.scheme_dict[scheme],

  ValueError

  path is on mount 'C:', start on mount 'D:'

  at C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\ntpath.py:703 in relpath
      699|         path_abs = abspath(normpath(path))
      700|         start_drive, start_rest = splitdrive(start_abs)
      701|         path_drive, path_rest = splitdrive(path_abs)
      702|         if normcase(start_drive) != normcase(path_drive):
    > 703|             raise ValueError("path is on mount %r, start on mount %r" % (
      704|                 path_drive, start_drive))
      705| 
      706|         start_list = [x for x in start_rest.split(sep) if x]
      707|         path_list = [x for x in path_rest.split(sep) if x]

Cannot install dunamai.

[dalito]

In the past several other projects hit the issue as well, which was solved, e.g. here https://github.com/Lightning-AI/pytorch-lightning/pull/16164/files, by special-casing os.relpath on windows.

[aaron-siegel]

We are having the same problem with our Windows runners on Github Actions while attempting to upgrade to Poetry 2.x.

The plugin we are trying to use is poetry-dynamic-versioning. We're using the latest version of poetry (2.1.1) and poetry-dynamic-versioning (1.7.1). We have two different configurations, running python 3.9 and 3.12 respectively. Same error in each case.

Same error message that others have reported:

Installing C:\Users\runneradmin\miniconda3\envs\pxt\Scripts\dunamai.exe over existing file

  ValueError

  path is on mount 'C:', start on mount 'D:'

  at ~\miniconda3\envs\pxt\lib\ntpath.py:703 in relpath
      699|         path_abs = abspath(normpath(path))
      700|         start_drive, start_rest = splitdrive(start_abs)
      701|         path_drive, path_rest = splitdrive(path_abs)
      702|         if normcase(start_drive) != normcase(path_drive):
    > 703|             raise ValueError("path is on mount %r, start on mount %r" % (
      704|                 path_drive, start_drive))
      705| 
      706|         start_list = [x for x in start_rest.split(sep) if x]
      707|         path_list = [x for x in path_rest.split(sep) if x]

Cannot install dunamai.

Please advise - what is the officially suggested solution / workaround? It is not clear to me from the thread on this ticket. This is blocking our migration to Poetry 2.x.

Thank you!
Aaron

[radoering]

what is the officially suggested solution / workaround?

Especially in CI, you can work around this issue by just installing the required plugins explicitly instead of relying on requires-plugins. (If the plugins are already installed, requires-plugins only verifies that they are installed.) The exact method depends on how you installed Poetry, see https://python-poetry.org/docs/plugins/#using-plugins

[aaron-siegel]

Well that worked. Thank you! But it makes me kind of sad, because proper management of plugin dependencies is one of the improvements in Poetry 2.x that I was excited about.

I hope that the poetry team is able to prioritize a fix for this.

Thanks again for your help,
Aaron

Especially in CI, you can work around this issue by just installing the required plugins explicitly instead of relying on requires-plugins. (If the plugins are already installed, requires-plugins only verifies that they are installed.) The exact method depends on how you installed Poetry, see https://python-poetry.org/docs/plugins/#using-plugins