Crashing with packaging.version.InvalidVersion: Invalid version: 'cpython'

[detrin]

What pipdeptree version are you using?

2.23.0

Are you running pipdeptree in a virtual environment?

No

Describe the problem

When running pipdeptree in databricks notebook I am getting the following error

Warning!!! Duplicate package metadata found:
"/usr/local/lib/python3.8/dist-packages"
  filelock                         3.6.0            (using 3.0.12, "/databricks/python3/lib/python3.8/site-packages")
  pip                              21.0.1           (using 24.1.2, "/databricks/python3/lib/python3.8/site-packages")
  wheel                            0.36.2           (using 0.36.2, "/databricks/python3/lib/python3.8/site-packages")
"/usr/lib/python3/dist-packages"
  distro                           1.4.0            (using 1.9.0, "/databricks/python3/lib/python3.8/site-packages")
  requests                         2.22.0           (using 2.32.3, "/databricks/python3/lib/python3.8/site-packages")
  certifi                          2019.11.28       (using 2020.12.5, "/databricks/python3/lib/python3.8/site-packages")
  chardet                          3.0.4            (using 4.0.0, "/databricks/python3/lib/python3.8/site-packages")
  urllib3                          1.25.8           (using 1.25.11, "/databricks/python3/lib/python3.8/site-packages")
  idna                             2.8              (using 2.10, "/databricks/python3/lib/python3.8/site-packages")
  six                              1.14.0           (using 1.15.0, "/databricks/python3/lib/python3.8/site-packages")
NOTE: This warning isn't a failure warning.
------------------------------------------------------------------------
Traceback (most recent call last):
  File "/databricks/python3/bin/pipdeptree", line 8, in <module>
    sys.exit(main())
  File "/databricks/python3/lib/python3.8/site-packages/pipdeptree/__main__.py", line 36, in main
    tree = PackageDAG.from_pkgs(pkgs)
  File "/databricks/python3/lib/python3.8/site-packages/pipdeptree/_models/dag.py", line 63, in from_pkgs
    req = next(requires_iterator)
  File "/databricks/python3/lib/python3.8/site-packages/pipdeptree/_models/package.py", line 118, in requires
    if not req.marker or req.marker.evaluate():
  File "/databricks/python3/lib/python3.8/site-packages/packaging/markers.py", line 325, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/databricks/python3/lib/python3.8/site-packages/packaging/markers.py", line 225, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
  File "/databricks/python3/lib/python3.8/site-packages/packaging/markers.py", line 183, in _eval_op
    return spec.contains(lhs, prereleases=True)
  File "/databricks/python3/lib/python3.8/site-packages/packaging/specifiers.py", line 552, in contains
    normalized_item = _coerce_version(item)
  File "/databricks/python3/lib/python3.8/site-packages/packaging/specifiers.py", line 28, in _coerce_version
    version = Version(version)
  File "/databricks/python3/lib/python3.8/site-packages/packaging/version.py", line 202, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: 'cpython'

[kemzeb]

What I think is happening is that you have some package metadata that contains an invalid requirement string. More specifically, a requirement string that contains an invalid environment marker.

I think the wrong marker variable is being used somewhere e.g. 'cpython'==python_version in some METADATA file.

I was able to reproduce the same exact stack trace after adding the following METADATA file into my environment:

Metadata-Version: 2.1
Name: fake 
Version: 1.2.3
Requires-Dist: covdefaults>=2.3; 'cpython' == python_version

The stack trace:

Traceback (most recent call last):
  File "/home/vscode/.local/bin/pipdeptree", line 8, in <module>
    sys.exit(main())
  File "/workspaces/pipdeptree/src/pipdeptree/__main__.py", line 36, in main
    tree = PackageDAG.from_pkgs(pkgs)
  File "/workspaces/pipdeptree/src/pipdeptree/_models/dag.py", line 63, in from_pkgs
    req = next(requires_iterator)
  File "/workspaces/pipdeptree/src/pipdeptree/_models/package.py", line 118, in requires
    if not req.marker or req.marker.evaluate():
  File "/home/vscode/.local/lib/python3.9/site-packages/packaging/markers.py", line 327, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/home/vscode/.local/lib/python3.9/site-packages/packaging/markers.py", line 227, in _evaluate_markers
    groups[-1].append(_eval_op(lhs_value, op, rhs_value))
  File "/home/vscode/.local/lib/python3.9/site-packages/packaging/markers.py", line 184, in _eval_op
    return spec.contains(lhs, prereleases=True)
  File "/home/vscode/.local/lib/python3.9/site-packages/packaging/specifiers.py", line 552, in contains
    normalized_item = _coerce_version(item)
  File "/home/vscode/.local/lib/python3.9/site-packages/packaging/specifiers.py", line 28, in _coerce_version
    version = Version(version)
  File "/home/vscode/.local/lib/python3.9/site-packages/packaging/version.py", line 202, in __init__
    raise InvalidVersion(f"Invalid version: '{version}'")
packaging.version.InvalidVersion: Invalid version: 'cpython'

It looks like the problem lies in _eval_op(), where it would create a valid Specifier e.g. Specifier('==3.12'), but it does not attempt to check if lhs (which equals "cpython") is a valid Version, which will cause it to raise InvalidVersion.

Because of this, I think this maybe an upstream bug and should be reported here.

[kemzeb]

Actually, it looks like this issue is already being tracked in pypa/packaging#767.

[detrin]

Thank you, do you think this could be somehow catched when evaluating the dependencies? Even if one package has bad definition in metadata it's maybe not reason to not show the partial tree of dependencies anyway. In my case I was looking what was requiring sklearn>=1.x.x, in the end I created an extra env in databricks and used the sklearn==0.23.x for loading of a model.

[kemzeb]

Seeing that the upstream issue has been open for some time, I think it would make sense for me to implement a workaround until they are able to fix it. I'll get to it when I have the time.