dependency resolution behaves differently with requirements.txt vs listing on the command-line

[stefansjs]

Description

When doing pip install -r requirements.txt the resolver notices a version conflict and fails quickly, whereas listing the same requirements on the command line causes an extended un-resolved backtracking. pip install -r failed in 20s on my machine, while the pip install command is still running after an hour so far.

Expected behavior

pip install -r and pip install behave the same with respect to the resolver

pip version

21.2.4

Python version

3.9

OS

Mac OS 10.15.6

How to Reproduce

1. pip install -r requirements.txt
2. pip install 'numpy>1.18' -e .

requirements.txt contains:

-e .
numpy>1.19

setup.py contains

from setuptools import setup
setup(
          install_requires=[
              "configparser>=3.5",
              "kafka>=1.3.5; python_version<'3.7'",
              "kafka-python>=1.4.3; python_version>='3.7'",
              "confluent-kafka[avro]==1.3.0",
              "fastavro",  # Whatever's needed by confluent-kafka[avro]
              'avro-python3==1.9.2.1',  # confluent-kafka[avro] defines a dependency on 1.9.2, but only 1.9.2.1 works
              "isort>=4",

              "scipy>=1.2.1, <=1.4.1; python_version<'3.7'",
              "scipy",
              "numpy>=1.17; python_version<'3.9'",
              "numpy>1.17.2; python_version>='3.9'",
              "teamcity-messages>=1.17",
              "vertica_python>=0.9.1",  # Literally any version since we only need imports to succeed
              "mido>=1.2.8",
              "requests<=2.23.0",  # We've tested with up to 2.23.0, but snowflake-connector-python requires less than that :'(
              "python-Levenshtein>=0.12",
              "aubio==0.4.9",
              "future>=0.17",
              "six>=1.11",
              "datasketch>=1.5",
              "jams>=0.3",
              "python-slugify>=3",
          ],
)

Output

$ pip install -r requirements.txt
Looking in indexes: https://pypi.org/simple, http://sfw.sf.smle.co:8080/sfw/pip
Looking in links: file:///Users/stefansullivan/code/test_pip_resolution.
Obtaining file:///Users/stefansullivan/code/test_pip_resolution (from -r requirements.in (line 3))
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Obtaining file:///<my project dir> (from -r requirements.in (line 4))
Requirement already satisfied: bokeh in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 5)) (2.4.0)
Requirement already satisfied: matplotlib in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 6)) (3.4.3)
Requirement already satisfied: pandas in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 7)) (1.3.3)
Requirement already satisfied: brigit in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 8)) (1.3.0)
Requirement already satisfied: tqdm in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 9)) (4.62.3)
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r requirements.in (line 10)) (1.21.2)
Collecting confluent-kafka[avro]==1.3.0
  Using cached confluent_kafka-1.3.0-cp39-cp39-macosx_10_15_x86_64.whl
Collecting mido<=1.2.9,>=1.2.8
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/mido/mido-1.2.9-py2.py3-none-any.whl (52 kB)
Collecting isort==4.3.21
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/isort/isort-4.3.21-py2.py3-none-any.whl (42 kB)
Collecting datasketch==1.5.1
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/datasketch/datasketch-1.5.1-py2.py3-none-any.whl (73 kB)
Collecting kafka==1.3.5
  Using cached kafka-1.3.5-py2.py3-none-any.whl (207 kB)
Collecting requests<=2.23.0,==2.22.0
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/requests/requests-2.22.0-py2.py3-none-any.whl (57 kB)
Collecting jams==0.3.4
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/jams/jams-0.3.4-py3-none-any.whl (64 kB)
Collecting avro-python3==1.9.2.1
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/avro-python3/avro_python3-1.9.2.1-py3-none-any.whl (43 kB)
Collecting configparser==3.5.0
  Using cached configparser-3.5.0-py3-none-any.whl
Collecting scipy<=1.4.1,>=1.2.1
  Using cached scipy-1.4.1.tar.gz (24.6 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting vertica-python<=0.10.2,>=0.9.1
  Using cached vertica_python-0.10.2-py2.py3-none-any.whl (152 kB)
Collecting python-Levenshtein==0.12.0
  Using cached python_Levenshtein-0.12.0-cp39-cp39-macosx_10_15_x86_64.whl
Collecting aubio==0.4.9
  Using cached aubio-0.4.9+smule.1-cp39-cp39-macosx_10_15_x86_64.whl

The conflict is caused by:
    The user requested numpy>1.18
    unknown 0.1 depends on numpy==1.17.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict





$ pip install 'numpy>1.18' -e .
Looking in indexes: https://pypi.org/simple, http://sfw.sf.smle.co:8080/sfw/pip
Looking in links: .
Obtaining file:///Users/stefansullivan/code/test_pip_resolution
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Obtaining file:///<my project dir>
Requirement already satisfied: bokeh in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (2.4.0)
Requirement already satisfied: matplotlib in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (3.4.3)
Requirement already satisfied: pandas in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.3.3)
Requirement already satisfied: brigit in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.3.0)
Requirement already satisfied: tqdm in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (4.62.3)
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.21.2)
Requirement already satisfied: pillow>=7.1.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (8.3.2)
Requirement already satisfied: Jinja2>=2.9 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (3.0.1)
Requirement already satisfied: tornado>=5.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (6.1)
Requirement already satisfied: packaging>=16.8 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (21.0)
Requirement already satisfied: PyYAML>=3.10 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (5.4.1)
Requirement already satisfied: typing-extensions>=3.10.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from bokeh) (3.10.0.2)
Requirement already satisfied: kiwisolver>=1.0.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from matplotlib) (1.3.2)
Requirement already satisfied: python-dateutil>=2.7 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from matplotlib) (2.8.2)
Requirement already satisfied: pyparsing>=2.2.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from matplotlib) (2.4.7)
Requirement already satisfied: cycler>=0.10 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from matplotlib) (0.10.0)
Requirement already satisfied: pytz>=2017.3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from pandas) (2021.1)
Requirement already satisfied: log-colorizer in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from brigit) (1.8.6)
Collecting configparser==3.5.0
  Using cached configparser-3.5.0-py3-none-any.whl
Collecting datasketch==1.5.1
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/datasketch/datasketch-1.5.1-py2.py3-none-any.whl (73 kB)
Collecting teamcity-messages==1.17
  Using cached teamcity_messages-1.17-py3-none-any.whl
Collecting requests<=2.23.0,==2.22.0
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/requests/requests-2.22.0-py2.py3-none-any.whl (57 kB)
Collecting isort==4.3.21
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/isort/isort-4.3.21-py2.py3-none-any.whl (42 kB)
Collecting six<=1.14.0,>=1.11.0
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
Collecting numpy>1.18
  Using cached numpy-1.21.2-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.21.1-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
  Using cached numpy-1.21.0-cp39-cp39-macosx_10_9_x86_64.whl (16.9 MB)
  Using cached numpy-1.20.3-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.2-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.1-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.0-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl (15.6 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.4-cp39-cp39-macosx_10_9_x86_64.whl (15.4 MB)
  Using cached numpy-1.19.3-cp39-cp39-macosx_10_9_x86_64.whl (15.9 MB)
  Using cached numpy-1.19.2.zip (7.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached numpy-1.19.1.zip (7.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.19.0.zip (7.3 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.18.5.zip (5.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached numpy-1.18.4.zip (5.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached numpy-1.18.3.zip (5.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached numpy-1.18.2.zip (5.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached numpy-1.18.1.zip (5.4 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
INFO: pip is looking at multiple versions of tqdm to determine which version is compatible with other requirements. This could take a while.
Collecting tqdm
  Using cached tqdm-4.62.3-py2.py3-none-any.whl (76 kB)
  Using cached tqdm-4.62.2-py2.py3-none-any.whl (76 kB)
  Using cached tqdm-4.62.1-py2.py3-none-any.whl (76 kB)
  Using cached tqdm-4.62.0-py2.py3-none-any.whl (76 kB)
  Using cached tqdm-4.61.2-py2.py3-none-any.whl (76 kB)
  Using cached tqdm-4.61.1-py2.py3-none-any.whl (75 kB)
  Using cached tqdm-4.61.0-py2.py3-none-any.whl (75 kB)
INFO: pip is looking at multiple versions of tqdm to determine which version is compatible with other requirements. This could take a while.
  Using cached tqdm-4.60.0-py2.py3-none-any.whl (75 kB)
  Using cached tqdm-4.59.0-py2.py3-none-any.whl (74 kB)
  Using cached tqdm-4.58.0-py2.py3-none-any.whl (73 kB)
  Using cached tqdm-4.57.0-py2.py3-none-any.whl (72 kB)
  Using cached tqdm-4.56.2-py2.py3-none-any.whl (72 kB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached tqdm-4.56.1-py2.py3-none-any.whl (72 kB)
  Using cached tqdm-4.56.0-py2.py3-none-any.whl (72 kB)
  Using cached tqdm-4.55.2-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.55.1-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.55.0-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.54.1-py2.py3-none-any.whl (69 kB)
  Using cached tqdm-4.54.0-py2.py3-none-any.whl (69 kB)
  Using cached tqdm-4.53.0-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.52.0-py2.py3-none-any.whl (71 kB)
  Using cached tqdm-4.51.0-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.50.2-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.50.1-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.50.0-py2.py3-none-any.whl (70 kB)
  Using cached tqdm-4.49.0-py2.py3-none-any.whl (69 kB)
  Using cached tqdm-4.48.2-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.48.1-py2.py3-none-any.whl (68 kB)
  Using cached tqdm-4.48.0-py2.py3-none-any.whl (67 kB)
  Using cached tqdm-4.47.0-py2.py3-none-any.whl (66 kB)
  Using cached tqdm-4.46.1-py2.py3-none-any.whl (63 kB)
  Using cached tqdm-4.46.0-py2.py3-none-any.whl (63 kB)
  Using cached tqdm-4.45.0-py2.py3-none-any.whl (60 kB)
  Using cached tqdm-4.44.1-py2.py3-none-any.whl (60 kB)
  Using cached tqdm-4.44.0-py2.py3-none-any.whl (60 kB)
  Using cached tqdm-4.43.0-py2.py3-none-any.whl (59 kB)
  Using cached tqdm-4.42.1-py2.py3-none-any.whl (59 kB)
  Using cached tqdm-4.42.0-py2.py3-none-any.whl (59 kB)
  Using cached tqdm-4.41.1-py2.py3-none-any.whl (56 kB)
  Using cached tqdm-4.41.0-py2.py3-none-any.whl (56 kB)
  Using cached tqdm-4.40.2-py2.py3-none-any.whl (55 kB)
  Using cached tqdm-4.40.1-py2.py3-none-any.whl (55 kB)
  Using cached tqdm-4.40.0-py2.py3-none-any.whl (54 kB)
  Using cached tqdm-4.39.0-py2.py3-none-any.whl (53 kB)
  Using cached tqdm-4.38.0-py2.py3-none-any.whl (53 kB)
  Using cached tqdm-4.37.0-py2.py3-none-any.whl (53 kB)
  Using cached tqdm-4.36.1-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.36.0-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.35.0-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.34.0-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.33.0-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.32.2-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.32.1-py2.py3-none-any.whl (49 kB)
  Using cached tqdm-4.32.0-py2.py3-none-any.whl (49 kB)
  Using cached tqdm-4.31.1-py2.py3-none-any.whl (48 kB)
  Using cached tqdm-4.31.0-py2.py3-none-any.whl (48 kB)
  Using cached tqdm-4.30.0-py2.py3-none-any.whl (47 kB)
  Using cached tqdm-4.29.1-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.29.0-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.28.1-py2.py3-none-any.whl (45 kB)
  Using cached tqdm-4.28.0-py2.py3-none-any.whl (43 kB)
  Using cached tqdm-4.27.0-py2.py3-none-any.whl (44 kB)
  Using cached tqdm-4.26.0-py2.py3-none-any.whl (43 kB)
  Using cached tqdm-4.25.0-py2.py3-none-any.whl (43 kB)
  Using cached tqdm-4.24.0-py2.py3-none-any.whl (43 kB)
  Using cached tqdm-4.23.4-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.23.3-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.23.2-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.23.1-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.23.0-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.22.0-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.21.0-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.20.0-py2.py3-none-any.whl (41 kB)
  Using cached tqdm-4.19.9-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.19.8-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.19.7-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.19.6-py2.py3-none-any.whl (52 kB)
  Using cached tqdm-4.19.5-py2.py3-none-any.whl (51 kB)
  Using cached tqdm-4.19.4-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.19.2-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.19.1.post1-py2.py3-none-any.whl (51 kB)
  Using cached tqdm-4.19.1-py2.py3-none-any.whl (50 kB)
  Using cached tqdm-4.18.0-py2.py3-none-any.whl (48 kB)
  Using cached tqdm-4.17.1-py2.py3-none-any.whl (47 kB)
  Using cached tqdm-4.17.0-py2.py3-none-any.whl (47 kB)
  Using cached tqdm-4.16.0-py2.py3-none-any.whl (47 kB)
  Using cached tqdm-4.15.0-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.14.0-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.13.0-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.12.0-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.11.2-py2.py3-none-any.whl (46 kB)
  Using cached tqdm-4.11.1-py2.py3-none-any.whl (45 kB)
  Using cached tqdm-4.11.0-py2.py3-none-any.whl (45 kB)
  Using cached tqdm-4.10.0-py2.py3-none-any.whl (43 kB)
  Using cached tqdm-4.9.0-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.8.4-py2.py3-none-any.whl (39 kB)
  Using cached tqdm-4.8.3-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.8.2-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.8.1-py2.py3-none-any.whl (42 kB)
  Using cached tqdm-4.7.6-py2.py3-none-any.whl (40 kB)
  Using cached tqdm-4.7.4-py2.py3-none-any.whl (40 kB)
  Using cached tqdm-4.7.2-py2.py3-none-any.whl (40 kB)
  Using cached tqdm-4.7.1-py2.py3-none-any.whl (39 kB)
  Using cached tqdm-4.7.0-py2.py3-none-any.whl (39 kB)
  Using cached tqdm-4.6.2-py2.py3-none-any.whl (38 kB)
  Using cached tqdm-4.6.1-py2.py3-none-any.whl (38 kB)
  Using cached tqdm-4.5.2-py2.py3-none-any.whl (34 kB)
  Using cached tqdm-4.5.0-py2.py3-none-any.whl (34 kB)
  Using cached tqdm-4.4.3-py2.py3-none-any.whl (33 kB)
  Using cached tqdm-4.4.1-py2.py3-none-any.whl (32 kB)
  Using cached tqdm-4.4.0-py2.py3-none-any.whl (32 kB)
  Using cached tqdm-4.1.0-py2.py3-none-any.whl (29 kB)
  Using cached tqdm-3.8.0-py2.py3-none-any.whl (28 kB)
  Using cached tqdm-3.7.1-py2.py3-none-any.whl (28 kB)
  Using cached tqdm-3.7.0-py2.py3-none-any.whl (28 kB)
  Using cached tqdm-3.4.0-py2.py3-none-any.whl (26 kB)
  Using cached tqdm-3.1.4-py2.py3-none-any.whl (19 kB)
  Using cached tqdm-3.1.3-py2.py3-none-any.whl (19 kB)
  Using cached tqdm-2.2.4-py2.py3-none-any.whl (17 kB)
  Using cached tqdm-2.2.3-py2.py3-none-any.whl (17 kB)
  Using cached tqdm-2.0.0-py2.py3-none-any.whl (16 kB)
  Using cached tqdm-1.0.tar.gz (1.8 kB)
INFO: pip is looking at multiple versions of brigit to determine which version is compatible with other requirements. This could take a while.
Collecting brigit
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/brigit/brigit-1.3.0-py2.py3-none-any.whl (8.6 kB)
  Using cached brigit-1.2.tar.gz (2.5 kB)
  Using cached brigit-1.1.tar.gz (2.5 kB)
  Using cached brigit-1.0.tar.gz (2.5 kB)
INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.
Collecting pandas
  Using cached pandas-1.3.3-cp39-cp39-macosx_10_9_x86_64.whl (11.6 MB)
INFO: pip is looking at multiple versions of brigit to determine which version is compatible with other requirements. This could take a while.
  Using cached pandas-1.3.2-cp39-cp39-macosx_10_9_x86_64.whl (11.6 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached pandas-1.3.1-cp39-cp39-macosx_10_9_x86_64.whl (11.3 MB)
  Using cached pandas-1.3.0-cp39-cp39-macosx_10_9_x86_64.whl (11.6 MB)
  Using cached pandas-1.2.5-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
  Using cached pandas-1.2.4-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
  Using cached pandas-1.2.3-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
INFO: pip is looking at multiple versions of pandas to determine which version is compatible with other requirements. This could take a while.
  Using cached pandas-1.2.2-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
  Using cached pandas-1.2.1-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
  Using cached pandas-1.2.0-cp39-cp39-macosx_10_9_x86_64.whl (10.7 MB)
  Using cached pandas-1.1.5-cp39-cp39-macosx_10_9_x86_64.whl (10.3 MB)
  Using cached pandas-1.1.4-cp39-cp39-macosx_10_9_x86_64.whl (10.3 MB)
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached pandas-1.1.3-cp39-cp39-macosx_10_9_x86_64.whl (10.3 MB)
  Using cached pandas-1.1.2.tar.gz (5.2 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached pandas-1.1.1.tar.gz (5.2 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached pandas-1.1.0.tar.gz (5.2 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached pandas-1.0.5.tar.gz (5.0 MB)
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  Using cached pandas-1.0.4.tar.gz (5.0 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-1.0.3.tar.gz (5.0 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-1.0.2.tar.gz (5.0 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-1.0.1.tar.gz (4.9 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-1.0.0.tar.gz (4.8 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-0.25.3.tar.gz (12.6 MB)
  Using cached pandas-0.25.2.tar.gz (12.6 MB)
  Using cached pandas-0.25.1.tar.gz (12.6 MB)
  Using cached pandas-0.25.0.tar.gz (12.6 MB)
  Using cached pandas-0.24.2.tar.gz (11.8 MB)
  Using cached pandas-0.24.1.tar.gz (11.8 MB)
  Using cached pandas-0.24.0.tar.gz (11.8 MB)
  Using cached pandas-0.23.4.tar.gz (10.5 MB)
  Using cached pandas-0.23.3.tar.gz (10.5 MB)
  Using cached pandas-0.23.2.tar.gz (10.0 MB)
  Using cached pandas-0.23.1.tar.gz (13.1 MB)
  Using cached pandas-0.23.0.tar.gz (13.1 MB)
  Using cached pandas-0.22.0.tar.gz (11.3 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-0.21.1.tar.gz (11.3 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-0.21.0.tar.gz (11.3 MB)
  Installing build dependencies: started
  Installing build dependencies: still running...
  Installing build dependencies: finished with status 'error'
  Using cached pandas-0.20.3.tar.gz (10.4 MB)
  Using cached pandas-0.20.2.tar.gz (10.3 MB)
  Using cached pandas-0.20.1.tar.gz (10.3 MB)
  Using cached pandas-0.20.0.tar.gz (10.3 MB)
  Using cached pandas-0.19.2.tar.gz (9.2 MB)

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

The resolver since 21.2 attempts to prefer the order in which you give it requirements.

In your requirements file example your order is effectively:

{your non-numpy requirements}
{your numpy requirement}

In your command line example your order is effectively:

{your numpy requirement}
{your non-numpy requirements}

Because in the former example numpy is attempted to be pinned last and in the latter example numpy is attempted to pinned first they are going to have very different dependency resolution behavior, especially if your requirements need to be backtracked. You should be able to confirm this by swapping the order you give the requirements in either of your examples.

I am hoping that the performance in both cases should be acceptable once this lands: #10479 but I don't have time to test right now. I'll see if I can test tomorrow evening.

[stefansjs]

So that's an interesting detail that I wasn't aware of. However, I can be quite certain that the order is not the culprit here. I've redone the test using only bash invocation differences and I can see the behavior is quite different. With a -r command it takes about 40s, and with command-line requirements it takes 3m40s.

3 minutes might not seem like an insane amount of time to wait (IMHO it is, but that's a different discussion), but this is actually a reduced example from what I was doing which was installing many more packages. They should behave the same when provided with a requirements file vs command-line arguments, especially if order is important.

It appears that when presented on the command-line, the resolver starts doing backtracking on the numpy package, whereas the requirements file fails much more quickly.

Here's another invocation that does a better job of showing the difference where I simply invoke

pip install -r <(echo "-e .\nnumpy>1.18")
pip install -e . "numpy>1.18"

$ time pip install -r <(echo "-e .\nnumpy>1.18")
Looking in indexes: https://pypi.org/simple, http://sfw.sf.smle.co:8080/sfw/pip
Obtaining file:///Users/stefansullivan/code/test_pip_resolution (from -r /dev/fd/11 (line 1))
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from -r /dev/fd/11 (line 2)) (1.21.2)
Collecting cysignals==1.6.5
  Using cached cysignals-1.6.5-cp39-cp39-macosx_10_15_x86_64.whl
Collecting python-Levenshtein==0.12.0
  Using cached python_Levenshtein-0.12.0-cp39-cp39-macosx_10_15_x86_64.whl
Requirement already satisfied: aubio==0.4.9 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from unknown==0.1->-r /dev/fd/11 (line 1)) (0.4.9+smule.1)
Requirement already satisfied: fastavro in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from unknown==0.1->-r /dev/fd/11 (line 1)) (1.4.5)
Collecting configparser==3.5.0
  Using cached configparser-3.5.0-py3-none-any.whl
Collecting scipy<=1.4.1,>=1.2.1
  Using cached scipy-1.4.1.tar.gz (24.6 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting isort==4.3.21
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/isort/isort-4.3.21-py2.py3-none-any.whl (42 kB)
     |████████████████████████████████| 42 kB 923 kB/s 
Collecting requests<=2.23.0,==2.22.0
  Downloading http://sfw.sf.smle.co:8080/sfw/pip/requests/requests-2.22.0-py2.py3-none-any.whl (57 kB)
     |████████████████████████████████| 57 kB 824 kB/s 
ERROR: Cannot install -r /dev/fd/11 (line 1) and numpy>1.18 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested numpy>1.18
    unknown 0.1 depends on numpy==1.17.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
pip install -r <(echo "-e .\nnumpy>1.18")  28.69s user 4.88s system 85% cpu 39.083 total


$ time pip install -e . "numpy>1.18"
Looking in indexes: https://pypi.org/simple, http://sfw.sf.smle.co:8080/sfw/pip
Obtaining file:///Users/stefansullivan/code/test_pip_resolution
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.21.2)
Collecting teamcity-messages==1.17
  Using cached teamcity_messages-1.17-py3-none-any.whl
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
Collecting numpy>1.18
  Using cached numpy-1.21.2-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.21.1-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
  Using cached numpy-1.21.0-cp39-cp39-macosx_10_9_x86_64.whl (16.9 MB)
  Using cached numpy-1.20.3-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.2-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.1-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.0-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl (15.6 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.4-cp39-cp39-macosx_10_9_x86_64.whl (15.4 MB)
  Using cached numpy-1.19.3-cp39-cp39-macosx_10_9_x86_64.whl (15.9 MB)
  Using cached numpy-1.19.2.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.19.1.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.19.0.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.18.5.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.4.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.3.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.2.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.1.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
ERROR: Cannot install numpy>1.18 and unknown==0.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested numpy>1.18
    unknown 0.1 depends on numpy==1.17.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
pip install -e . "numpy>1.18"  164.19s user 44.28s system 94% cpu 3:40.53 total

[notatallshaw]

I can't reproduce your issue but I don't have an OS X device to test on and I don't know what or how you've configured this second index "http://sfw.sf.smle.co:8080/sfw/pip" which definitely appears to be contributing to the backtracking.

If I could reproduce the issue I would attempt to look at what is getting ordered different when self._get_preference is called: https://github.com/pypa/pip/blob/21.2.4/src/pip/_vendor/resolvelib/resolvers.py#L365 which ultimately calls this logic: https://github.com/pypa/pip/blob/21.2.4/src/pip/_internal/resolution/resolvelib/provider.py#L69 . Debugging why the resolver does something is unfortunately a little tricky.

[stefansjs]

It's an apache http server with vanilla directory listings as a simple repository server. What's your hypothesis about that index? I don't see it contributing to the longer run from the log I posted.

If I run the command again without the extra index server in my pip.conf, I still see the behavior

$ time pip install -e . "numpy>1.18"       
Obtaining file:///Users/stefansullivan/code/test_pip_resolution
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.21.2)
Requirement already satisfied: configparser>=3.5 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (5.0.2)
Requirement already satisfied: confluent-kafka[avro]==1.3.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.3.0)
Requirement already satisfied: fastavro in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.4.5)
Requirement already satisfied: avro-python3==1.9.2.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.9.2.1)
Requirement already satisfied: isort>=4 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (5.9.3)
Requirement already satisfied: scipy in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.7.1)
Requirement already satisfied: teamcity-messages>=1.17 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.29)
Requirement already satisfied: vertica_python>=0.9.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.0.1)
Requirement already satisfied: mido>=1.2.8 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.2.10)
Requirement already satisfied: requests<=2.23.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (2.23.0)
Requirement already satisfied: python-Levenshtein>=0.12 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (0.12.2)
Requirement already satisfied: aubio==0.4.9 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (0.4.9+smule.1)
Requirement already satisfied: future>=0.17 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (0.18.2)
Requirement already satisfied: six>=1.11 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.16.0)
Requirement already satisfied: datasketch>=1.5 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.5.3)
Requirement already satisfied: jams>=0.3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (0.3.4)
Requirement already satisfied: python-slugify>=3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (5.0.2)
Requirement already satisfied: kafka-python>=1.4.3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (2.0.2)
Requirement already satisfied: mir-eval>=0.5 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jams>=0.3->UNKNOWN==0.0.0) (0.6)
Requirement already satisfied: sortedcontainers>=2.0.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jams>=0.3->UNKNOWN==0.0.0) (2.4.0)
Requirement already satisfied: decorator in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jams>=0.3->UNKNOWN==0.0.0) (5.1.0)
Requirement already satisfied: jsonschema>=3.0.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jams>=0.3->UNKNOWN==0.0.0) (3.2.0)
Requirement already satisfied: pandas in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jams>=0.3->UNKNOWN==0.0.0) (1.3.3)
Requirement already satisfied: attrs>=17.4.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jsonschema>=3.0.0->jams>=0.3->UNKNOWN==0.0.0) (21.2.0)
Requirement already satisfied: setuptools in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jsonschema>=3.0.0->jams>=0.3->UNKNOWN==0.0.0) (57.4.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from jsonschema>=3.0.0->jams>=0.3->UNKNOWN==0.0.0) (0.18.0)
Requirement already satisfied: text-unidecode>=1.3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from python-slugify>=3->UNKNOWN==0.0.0) (1.3)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from requests<=2.23.0->UNKNOWN==0.0.0) (1.25.11)
Requirement already satisfied: idna<3,>=2.5 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from requests<=2.23.0->UNKNOWN==0.0.0) (2.10)
Requirement already satisfied: chardet<4,>=3.0.2 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from requests<=2.23.0->UNKNOWN==0.0.0) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from requests<=2.23.0->UNKNOWN==0.0.0) (2020.12.5)
Requirement already satisfied: python-dateutil>=1.5 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from vertica_python>=0.9.1->UNKNOWN==0.0.0) (2.8.2)
Requirement already satisfied: pytz>=2017.3 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from pandas->jams>=0.3->UNKNOWN==0.0.0) (2021.1)
Installing collected packages: UNKNOWN
  Running setup.py develop for UNKNOWN
Successfully installed UNKNOWN-0.0.0
pip install -e . "numpy>1.18"  3.78s user 0.79s system 88% cpu 5.163 total
(mir39) ~/code/test_pip_resolution % time pip install -e . "numpy>1.18"
Obtaining file:///Users/stefansullivan/code/test_pip_resolution
Requirement already satisfied: numpy>1.18 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (1.21.2)
Collecting cysignals==1.6.5
  Using cached cysignals-1.6.5-cp39-cp39-macosx_10_15_x86_64.whl
Collecting configparser==3.5.0
  Using cached configparser-3.5.0.tar.gz (39 kB)
Collecting kafka==1.3.5
  Using cached kafka-1.3.5-py2.py3-none-any.whl (207 kB)
Requirement already satisfied: confluent-kafka[avro]==1.3.0 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.3.0)
Requirement already satisfied: fastavro in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.4.5)
Requirement already satisfied: avro-python3==1.9.2.1 in /Users/stefansullivan/venvs/mir39/lib/python3.9/site-packages (from UNKNOWN==0.0.0) (1.9.2.1)
Collecting isort==4.3.21
  Using cached isort-4.3.21-py2.py3-none-any.whl (42 kB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
Collecting numpy>1.18
  Using cached numpy-1.21.2-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.21.1-cp39-cp39-macosx_10_9_x86_64.whl (17.0 MB)
  Using cached numpy-1.21.0-cp39-cp39-macosx_10_9_x86_64.whl (16.9 MB)
  Using cached numpy-1.20.3-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.2-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.1-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
  Using cached numpy-1.20.0-cp39-cp39-macosx_10_9_x86_64.whl (16.1 MB)
INFO: pip is looking at multiple versions of numpy to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.5-cp39-cp39-macosx_10_9_x86_64.whl (15.6 MB)
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
  Using cached numpy-1.19.4-cp39-cp39-macosx_10_9_x86_64.whl (15.4 MB)
  Using cached numpy-1.19.3-cp39-cp39-macosx_10_9_x86_64.whl (15.9 MB)
  Using cached numpy-1.19.2.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.19.1.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.19.0.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
  Using cached numpy-1.18.5.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.4.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.3.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.2.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
  Using cached numpy-1.18.1.zip (5.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
ERROR: Cannot install numpy>1.18 and unknown==0.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested numpy>1.18
    unknown 0.0.0 depends on numpy==1.17.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
pip install -e . "numpy>1.18"  341.89s user 91.06s system 93% cpu 7:44.40 total

[stefansjs]

I took a quick look (without debugging). It looks to me like the difference in behavior here https://github.com/pypa/pip/blob/21.2.4/src/pip/_internal/cli/req_command.py#L363 vs https://github.com/pypa/pip/blob/21.2.4/src/pip/_internal/cli/req_command.py#L387 which calls out to https://github.com/pypa/pip/blob/21.2.4/src/pip/_internal/req/constructors.py#L430 is notable. In one case pip does

        for req in args:
            req_to_add = install_req_from_line(
                req,
                None,
                isolated=options.isolated_mode,
                use_pep517=options.use_pep517,
                user_supplied=True,
            )
            requirements.append(req_to_add)

        for req in options.editables:
            req_to_add = install_req_from_editable(
                req,
                user_supplied=True,
                isolated=options.isolated_mode,
                use_pep517=options.use_pep517,
            )
            requirements.append(req_to_add)

and in the other, it calls install_req_from_parsed_requirements (I presume line-by-line without reordering them), which itself does:

    if parsed_req.is_editable:
        req = install_req_from_editable(
            parsed_req.requirement,
            comes_from=parsed_req.comes_from,
            use_pep517=use_pep517,
            constraint=parsed_req.constraint,
            isolated=isolated,
            user_supplied=user_supplied,
        )

    else:
        req = install_req_from_line(
            parsed_req.requirement,
            comes_from=parsed_req.comes_from,
            use_pep517=use_pep517,
            isolated=isolated,
            options=parsed_req.options,
            constraint=parsed_req.constraint,
            line_source=parsed_req.line_source,
            user_supplied=user_supplied,
        )

IIUC, it looks like passing arguments on the command line orders non-editables before editables, whereas the requirements file is parsed line by line and added in the order in the file. That of course assumes that the downstream resolver depends on order and doesn't do any further reordering.

If I'm right, I see a few solutions

1. one solution is to modify the requirements parser to put editables after non-editables. That seems like it's extremely likely to break somebody, but I don't know to what degree requirements.txt ordering is a guaranteed API. It's probably the simplest fix.

Index: src/pip/_internal/req/req_file.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/pip/_internal/req/req_file.py b/src/pip/_internal/req/req_file.py
--- a/src/pip/_internal/req/req_file.py	(revision 0981e071bee8ac3ca2497813fa0863005033c6c2)
+++ b/src/pip/_internal/req/req_file.py	(date 1633472570176)
@@ -328,6 +328,8 @@
     def _parse_and_recurse(
         self, filename: str, constraint: bool
     ) -> Iterator[ParsedLine]:
+        editable_requirements = []
+        
         for line in self._parse_file(filename, constraint):
             if not line.is_requirement and (
                 line.opts.requirements or line.opts.constraints
@@ -353,8 +355,12 @@
                     )
 
                 yield from self._parse_and_recurse(req_path, nested_constraint)
+            elif line.is_editable:
+                editable_requirements.append(line)
             else:
                 yield line
+                
+        yield from editable_requirements
 
     def _parse_file(self, filename: str, constraint: bool) -> Iterator[ParsedLine]:
         _, content = get_file_content(filename, self._session)

2. Alternatively, it seems like either doing some more complex handling of arguments ordering together with options would be necessary. Given the use of argparse, this seems a little more involved but not insurmountable. You'd just have to find the order of args and -e opts in sys.argv and reorder the args + options.editables according to their command-line order.

3. Finally, (still assuming that I'm right about the underlying bug), a third option would be to have the install command inspect the order of requirements and re-order editables according to whatever ordering is most correct.

What's the correct ordering of requirements? Is it by command-line order, or should editables always be last? Is it considered ok to have requirements.txt install differently than command-line invocations re: editable installs? Is it considered a guarantee that requirements.txt ordering implies a specific resolution ordering, and does that same guarantee apply to command-line invocations?

For my mental model, it seems like they should be the same. But I completely understand the engineering decision to not offer it as a guarantee. It also seems like this may have been a difference for years, but now that resolvers routinely take minutes and hours to find/fail to find installation candidates, suddenly the route to failure is relevant. Not sure if this is going to end up causing other differences in resolution.

[uranusjr]

Assuming your analysis is correct (I'm quite confident it is, but didn't actually check, and nobody should trust my memory on this kind of stuff 🙂), ordering requirements by command line order is the most sensible solution. There is no "correct" ordering because everything could be editable or not depending on the situation (conceptually editable-ness is completely orthogontal to dependency resolution), so we can only use an ordering that surprises the least people.

[stefansjs]

Also, I made a repository to make testing a little easier https://github.com/stefansjs/test_pip_install_differences.

[pfmoore]

I suspect that at least part of the problem here is that on the command line, -e file is an option rather than an argument. So it will be collected differently by our argument parsing code, and any "original" order of interspersed options and arguments might be hard to recover. Also, needing to retain that order may constrain us when we finally get around to moving to a newer, better, command line parsing library.

Unfortunately, the decision to use an option to signal an editable install is long-established, and would be very hard to change at this point.

(Please check before acting on the above, this is purely from memory and I'm not that familiar with our option parsing code).