[Bug report] Pip install cocoex fails

[brockho]

Describe the bug
I cannot install cocoex from the development branch on my Windows machine (via pip install .):

Processing c:\users\dimo\desktop\coco\code-experiments\build\python
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy>=1.24.0 in c:\users\dimo\anaconda3new\lib\site-packages (from cocoex==2.6.4.dev216+g647b7193) (1.24.3)
Building wheels for collected packages: cocoex
  Building wheel for cocoex (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for cocoex (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [30 lines of output]
      running bdist_wheel
      running build
      running build_py
      copying src\cocoex\exceptions.py -> build\lib.win-amd64-cpython-311\cocoex
      copying src\cocoex\solvers.py -> build\lib.win-amd64-cpython-311\cocoex
      copying src\cocoex\utilities.py -> build\lib.win-amd64-cpython-311\cocoex
      copying src\cocoex\_version.py -> build\lib.win-amd64-cpython-311\cocoex
      copying src\cocoex\__init__.py -> build\lib.win-amd64-cpython-311\cocoex
      running egg_info
      writing src\cocoex.egg-info\PKG-INFO
      writing dependency_links to src\cocoex.egg-info\dependency_links.txt
      writing requirements to src\cocoex.egg-info\requires.txt
      writing top-level names to src\cocoex.egg-info\top_level.txt
      reading manifest file 'src\cocoex.egg-info\SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no previously-included files found matching 'src\cocoex\interface.c'
      warning: no previously-included files found matching 'src\cocoex\function.c'
      warning: no previously-included files matching '*.pyc' found anywhere in distribution
      warning: no previously-included files matching '__pycache__' found anywhere in distribution
      adding license file 'LICENSE'
      writing manifest file 'src\cocoex.egg-info\SOURCES.txt'
      copying src\cocoex\coco.c -> build\lib.win-amd64-cpython-311\cocoex
      copying src\cocoex\coco.h -> build\lib.win-amd64-cpython-311\cocoex
      copying src\cocoex\function.pyx -> build\lib.win-amd64-cpython-311\cocoex
      copying src\cocoex\interface.pyx -> build\lib.win-amd64-cpython-311\cocoex
      running build_ext
      Compiling src/cocoex/interface.pyx because it depends on C:\Users\dimo\AppData\Local\Temp\pip-build-env-42elwzsh\overlay\Lib\site-packages\Cython\Includes\libc\string.pxd.
      [1/1] Cythonizing src/cocoex/interface.pyx
      building 'cocoex.interface' extension
      error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for cocoex
Failed to build cocoex
ERROR: Could not build wheels for cocoex, which is required to install pyproject.toml-based projects

I uninstalled and then re-installed the Microsoft C++ Build Tools via the given webpage, but it still doesn't work.

Your System (please complete the following information when it seems relevant):

• Operating system: Windows 10
• COCO version: latest development branch
• Python 3.11.5
• I have gcc.exe (MinGW.org GCC-6.3.0-1) 6.3.0 also, maybe this is the problem?

[olafmersmann]

When installing the Microsoft C++ Build Tools, did you check the optional "MSVCv142 - VS 2019 C++ ..." component?

It's kind of non-obvious, that "just" installing the build tools is not enough.
After installing with the above optional component pip install . "just works" on a clean Windows 11 laptop for me.

[brockho]

Thanks for this hint, but I also found this already before with @nikohansen. Unfortunately, even after uninstalling, reinstalling, and rebooting the laptop, it is still not working (same error message). I guess, it is related to some paths not set correctly. I will try out a bit more things.

[olafmersmann]

Can you try running python -m build . instead of pip install .. That should give you much more info about the build process and might give a hint why it cannot find the MSVC compiler.

Also, have you tried updating the setuptools package (pip install --upgrade setuptools)? Apparently the heuristics to find compilers are continually improved.

[brockho]

Thanks, @olafmersmann. With your suggestion, I found at least that I had incompatible versions of requests_mock, clyent, pyyaml, and conda-repo-cli (I have no idea why). I now upgraded them all, such that pip install --upgrade setuptools runs without complaining. Same problem though for pip install .. The new suggestion python -m build . gives indeed more information, but the actual error is still the same:

PS C:\Users\dimo\Desktop\coco\code-experiments\build\python> pip install --upgrade build
Collecting build
  Downloading build-1.2.1-py3-none-any.whl.metadata (4.3 kB)
Requirement already satisfied: packaging>=19.1 in c:\users\dimo\anaconda3new\lib\site-packages (from build) (23.1)
Collecting pyproject_hooks (from build)
  Downloading pyproject_hooks-1.1.0-py3-none-any.whl.metadata (1.3 kB)
Requirement already satisfied: colorama in c:\users\dimo\anaconda3new\lib\site-packages (from build) (0.4.6)
Downloading build-1.2.1-py3-none-any.whl (21 kB)
Downloading pyproject_hooks-1.1.0-py3-none-any.whl (9.2 kB)
Installing collected packages: pyproject_hooks, build
Successfully installed build-1.2.1 pyproject_hooks-1.1.0
PS C:\Users\dimo\Desktop\coco\code-experiments\build\python> python -m build .
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - cython     ~= 3.0
  - numpy
  - setuptools >= 67.0
  - wheel
* Getting build dependencies for sdist...
running egg_info
writing src\cocoex.egg-info\PKG-INFO
writing dependency_links to src\cocoex.egg-info\dependency_links.txt
writing requirements to src\cocoex.egg-info\requires.txt
writing top-level names to src\cocoex.egg-info\top_level.txt
reading manifest file 'src\cocoex.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'src\cocoex\interface.c'
warning: no previously-included files found matching 'src\cocoex\function.c'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
adding license file 'LICENSE'
writing manifest file 'src\cocoex.egg-info\SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing src\cocoex.egg-info\PKG-INFO
writing dependency_links to src\cocoex.egg-info\dependency_links.txt
writing requirements to src\cocoex.egg-info\requires.txt
writing top-level names to src\cocoex.egg-info\top_level.txt
reading manifest file 'src\cocoex.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'src\cocoex\interface.c'
warning: no previously-included files found matching 'src\cocoex\function.c'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
adding license file 'LICENSE'
writing manifest file 'src\cocoex.egg-info\SOURCES.txt'
running check
creating cocoex-2.6.4.dev216+g647b7193
creating cocoex-2.6.4.dev216+g647b7193\src
creating cocoex-2.6.4.dev216+g647b7193\src\cocoex
creating cocoex-2.6.4.dev216+g647b7193\src\cocoex.egg-info
creating cocoex-2.6.4.dev216+g647b7193\test
copying files to cocoex-2.6.4.dev216+g647b7193...
copying LICENSE -> cocoex-2.6.4.dev216+g647b7193
copying MANIFEST.in -> cocoex-2.6.4.dev216+g647b7193
copying README.md -> cocoex-2.6.4.dev216+g647b7193
copying README.txt -> cocoex-2.6.4.dev216+g647b7193
copying pyproject.toml -> cocoex-2.6.4.dev216+g647b7193
copying setup.py -> cocoex-2.6.4.dev216+g647b7193
copying src/cocoex\coco.c -> cocoex-2.6.4.dev216+g647b7193\src/cocoex
copying src/cocoex\function.pyx -> cocoex-2.6.4.dev216+g647b7193\src/cocoex
copying src/cocoex\interface.pyx -> cocoex-2.6.4.dev216+g647b7193\src/cocoex
copying src\cocoex\__init__.py -> cocoex-2.6.4.dev216+g647b7193\src\cocoex
copying src\cocoex\_version.py -> cocoex-2.6.4.dev216+g647b7193\src\cocoex
copying src\cocoex\coco.c -> cocoex-2.6.4.dev216+g647b7193\src\cocoex
copying src\cocoex\coco.h -> cocoex-2.6.4.dev216+g647b7193\src\cocoex
copying src\cocoex\exceptions.py -> cocoex-2.6.4.dev216+g647b7193\src\cocoex
copying src\cocoex\function.pyx -> cocoex-2.6.4.dev216+g647b7193\src\cocoex
copying src\cocoex\interface.pyx -> cocoex-2.6.4.dev216+g647b7193\src\cocoex
copying src\cocoex\solvers.py -> cocoex-2.6.4.dev216+g647b7193\src\cocoex
copying src\cocoex\utilities.py -> cocoex-2.6.4.dev216+g647b7193\src\cocoex
copying src\cocoex.egg-info\PKG-INFO -> cocoex-2.6.4.dev216+g647b7193\src\cocoex.egg-info
copying src\cocoex.egg-info\SOURCES.txt -> cocoex-2.6.4.dev216+g647b7193\src\cocoex.egg-info
copying src\cocoex.egg-info\dependency_links.txt -> cocoex-2.6.4.dev216+g647b7193\src\cocoex.egg-info
copying src\cocoex.egg-info\requires.txt -> cocoex-2.6.4.dev216+g647b7193\src\cocoex.egg-info
copying src\cocoex.egg-info\top_level.txt -> cocoex-2.6.4.dev216+g647b7193\src\cocoex.egg-info
copying test\bbob2009_testcases.txt -> cocoex-2.6.4.dev216+g647b7193\test
copying test\bbob2009_testcases2.txt -> cocoex-2.6.4.dev216+g647b7193\test
copying test\test_function.py -> cocoex-2.6.4.dev216+g647b7193\test
copying test\test_regression.py -> cocoex-2.6.4.dev216+g647b7193\test
copying test\test_suite.py -> cocoex-2.6.4.dev216+g647b7193\test
copying src\cocoex.egg-info\SOURCES.txt -> cocoex-2.6.4.dev216+g647b7193\src\cocoex.egg-info
Writing cocoex-2.6.4.dev216+g647b7193\setup.cfg
Creating tar archive
removing 'cocoex-2.6.4.dev216+g647b7193' (and everything under it)
* Building wheel from sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - cython     ~= 3.0
  - numpy
  - setuptools >= 67.0
  - wheel
* Getting build dependencies for wheel...
running egg_info
writing src\cocoex.egg-info\PKG-INFO
writing dependency_links to src\cocoex.egg-info\dependency_links.txt
writing requirements to src\cocoex.egg-info\requires.txt
writing top-level names to src\cocoex.egg-info\top_level.txt
reading manifest file 'src\cocoex.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'src\cocoex\interface.c'
warning: no previously-included files found matching 'src\cocoex\function.c'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
adding license file 'LICENSE'
writing manifest file 'src\cocoex.egg-info\SOURCES.txt'
* Installing packages in isolated environment:
  - wheel
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-cpython-311
creating build\lib.win-amd64-cpython-311\cocoex
copying src\cocoex\exceptions.py -> build\lib.win-amd64-cpython-311\cocoex
copying src\cocoex\solvers.py -> build\lib.win-amd64-cpython-311\cocoex
copying src\cocoex\utilities.py -> build\lib.win-amd64-cpython-311\cocoex
copying src\cocoex\_version.py -> build\lib.win-amd64-cpython-311\cocoex
copying src\cocoex\__init__.py -> build\lib.win-amd64-cpython-311\cocoex
running egg_info
writing src\cocoex.egg-info\PKG-INFO
writing dependency_links to src\cocoex.egg-info\dependency_links.txt
writing requirements to src\cocoex.egg-info\requires.txt
writing top-level names to src\cocoex.egg-info\top_level.txt
reading manifest file 'src\cocoex.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no previously-included files found matching 'src\cocoex\interface.c'
warning: no previously-included files found matching 'src\cocoex\function.c'
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '__pycache__' found anywhere in distribution
adding license file 'LICENSE'
writing manifest file 'src\cocoex.egg-info\SOURCES.txt'
copying src\cocoex\coco.c -> build\lib.win-amd64-cpython-311\cocoex
copying src\cocoex\coco.h -> build\lib.win-amd64-cpython-311\cocoex
copying src\cocoex\function.pyx -> build\lib.win-amd64-cpython-311\cocoex
copying src\cocoex\interface.pyx -> build\lib.win-amd64-cpython-311\cocoex
running build_ext
Compiling src/cocoex/interface.pyx because it changed.
[1/1] Cythonizing src/cocoex/interface.pyx
building 'cocoex.interface' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

ERROR Backend subprocess exited when trying to invoke build_wheel

[olafmersmann]

Some more hints about my setup:

• MSVC is installed to

  C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807

  on my computer. The compiler used is "C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.40.33807\bin\HostX86\x64\cl.exe and is not in my PATH. I also don't see any environment variables that point to the MSVC directory.

• Not sure if it matters, but I run all commands from PowerShell and not cmd.exe.

[brockho]

Together with @olafmersmann, we found some other inconsistencies today (for example two BuildTools installations on my machine of which only one had the ""MSVCv142 - VS 2019 C++ ..." component selected), but this did still not solve the problem. Also nothing seems problematic in my Path variable.

What worked like a charm though (as a workaround for now) is to download and extract the wheels from the latest build (during our CI), for example from here: https://github.com/numbbo/coco/actions/runs/9515689063 (list of the built artifacts can be found at the bottom of the page) and then pip install .\cocoex-2.6.4.dev231+gfa992e72-cp311-cp311-win_amd64.whl [use the right wheel depending on your architecture, for me, this was for python 3.11 (i.e. 'cp311' in the file name) and for Windows 64bit].

[nikohansen]

Is this still open?

[brockho]

Unfortunately, yes. I just tried today to fully uninstall my python/Anaconda/VSTools and re-install everything, but with no success :-(