[BUG]: CMake FindPython issues on CMake>3.23, and Python linking issue in debug mode

[patrikhuber]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

2.13.6

Problem description

Setting:

• Compiling my Python bindings using C++/CMake. Using CMAKE_TOOLCHAIN_FILE=...vcpkg - I am using vcpkg for my code's dependencies but not for pybind11.
• cmake_minimum_required(VERSION 3.23...3.29)
• Python 3.12, Windows 11, Visual Studio 2022 (latest), using VS's CMake (VS File -> Open CMake...).
• Using pybind11 as submodule, with add_subdirectory("3rdparty/pybind11")
• These issues started happening when I switched cmake_minimum_required from 3.10 to 3.23...3.29. As far as I understand, this will have changed CMake's behaviour from using its old FindPython logic to the new FindPython logic.

Settings/Issues:

Release mode compilation and set(PYBIND11_FINDPYTHON ON)

No issues: CMake runs. Compilation works, linking works.

Release mode compilation and set(PYBIND11_FINDPYTHON OFF) and find_package(Python 3.12 COMPONENTS Interpreter Development REQUIRED)

CMake error: Could not find a package configuration file provided by "PythonInterp" (requested version 3.7) with any of the following names: PythonInterpConfig.cmake pythoninterp-config.cmake. As far as I could find out, "PythonInterp" is the old deprecated CMake FindPython module, and it shouldn't ever be called in this case. I've investigated this using VS's CMake debugger and in pybind11Common.cmake line 228 it's using the "else()" with the "Classic mode" and calls pybind11Tools.cmake, which in turn on L50 calls find_package(PythonLibsNew ${PYBIND11_PYTHON_VERSION} MODULE REQUIRED ${_pybind11_quiet}), which in turn calls FindPythonLibsNew.cmake, which on L114 calls find_package(PythonInterp ${PythonLibsNew_FIND_VERSION} ${_pythonlibs_required} ${_pythonlibs_quiet}) with 3.7 as version, and this call obviously fails. I don't understand why pybind11's CMake scripts are taking this route, first, since I'm using a recent CMake version and policies, and second, why would it try and find Python at all, if I've got PYBIND11_FINDPYTHON OFF and my own find_package(Python...) call has already found 3.12. Yes, I've cleared the CMake cache / build folder.

Debug mode compilation and set(PYBIND11_FINDPYTHON OFF) and find_package(Python 3.12 COMPONENTS Interpreter Development REQUIRED)

Same issues as above with PythonInterp

Debug mode compilation and set(PYBIND11_FINDPYTHON ON)

CMake runs. Compilation works. Linker produces an error: out\build\x64-debug\LINK : fatal error LNK1104: cannot open file 'python312.lib'. This is the full linker command-line:

>------ Build started: Configuration: x64-debug -------
  [1/1] Linking CXX shared module python\eos.cp312-win_amd64.pyd
  FAILED: python/eos.cp312-win_amd64.pyd 
  C:\Windows\system32\cmd.exe /C "cd . && "C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin\cmake.exe" -E vs_link_dll --intdir=python\CMakeFiles\python-bindings.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100226~1.0\x64\mt.exe --manifests  -- C:\PROGRA~1\MIB055~1\2022\COMMUN~1\VC\Tools\MSVC\1442~1.344\bin\Hostx64\x64\link.exe /nologo python\CMakeFiles\python-bindings.dir\generate-python-bindings.cpp.obj  /out:python\eos.cp312-win_amd64.pyd /implib:python\eos.lib /pdb:python\eos.pdb /dll /version:0.0 /machine:x64 /debug /INCREMENTAL  C:\Users\PatrikHuber\AppData\Local\Programs\Python\Python312\libs\python312_d.lib  kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib && C:\Windows\system32\cmd.exe /C "cd /D C:\Users\PatrikHuber\Projects\eos\eos\out\build\x64-debug\python && "C:\Program Files\PowerShell\7\pwsh.exe" -noprofile -executionpolicy Bypass -file "C:/Program Files/Microsoft Visual Studio/2022/Community/VC/vcpkg/scripts/buildsystems/msbuild/applocal.ps1" -targetBinary C:/Users/PatrikHuber/Projects/eos/eos/out/build/x64-debug/python/eos.cp312-win_amd64.pyd -installedDir C:/Users/PatrikHuber/Projects/eos/eos/out/build/x64-debug/vcpkg_installed/x64-windows/debug/bin -OutVariable out""
  LINK Pass 1: command "C:\PROGRA~1\MIB055~1\2022\COMMUN~1\VC\Tools\MSVC\1442~1.344\bin\Hostx64\x64\link.exe /nologo python\CMakeFiles\python-bindings.dir\generate-python-bindings.cpp.obj /out:python\eos.cp312-win_amd64.pyd /implib:python\eos.lib /pdb:python\eos.pdb /dll /version:0.0 /machine:x64 /debug /INCREMENTAL C:\Users\PatrikHuber\AppData\Local\Programs\Python\Python312\libs\python312_d.lib kernel32.lib user32.lib gdi32.lib winspool.lib shell32.lib ole32.lib oleaut32.lib uuid.lib comdlg32.lib advapi32.lib /MANIFEST /MANIFESTFILE:python\CMakeFiles\python-bindings.dir/intermediate.manifest python\CMakeFiles\python-bindings.dir/manifest.res" failed (exit code 1104) with the following output:
C:\Users\PatrikHuber\Projects\eos\eos\out\build\x64-debug\LINK : fatal error LNK1104: cannot open file 'python312.lib'
  
  ninja: build stopped: subcommand failed.

Build failed.

So this is really weird. On the linker command-line it shows it's linking to python312_d.lib, and python312.lib isn't even part of the linker command-line call. Why would it complain that it can't find a file that is not part of its commandline-arguments anyway? Also: In my C:\Users\PatrikHuber\AppData\Local\Programs\Python\Python312\libs folder, I do have both python312.lib and python312_d.lib. So if it was needed, it should find it there. I think something odd is going on here.

Reproducible example code

See above.
For the full code, it's the devel branch of my library here https://github.com/patrikhuber/eos/tree/devel. It's self-contained and builds with vcpkg plus the rest of the dependencies from submodules (git submodule update --init --recursive).

Is this a regression? Put the last known working version here if it is.

Not a regression

[patrikhuber]

Update: I was able to fix the pybind312.lib error in debug compilation by adding set(PYTHON_IS_DEBUG ON) before add_subdirectory("3rdparty/pybind11"). I've done a lot of debug builds before and never needed this - I think this is new behaviour introduced also within the last couple of years or so. Since debug builds are a very common thing to do while developing C++ Python modules, I think it would be great to add this to pybind11's docs - I can't find it mentioned in the docs anywhere?

The PythonInterp issue is still there.

[nsurbay]

I have a simillar issue on Linux, where pybind11 use the classic mode when PYBIND11_FINDPYTHON is not set and after a find_package(Python 3.12 EXACT REQUIRED COMPONENTS ...) and finally link with python3.7 (also present in manylinux docker image). I review the code (tag 2.13.6) and found the followed :

CMakeLists.txt#L136-L137 : When use add_subdirectory (or FetchContent in my case), _pybind11_findpython_default is always set to OFF (as we are not the main project), so PYBIND11_FINDPYTHON is never not defined.
tools/pybind11Common.cmake#L215 : If PYBIND11_FINDPYTHON is not explicitly set to ON, the old tools will be used.

For the version 2.13.6, the variable PYBIND11_FINDPYTHON alone seems to define if we use the Classic mode or the new FindPython mode. The commit to blame seems to be ddb8b67 (introduce in 2.12.0).

I also note that the variable PYBIND11_FINDPYTHON is not use anywhere else in the code. On my side, I simply set PYBIND11_FINDPYTHON to ON AND call find_package(Python ...) before including pybind11, and the new mode is use with my version of python (see pybind11NewTools.cmake#L25 and tools/pybind11NewTools.cmake#L58)