[bug] Error when detecting CPython version when included via CMake

[wawanbreton]

Describe the bug

• OS: KUbuntu 24.04.1
• Conan version: 2.7.0
• CMake version: 3.28.3

Hi,
I'm trying to consume the cpython/3.12.2 package for my project, which is Python binding.

My conanfile.py contains the following:

def requirements(self):
    self.requires("cpython/3.12.2")

def configure(self):
    self.options["cpython"].shared = True

And also the classic CMake generator/build

Now my CMakeLists.txt contains:

cmake_minimum_required(VERSION 3.20)
find_package(Python REQUIRED)
...

Building CPython 2.12.2 works like a charm, and the FindPython.cmake file and associated are properly generated. But when I run my conan build ., I get the following error:

...
-- Conan: Including build module from '/home/erwan/.conan2/p/b/cpyth65d041356dc8d/p/lib/cmake/use_conan_python.cmake'
-- Found Python: /home/erwan/.conan2/p/b/cpyth65d041356dc8d/p/lib/cmake/../../bin/python3.12 (found version "3.12.3") found components: Interpreter 
CMake Error at /home/erwan/.conan2/p/b/cpyth65d041356dc8d/p/lib/cmake/use_conan_python.cmake:18 (message):
  CMake detected wrong cpython version - this is likely a bug with the
  cpython Conan package
Call Stack (most recent call first):
  build/Release/generators/FindPython.cmake:38 (include)
  CMakeLists.txt:6 (find_package)


-- Configuring incomplete, errors occurred!

ERROR: conanfile.py (pyarcus/5.4.0-alpha.0): Error in build() method, line 155
        cmake.configure()
        ConanException: Error 1 while executing

Apparently, when trying to detect the Python version, it finds Python 3.12.3 which is the one installed on my system. I indeed have the following:

$ /home/erwan/.conan2/p/b/cpyth65d041356dc8d/p/lib/cmake/../../bin/python3.12
Python 3.12.3 (main, Jul 31 2024, 17:43:48) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
$ LD_LIBRARY_PATH=/home/erwan/.conan2/p/b/cpyth65d041356dc8d/p/lib/ /home/erwan/.conan2/p/b/cpyth65d041356dc8d/p/lib/cmake/../../bin/python3.12
Python 3.12.2 (main, Sep  6 2024, 16:01:31) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

I can then use the LD_LIBRARY_PATH trick to start the build, and this time the detection works properly. But this really sound like a dirty hack 😃

Do you think this is something about my environment, or is there indeed a bug somewhere ? In this case, I can try to provide a fix, but I don't know what part of the process is faulty.

How to reproduce it

The provided bug description should allow to reproduce it. Otherwise, I can make a minimal projet.

[memsharded]

Hi @wawanbreton

Thanks very much for your report.
This seems indeed an issue in the cpython package recipe in ConanCenter, not necessarily in Conan itself. I'll check it, and if true I will transfer the ticket to the conan-center-index repo.

[valgur]

Related:

• cpython: Add native support for CMake's builtin FindPython(3) conan-center-index#23394
• [package] CPython: Misc recipe improvements conan-center-index#23151

Adding self.tool_requires("cpython/<host_version>") and a VirtualBuildEnv(self).generate() will likely fix your issue, though.

[memsharded]

Hi @wawanbreton

The reason it is failing is because being a shared-library, it is not found in the runtime paths.
Conan by default generates in the conanrun.sh information the necessary variables for runtime paths information, but the issue is doing a find_package(Python that happens to internally rely on the actual execution of the Python executable.

Then it means that the runtime information is needed also at build time (the build-time information contains the runtime paths of the tool_requires, so they can be used in the build, but not the runtime information of requires which in theory shouldn't be needed at build time).

This can be modeled in different ways with Conan, for example:

from conan import ConanFile
from conan.tools.cmake import CMake, cmake_layout
from conan.tools.env import VirtualRunEnv

class Pkt(ConanFile):
    settings = "os", "arch", "compiler", "build_type"
    generators = "CMakeToolchain", "CMakeDeps"
    requires = "cpython/3.12.2"

    def configure(self):
        self.options["cpython"].shared = True
    
    def layout(self):
        cmake_layout(self)

    def generate(self):
        VirtualRunEnv(self).generate(scope="build")
    
    def build(self):
        cmake = CMake(self)
        cmake.configure()
        cmake.build()

It can be forced with VirtualRunEnv(self).generate(scope="build") to also source the conanrun.sh runtime-environment script also from the conanbuild.sh, and then everything works.

Please try that and let us know if this makes sense.

[wawanbreton]

Hi, and thank you very much for you answers. Sorry for the delay in answering, I have been a bit sick.

So, setting VirtualRunEnv(self).generate(scope="build") does allow the Python detection to work ! This is a very acceptable solution for me, as it does not involve using a customized recipe, or something hacky/dirty.

On the other hand, I am probably not the only one who will run into this issue, in this case I don't have a very exotic setup. But maybe other people can find this discussion, and then the solution for it ? I also see that the VirtualRunEnv documentation does not list what scope values you can actually give, and for what purpose. Maybe that could be improved ? I'm merely suggesting, do what you want with it :)

Thanks again for you help !

[memsharded]

Thanks for the feedback!

yes, lets leave it open, at least some improvements to the docs could be made.

The thing is that "runtime" shared dependencies for "build time" are intrinsically complex, they don't have a great solution, and Conan can in fact do even more than some build systems can do by themselves. Still, it is very common for tools used at build-time to be statically linked to avoid these problems.

But yes, at least some docs can be added to explain this, thanks again!

[Ahajha]

Just chiming in - I'm glad that sanity check code I added actually caught something :)