python_requires behavior in Conan2 is different

[Jannis1994]

Hello together,
we have a complex project where we also use the [python_requires] dependency app-base in our application deliveries.
The setup was working fine in Conan1 but in Conan2 we have problems.

Example in Conan2:

conaninfo.txt file of application xy:

[options]
a_compatibility=99.99.1 
dfl=1234
variant=test
[python_requires]
app-base/1.1.Z@company/testing

We have this application xy in our dependency list as build_requires.
We call conan create like this:

conan create . -o *:variant=$variant -o *:a_compatibility=$a_version --user company -f json --update > conan_info.json

Error log:

app-base/1.1.4@conti/testing: Checking remote: project
....
application xy/version_nr@company/stable: WARN: [Errno 2] No such file or directory: '/home/jenkins/a_compatibility.txt'
application xy/version_nr@company/stable: WARN: Could not read a_compatibility file. a_compatibility set to 'None'
application xy/version_nr@company/stable: WARN: This build can not be integrated into our image

and:

ERROR: There are invalid packages:
root_application/version_number@company: Invalid: Invalid a_compatibility detected: None for: application xy
Returns 6

With conan list command I can see that a_compatibility is set inside the options of package application_xy.

Conanfile.py from app-base:

import os, re
from conan import ConanFile, tools

class AppBase(object):
    def set_version(self):
        if not hasattr(self, 'src_directory'):
             raise ConanInvalidConfiguration("src_directory not set")
        if(os.path.isfile(f'{self.src_directory}/version-overwrite')):
            version = tools.files.load(self,os.path.join(self.src_directory, "version-overwrite"))
        else:
            content = tools.files.load(self,os.path.join(self.src_directory, "CMakeLists.txt"))
            version = re.search(r"  VERSION (.*)", content).group(1)
        self.version = version.strip()

    def export_sources(self):
        tools.files.copy(self, f"{self.src_directory}*", self.recipe_folder, self.export_sources_folder)

    def configure(self):
        if not self.options._a_compatibility:
            try:
                content = tools.files.load(self,"/home/jenkins/a_compatibility.txt")
                for line in content.splitlines():
                    if line.startswith("a_compatibility"):
                        self.options.a_compatibility = line.split("=")[1].strip()
                        break
                if not self.options.a_compatibility:
                    raise Exception("Could not find a_compatibility value")

            except Exception as err:
                self.output.warning(err)
                self.output.warning("Could not read a_compatibility file. a_compatibility set to 'None'")
                self.output.warning("This build can not be integrated into our image")
                self.options.a_compatibility = "None"


class AppBaseRecipe(ConanFile):
    name = "app-base"
    version = "1.1.4"

application xy conanfile.py:

class ApplicationConan(ConanFile):
    name = "application_xy"   
    options = {
        "variant": ["test", "test2"],
        "a_compatibility": ["ANY"],
        "dfl": ["ANY"]
    } 
    default_options = {
        "dfl": "1234",
        "variant": "test"
    }
    
    # app-base contains generic recipe parts    
    python_requires = "app-base/1.1.4@company/testing"
    python_requires_extend = "app-base.AppBase"

def export(self):  
...

Do you know what could be the problem?
It looks like that it tries to execute python_requires dependency again, but this dependency could only be solved at the creation of the application. Because it uses a Docker container to read out a_compatibility option value. During our conan create call it should not try to resolve the python requires of the dependency application. It should just take the binaries and the options. Was there a big change between Conan1 and 2?

Already took a look here:
https://docs.conan.io/2/reference/extensions/python_requires.html
and:
#13486

Thanks for your help.

Best regards,
Jannis

[memsharded]

Hi @Jannis1994

Some preliminary feedback. Note many of those also applied for Conan 1, they are not strictly new of Conan 2.

• content = tools.files.load(self,"/home/jenkins/a_compatibility.txt") doesn't seem correct. The configure() method cannot rely on external files, everything has to be self content, or it is undefined behavior. If you need files for the recipe to work, they must be exported with exports and they must be read from inside the recipe.
• The raise ConanInvalidConfiguration("src_directory not set") is not valid for this use case. This exception should be raised only in validate() scenarios, for raising binary compatibility issues, for example if the package cannot be built in Windows. You might want to raise a regular Exception instead
• The self.src_directory usage sounds like it should be avoided. The directory to obtain the CMakeLists.txt should be relative to the self.recipe_folder, so I'd suggest using self.recipe_folder instead.
• The if not self.options._a_compatibility doesn't look correct, the option is called options.a_compatibility. For reading safely an option that might not exist, you might want to use instead self.options.get_safe("a_compatibility")
• (object) is no longer necessary with Python 3
• The recommended way to define options values for the current package is not configure(), but config_options()

It looks like that it tries to execute python_requires dependency again, but this dependency could only be solved at the creation of the application. Because it uses a Docker container to read out a_compatibility option value. During our conan create call it should not try to resolve the python requires of the dependency application. It should just take the binaries and the options. Was there a big change between Conan1 and 2?

But this is nothing new of Conan 2.
python_requires in Conan 1 also were necessary for conan create, this hasn't changed. The way it works is that a python-requires is basically part of the package conanfile.py. The conanfile.py is not complete and cannot work without the python-requires, so conan create necessarily needs to get the python-requires in order to create the package. I think there might be a gap or something missing here.

[Jannis1994]

Hi @memsharded, thank you for your valauble feedback. I will talk with our DevOps team about the improvement suggestions from your side.

With regards to the main question,
If I call conan create like this:

conan create . -o *:variant=$variant -o:a *:a_compatibility=$a_version --user company -f json --update > conan_info.json

then the a_compatibility is transferred to all packages as it is visible in Profile host and Profile build.
Example:

======== Input profiles ========
Profile host:
[settings]
arch=x86_64
build_type=Release
os=Linux
[options]
*:a_compatibility=99.99.1
*:variant=Test

Profile build:
[settings]
arch=x86_64
build_type=Release
os=Linux
[options]
*:a_compatibility=99.99.1

Unfortunately this is causing another problem:

aplication_zzz/version_nr@company/stable: Checking remote:project
ERROR: '99.99.1' is not a valid 'options.a_compatibility' value.
Possible values are ['99.99.0']
Returns 1

So I guess our problem is mostly related to option inheritance to dependencies. In Conan1 this was working different.
As you mentioned in the past: In Conan 2, the user inputs from profiles and command line always have highest precedence than recipe defined ones. Could that also cause here the problems?

For aplication_zzz we for instance had to override a_compatibility from 99.99.1 to 99.99.0 because the application team delivered with this wrong/old version.

def configure(self):
    setattr(self.options["application_zzz"], "a_compatibility!", "99.99.0") 

Thank you.

[Jannis1994]

In Conan1 the configuration was looking like this:

command: 
conan create . company/testing -o *:variant=$variant -o *:a_compatibility=$a_version --json=conan_info.json --update $conanArgs

Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=7
os=Linux
os_build=Linux
[options]
*:a_compatibility=99.99.1
*:variant=Test
[build_requires]
[env]

I also tried to understand -o options:
-o:b: This option is used to specify settings for the build context
-o:h: This option is used to specify settings for the host context.
-o:a: This option is used to specify settings for all contexts

[memsharded]

ERROR: '99.99.1' is not a valid 'options.a_compatibility' value.

This looks weird, are you sure there is no other upstream recipe that overrides options={"a_compatibility": ['99.99.0']}?

I see the setattr(self.options["application_zzz"], "a_compatibility!", "99.99.0"), but I am not sure this should cause it.
I am struggling a bit to connect all the dots, do you think it would be possible to put together (in a repo, or a .zip) a small reproducible project, removing unrelated things, that reproduce the issue?

[Jannis1994]

I just checked it again and this error message was thrown by a bad conanfile.py that was used as a dummy for copy the first packages from conan1 style to conan2 style.
The options where not set to "a_compatibility": ["ANY"]" but "corbos_adg_compatibility": ["50.1.1"] instead.

I have to verify things now, thank you already for the help. Sometimes it helps to try to explain a problem.

[memsharded]

I have to verify things now, thank you already for the help. Sometimes it helps to try to explain a problem.

Rubber duck debugging for the win! 😂

[Jannis1994]

Yep that is right ;).

But there are still some problems - different - yet.
Desired a_compatibility is: 50.10.1

conan create . -o:a *:variant=$variant -o:a *:a_compatibility=$a_version --user company -f json --update > conan_info.json 

ERROR: Missing binary: application_abc/version_nr@ccp/testing:b4a4e6d7e39d6fe1583d8c14b

application_abc/version_nr@ccp/testing: WARN: Can't find a 'application_abc/version_nr@ccp/testing' package binary 'b4a4e6d7e39d6fe1583d8c14b' for the configuration:
[options]
a_compatibility=50.10.1
variant=Test

ERROR: Missing prebuilt package for 'application_abc/version_nr@ccp/testing'. You can try:
    - List all available packages using 'conan list "application_abc/version_nr@ccp/testing:*" -r=remote'
    - Explain missing binaries: replace 'conan install ...' with 'conan graph explain ...'
    - Try to build locally from sources using the '--build=application_abc/version_nr@ccp/testing' argument

More Info at 'https://docs.conan.io/2/knowledge/faq.html#error-missing-prebuilt-package'

The package application_abc has the following conanfile.py with a different a_compatibility set to 50.1.1 (could be the case if application delivered not as desired, but should not stop the conan create command):

from conan import ConanFile, tools


class WrappedPkg(ConanFile):
    name = "application_abc"
    version = "version_nr"
    options = {
        "a_compatibility": ["ANY"],
        "variant": ["Test"],
    }
    default_options = {
        "a_compatibility": "50.1.1",
        "variant": "Test",
    }

    def package(self):
        tools.files.copy(self, "*", src="application_abc", dst=self.package_folder)

Cannot override this with
setattr(self.options["application_abc"], "a_compatibility!", "50.1.1") in configure method of conanfile.py

Only way I see currently is a new package with the following content:
default_options = {
"a_compatibility!": "50.1.1",
"variant": "Test",
}
But there should be a better way, or?
I guess it is this priority topic again: #17476

[memsharded]

This is probably a bit off-topic, but as I see the compatibility thing, I tend to think that there might be better ways to achieve the desired behavior, specially because Conan 2 contains new features like the compatibility.py plugin to define custom compatibility rules over settings and options, in a cross-recipe way, which can be very convenient. This is something that we can discuss in other thread if you want to.

[memsharded]

Cannot override this with
setattr(self.options["application_abc"], "a_compatibility!", "50.1.1") in configure method of conanfile.py

Sorry, it is not very clear who is doing this. Is there are another package with a requires("application_abc/version") or something like that that will contain such configure() code? Having some minimal but full recipes will help to understand, I am trying to write a unit test that captures the behavior, but can't figure out the full setup.

[Jannis1994]

This is probably a bit off-topic, but as I see the compatibility thing, I tend to think that there might be better ways to achieve the desired behavior, specially because Conan 2 contains new features like the compatibility.py plugin to define custom compatibility rules over settings and options, in a cross-recipe way, which can be very convenient. This is something that we can discuss in other thread if you want to.

I also have maybe a better solution that I am currently testing. It is something like the following:

conanOverrides="-o:b application_xy/*:a_compatibility=99.1.1"
echo "Used Conan package option overrides: $conanOverrides"

conan create . -o:a *:variant=$variant -o:a *:a_compatibility=99.10.1 $conanOverrides --user company -f json --update > conan_info.json 

Could get nasty if we have more overrides. But that is something the applications should prevent and need to be enforced by us.

In general I would be interested in your compatibility.py approach 👍

[Jannis1994]

Cannot override this with
setattr(self.options["application_abc"], "a_compatibility!", "50.1.1") in configure method of conanfile.py

Sorry, it is not very clear who is doing this. Is there are another package with a requires("application_abc/version") or something like that that will contain such configure() code? Having some minimal but full recipes will help to understand, I am trying to write a unit test that captures the behavior, but can't figure out the full setup.

Yes we have one big package that has like 17 dependencies (1 requires - base package and 16 build requires -applications) to other packages. We integrate them and then deliver a new package.
In the conanfile for our big package I had this configure method. But this configure method override does not work if I use conan create -o:a option. only without :a