Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug] dependency cycle can't be avoided when bootstrapping a compiler #16758

Closed
smoofra opened this issue Jul 31, 2024 · 9 comments · Fixed by #16870
Closed

[bug] dependency cycle can't be avoided when bootstrapping a compiler #16758

smoofra opened this issue Jul 31, 2024 · 9 comments · Fixed by #16870
Assignees
Milestone

Comments

@smoofra
Copy link

smoofra commented Jul 31, 2024

Describe the bug

I'm not sure if this ought to be a bug report or a feature request.

I'm working on a recipe for gcc, and I'm running into a bootstrapping
issue that I don't know how to solve.

My goals are:

  • Build gcc cross compilers for linux and package them with conan.

  • The minimum versions of the linux kernel and glibc that are being
    targeted by the cross compiler should be specified in the settings.

  • Build a bootstrap gcc using whatever C compiler is available in path, and
    then use that to build the main cross compiler.

I thought I had the whole scheme working.

This command does indeed succeed. It creates a bootstrap gcc, and then uses
the bootstrap gcc to build a final gcc.

$ conan create ./cc -pr:b ./Linux-bootstrap -pr:h ./Linux-x86_64

But then when I try to use it, it all falls apart.

$ conan create foo -pr:h ./Linux-x86_64 -pr:b ./Linux-x86_64
...
ERROR: There is a cycle/loop in the graph:
    Initial ancestor: cc/1.0
    Require: cc/1.0
    Dependency: cc/1.0

This dependency cycle would be a problem if I really was trying to build cc
using Linux-x86_64 as the build profile. But I don't need to build it. I
built it already using Linux-bootstrap as the build profile. The package_id
has not changed. Now I just want to use it. But because there'd be a dependency
cycle if I tried to build it this way, it doesn't even get that far.

What can I do?

Thanks for reading.

How to reproduce it

example code is here https://github.com/smoofra/conan-cc-bootstrap

@memsharded
Copy link
Member

Hi @smoofra

Thanks for your report.

I have just tried it and it works, I can do your above steps and it doesn't raise any error.

You didn't report the Conan version, so maybe you are using Conan 1? I was trying with Conan 2.
One of the aspects that Conan 2 greatly improve over Conan 1 is the graph resolution capabilities, allowing more advanced cases like this kind of bootstrapping.

@memsharded memsharded self-assigned this Jul 31, 2024
@smoofra
Copy link
Author

smoofra commented Jul 31, 2024

I'm using conan 2.5.0

@smoofra
Copy link
Author

smoofra commented Jul 31, 2024

I just tried it again in a fresh environment and I still get the erorr

ubuntu 24.04, python 3.12.3

certifi==2024.7.4
charset-normalizer==3.3.2
colorama==0.4.6
conan==2.5.0
distro==1.8.0
fasteners==0.19
idna==3.7
Jinja2==3.1.4
MarkupSafe==2.1.5
patch-ng==1.17.4
python-dateutil==2.9.0.post0
PyYAML==6.0.1
requests==2.32.3
six==1.16.0
urllib3==1.26.19

@smoofra
Copy link
Author

smoofra commented Jul 31, 2024

I have added a dockerfile to the example
run docker build . and it should reproduce

@smoofra
Copy link
Author

smoofra commented Aug 2, 2024

well, I found a workaround

    def build_requirements(self):
        # when this recipe is being used as a build tool, hide all of its
        # requirements to avoid graph cycle errors due to bootstrapping
        if self.context == "build":
            return
        ....

It means that the gcc recipe can't be built using --build=missing, because the dependencies would be missing from conan.

But my gcc recipe is already only possible to build inside its own special CI pipeline anyway for reasons so this is no great loss to me

@memsharded memsharded added this to the 2.7.0 milestone Aug 5, 2024
@memsharded
Copy link
Member

Sorry, I have limited connectivity now, can't really use docker now, but assigned it to have a look at my return.
It is possible that the if self.context == "build": is not a workaround, it sounds like a valid and correct way to break an infinite loop that would happen otherwise (I need to check it properly first, last time I couldn't reproduce)

@memsharded
Copy link
Member

I have been investigating this.

At the moment, it is not possible to have a dependency on the same pkg/version as the current package, being the only exception the one that does something like:

def build_requirements(self):
     if cross_building(self):
           self.tool_requires("pkg/version")

The reason it is this way is because it is very easy to fall in an infinite loop which results in the Conan application hanging forever until it exhausts and fail, which is very bad UX.

The cross_building() condition acts very much like the if self.context == "build" in your case, pre-building it in the way you do it, as it is not possible to build it as part of a larger graph with --build=missing if the package is already in the "build" context.

It is not possible then to use settings to condition in the build_requirements(), because it doesn't really break the loop, the loop is checked before that and it fails, except for the case of the change of context from host to build.

I'd say that your approach is good enough at the moment.

I am not saying that we will not try to expand the bootstrapping capabilities, I will do some experiments to check what might be possible.

@memsharded
Copy link
Member

This would be an experimental branch to investigate the possibilities: #16870

@memsharded memsharded modified the milestones: 2.7.0, 2.8.0 Aug 28, 2024
@memsharded
Copy link
Member

#16870 was merged, so this will be part of Conan 2.8. Feedback welcome!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants