Feature/project search

[grafikrobot]

Proposed changes

Implements an automatic method for searching and resolving global project references.

Types of changes

What types of changes does your code introduce?

• New feature (non-breaking change which adds functionality)

Checklist

• I searched the discussions
• I searched the closed and open issues
• I read the contribution guidelines
• I added myself to the copyright attributions for significant changes
• I checked that tests pass locally with my changes
• I added necessary documentation (if appropriate)

[grafikrobot]

@Kojoley - @grisumbras - @pdimov comments appreciated on this.

[grisumbras]

Seems good to me. The only thing I have reservations about is directory search order. It seems to me it's logical to start from the most specific and and with /, but it starts with /.

Also, it seems to not support setting B2_PROJECT_PATH=/boost/core:/opt/boost_core and searching for /boost/core (because it only searches for path.parent $(name) here).

[Kojoley]

1. IIRC for the B2/Jam the target id/name by default is a file path that it needs to create, that is already an issue for reporting precise error messages to users.
2. Distros users will certainly want a system installed libraries discover out-of-the box (which they should put in site-config.jam I suppose). That will lead to having multiple versions of the same library in the discovery paths. There probably will be a need to favor/force use of system provided libraries too.
3. There is certainly will be a future request to have an ability to pin/specify a version of a needed dependency.
4. I don't think it is a good idea to search for projects which are actually subprojects (Jamfile vs Jamroot). It seems that it will easily allow a situation where you don't know what you are actually using, like there will be cases when someone builds Boost slice and will end up in a mix of system Boost and own build. I feel like when you want to use an external library it should be stated explicitly. Am I overthinking?

[Kojoley]

key-value

Do you really think this complication worth it? I would guess that in the most user cases all the projects under specified paths will be loaded anyway, and for a package manager like vcpkg it is not a big deal to customize/optimize search scope via project-search rule.

[grafikrobot]

key-value

Do you really think this complication worth it? I would guess that in the most user cases all the projects under specified paths will be loaded anyway, and

Not sure I follow that. The use case I'm working on ATM is the case of a modular Boost. Where definitely not all projects would be loaded even though they could be under the same specified path. But maybe I'm not understanding what you are thinking of.

for a package manager like vcpkg it is not a big deal to customize/optimize search scope via project-search rule.

Recently I've come to learn that everything is a big deal in vcpkg because it actually does very little for you :-)

[Kojoley]

The project (Boost) that has Jamroot will eventually include everything in its subdirectories

he use case I'm working on ATM is the case of a modular Boost. Where definitely not all projects would be loaded even though they could be under the same specified path.

Can't Boost superproject (Jamroot) have that loader which will load a needed subproject when you request a library? Or your vision is that every Boost library will get its own Jamroot and libraries will no longer will be rooted to the superproject? But then I don't understand who and how will set and update these mappings for every library.

[grafikrobot]

The project (Boost) that has Jamroot will eventually include everything in its subdirectories

But it doesn't have to.

he use case I'm working on ATM is the case of a modular Boost. Where definitely not all projects would be loaded even though they could be under the same specified path.

Can't Boost superproject (Jamroot) have that loader which will load a needed subproject when you request a library? Or your vision is that every Boost library will get its own Jamroot and libraries will no longer will be rooted to the superproject? But then I don't understand who and how will set and update these mappings for every library.

A project that has just a top-level Jamfile automatically also works as if that Jamfile was the Jamroot. That's been around for some time. The goal is two fold.. 1. support the use case, for Boost, where there is a super-project Jamroot and 2. support the use case, for Boost, where the super-project doesn't exist (or there is no Jamroot/Jamfile at that root.

But perhaps a concrete example is better. Here's the modular Boost I've been working on https://github.com/grafikrobot/boost-b2-modular. As it currently stands there the following are possible:

1. It works as is where the Jamroot loads the libs projects.
2. You can replace everything after the project boost .. ; declaration with project-search /boost : libs ; and everything will still work.
3. You can delete the Jamroot entirely and do cd /*/boosroot/libs/serialization/build && b2 --project-search=/boost:/*/boostroot/libs ... and it will build serialization without problems. (* = absolute or relative path).

A package manager, or equivalent, would be doing (3) if they want modular Boost packages. Although they would probably do a bunch of individual key:value mappings for each. And they could also do (1), or (2), for monolithic builds (probably .. as I'm still working out all the details on that).

Does that help answer the questions? Or is it creating more questions? ;-)

[Kojoley]

3. You can delete the Jamroot entirely and do cd /*/boosroot/libs/serialization/build && b2 --project-search=/boost:/*/boostroot/libs ... and it will build serialization without problems. (* = absolute or relative path).

From my understanding of the written here is that this will only work because boost superproject loads serialization and all its dependencies, am I wrong?

A package manager, or equivalent, would be doing (3) if they want modular Boost packages. Although they would probably do a bunch of individual key:value mappings for each.

On the one hand it is expected from package managers to know all the dependencies of the requested library, on the other hand the current solution does not provide a way to discover dependency tree to construct such b2 invocation. vcpkg struggled with that quite a bunch and currently solves the issue with their own header scanning script + boostdep mappings.

[grafikrobot]

3. You can delete the Jamroot entirely and do cd /*/boosroot/libs/serialization/build && b2 --project-search=/boost:/*/boostroot/libs ... and it will build serialization without problems. (* = absolute or relative path).

From my understanding of the written here is that this will only work because boost superproject loads serialization and all its dependencies, am I wrong?

You are wrong :-) The way it will works in modular Boost is that individual libraries will need to specify their dependencies in Jamfiles. The method is the usual one of just having a target reference. For example serialization/build.jam (aka jamfile) would look like this https://github.com/grafikrobot/boost-b2-modular/blob/b2-modular/patch/libs/serialization/build.jam#L8. And so one for dependencies recursively. With this PR B2 when it tries to resolve those targets would split them into the /project//target parts and then use the user specified mappings to find the, for example, array/build.jam and do a use-porject /boost/array : .../array ; And then look up the boost_array target. Hence no need or involvement from a super-project.

A package manager, or equivalent, would be doing (3) if they want modular Boost packages. Although they would probably do a bunch of individual key:value mappings for each.

On the one hand it is expected from package managers to know all the dependencies of the requested library, on the other hand the current solution does not provide a way to discover dependency tree to construct such b2 invocation. vcpkg struggled with that quite a bunch and currently solves the issue with their own header scanning script + boostdep mappings.

But it could provide such information since it's going to be present in the build of the library itself. I.e. we could add an introspection feature to B2 that, given a target, spits out the list of external project dependencies (which would be the other Boost libs in this case).

[Kojoley]

I see what I have missed: libs part in --project-search=/boost:/*/boostroot/libs, which looks so strange and alien while still not working for sublibs and stuff in tools folder.

grafikrobot/boost-b2-modular@b2-modular/patch/libs/serialization/build.jam#L8.

I'm sorry to say that but I have to -- it looks ugly, writing the same stuff twice... Is it impossible to make <library>/boost/array work? Even better if it could be just simple /boost/array. It probably could be achieved by considering project-id reference as implying adding every project top-level target as <source>.

[grafikrobot]

I see what I have missed: libs part in --project-search=/boost:/*/boostroot/libs, which looks so strange and alien while still not working for sublibs and stuff in tools folder.

Well.. it's something entirely new. So some strangeness is expected. I thought about a couple different syntax options on the CLI and ENV. But this was the least problematic given all the constraints of the multiple platforms.

grafikrobot/boost-b2-modular@b2-modular/patch/libs/serialization/build.jam#L8.

I'm sorry to say that but I have to -- it looks ugly, writing the same stuff twice... Is it impossible to make <library>/boost/array work?

Maybe. It would mean finding a way for a project declaring some target as the one to add to the requirements and usage-requirements. Which would be used if a bare project reference is asked for.

Even better if it could be just simple /boost/array. It probably could be achieved by considering project-id reference as implying adding every project top-level target as <source>.

Maybe. The easier way would be to add some other init argument to define those project references.

So certainly some things to think about. But those are orthogonal to this PR. As they would be additions to synthesize the references. I.e. things for some future PRs.

[grisumbras]

2. Distros users will certainly want a system installed libraries discover out-of-the box (which they should put in site-config.jam I suppose). That will lead to having multiple versions of the same library in the discovery paths. There probably will be a need to favor/force use of system provided libraries too.

This can only be a problem if system packagers start putting .jam files for libraries into packages. I don't expect this to happen, to be honest.

3. There is certainly will be a future request to have an ability to pin/specify a version of a needed dependency.

This would be achieved with B2_PROJECT_PATH=/project/name:/opt/project_name_xx_yy_zz

4. I don't think it is a good idea to search for projects which are actually subprojects (Jamfile vs Jamroot). It seems that it will easily allow a situation where you don't know what you are actually using, like there will be cases when someone builds Boost slice and will end up in a mix of system Boost and own build. I feel like when you want to use an external library it should be stated explicitly. Am I overthinking?

Are you talking about something like B2_PROJECT_PATH=/boost:/opt/boost_1.72:/boost/core:/home/me/my_boost_core ?

BTW, I've created a module for explcictly requesting external libraries in #296. As far as I can see, it will also work with the mechanism in this PR.

[Kojoley]

2. Distros users will certainly want a system installed libraries discover out-of-the box (which they should put in site-config.jam I suppose). That will lead to having multiple versions of the same library in the discovery paths. There probably will be a need to favor/force use of system provided libraries too.

This can only be a problem if system packagers start putting .jam files for libraries into packages. I don't expect this to happen, to be honest.

They won't start putting them in the packages if that leads to ambiguity errors.

3. There is certainly will be a future request to have an ability to pin/specify a version of a needed dependency.

This would be achieved with B2_PROJECT_PATH=/project/name:/opt/project_name_xx_yy_zz

It doesn't. I won't receive an error that an available version is incompatible, and the needed version will not be found because the search will stop on the first found one.

4. I don't think it is a good idea to search for projects which are actually subprojects (Jamfile vs Jamroot). It seems that it will easily allow a situation where you don't know what you are actually using, like there will be cases when someone builds Boost slice and will end up in a mix of system Boost and own build. I feel like when you want to use an external library it should be stated explicitly. Am I overthinking?

Are you talking about something like B2_PROJECT_PATH=/boost:/opt/boost_1.72:/boost/core:/home/me/my_boost_core ?

That's the one way to shoot yourself in the foot. The /boost:/opt/boost_1.72 part could be coming from something like site-config.jam. And the /boost/core:/home/me/my_boost_core part is not needed when you will build Boost itself but didn't recursively checkout everything.

[grafikrobot]

4. It seems that it will easily allow a situation where you don't know what you are actually using, like there will be cases when someone builds Boost slice and will end up in a mix of system Boost and own build.

This already happens without this PR. And it's not possible prevent it as changing the default search paths of the compiler is usually not possibly. The only remedy I know of in this general case it to use virtual envs (or equivalent). Hence, I don't think it's worth trying to deal with here.

[grafikrobot]

3. There is certainly will be a future request to have an ability to pin/specify a version of a needed dependency.

Thinking about this.. Yes, probably. But I don't think I'm prepared to try and deal with that now. It's not an easy thing to deal with.

[grafikrobot]

@Kojoley & @grisumbras any other comments before I merge this in? Note, my goal is to push out a 4.10.0 version this coming weekend.

[grisumbras]

None from me.

[grafikrobot]

/backport

[github-actions]

Successfully created backport PR for version/4.10.0:

• [Backport version/4.10.0] Feature/project search #321