Solution Survey

[grafikrobot]

One aspect we need to deal with is gathering a corpus of existing solutions that attempt to solve the problem of library and package metadata. Lets use this discussion to collect such instances we find and corresponding discussion as to what each such solution accomplishes, pros, cons, etc. For each solution we should start out with references to documentation and/or standard documents that are available.

[grafikrobot]

pkg-config

• Documentation: https://people.freedesktop.org/~dbn/pkg-config-guide.html
• Website: https://www.freedesktop.org/wiki/Software/pkg-config/
• Wikipedia: https://en.wikipedia.org/wiki/Pkg-config

Starting of with the well know, but not universally used pkg-config tool and format.

[bretbrownjr]

Quick thoughts:

Pros:

• Currently sees some adoption, though not universal
• Conceptually works with just about any language, though seems mostly used for C and C++
• Accessible to even simple build systems like shell scripts and simple Makefiles
• Supports some variable expansion, so it's possible to override the install prefix as needed
• Supports adding arbitrary variables to individual metadata files
• Does not need to be altered to model new features added to compilation toolchains as all flags are opaque
• Supports transitive requirement modelling for both static and shared dependencies

Cons:

• Is more of a tool for communicating and arranging command line fragments than a metadata system as such
• Because of that, the contents are not portable; flags can and are specific to specific version ranges of specific toolchains, even
• Also because it's a tool for arranging CLI arguments, it's not possible to express more logical requirements like conflicting choices or what language runtimes are required by a given dependency
• Poor support for concepts that aren't expressible in portable flag combinations
• Does not support querying transitive dependencies for all variables; this limits the extensibility of the format
• Because of that, there is an unclear path toward full C++ or Fortran modules support for pkg-config

[bretbrownjr]

Also note that there's a pkgconf project that aims to extend and replace pkg-config, more or less. I'm fine analyzing them both as pkg-config, but if someone feels strongly enough to split pkgconf out into its own thread, I won't object.

[mathstuf]

I think discussing the .pc format is fine. pkg-config is essentially replaced at this point.

Other things include context-sensitive sets of flags. For example, using MPI from C and C++ can differ because some libraries require defines to suppress the inclusion of C++ bits when using a C++ compiler. Python modules either need a cocktail of flags to suppress undefined symbol errors or the real library depending on whether they are a module or embedding an interpreter.

[bretbrownjr]

CPS (Common Package Specification)

• Documentation/Website: https://mwoehlke.github.io/cps
• ISO paper: https://wg21.link/P1313

[mathstuf]

One thing that has occurred to me since we hashed out the design is that I'd like to see the ability to declare custom usage requirements. For example:

• Python packages should be able to provide PYTHONPATH fragments
• ParaView plugins need to update PV_PLUGIN_PATH

The definition of a usage requirement is essentially a description and a function taking [T] -> Action where Action is something like "use these flags" (e.g., @rpath/ bits or include directories) or "set envvar X to value Y" and T is some "shape" of input (usually a path of some kind, but it could be a path and a flag for "use system inclusion if possible").

[alexreinking]

The definition of a usage requirement is essentially a description and a function taking [T] -> Action

Maybe the right package format is Prolog/Datalog 😉

1. Provide a bunch of default rules for propagating transitive usage requirements.
2. Packages provide a bunch of facts about themselves.
3. Packages can also provide their own usage requirements, written as datalog rules.
4. ???
5. Profit!

% Package metadata
name(MyPkg, "MyPkg").
description(MyPkg, "An example").
pythonpath(MyPkg, "/path/to/prefix").

% Package rules
pythonpath(X, Y) :- link_libraries(X, Z), pythonpath(Z, Y).

[alexreinking]

Actually, now that I'm thinking about it, you could implement a whole build system like this. Hmm...

[bretbrownjr]

CMake Config Packages

• Documentation: https://cmake.org/cmake/help/latest/guide/importing-exporting/index.html

[mwoehlke-kitware]

Thanks, @bretbrownjr!

At first blush, that looks like a reasonable overview of pkg-config. The biggest thing I'd note is that, over many years of "use in anger", CMake has found the pkg-config approach, at least in terms of its well-defined schema, to be lacking in a number of ways, which is why we rolled our own. It isn't perfect, and of course its biggest flaw is that it is expressed in procedural, CMake code, which makes it extremely difficult for anything but CMake to consume.

CPS was conceived to be a replacement for CMake's "native" package specification, both applying lessons learned, and moving to a declarative mode. I would also strongly recommend using a well-known and widely supported encapsulation so that users don't need to roll their own parsers. That's another down-side of pkg-config, and while it could be solved by a reference parser, I think it probably makes more sense to look at something like XML or JSON... and almost certainly the latter rather than the former. Strictly speaking, CPS is intended to not be wedded to a particular encapsulation, but the specification was written with the assumption that JSON would be used.

It's probably worth inviting @mathstuf to this conversation as well, as he helped with the design of CPS and knows the pitfalls of pkg-config better than I do.

(p.s. In case it's not obvious, @mwoehlke and @mwoehlke-kitware are both me.)

[GabrielDosReis]

We need a format that makes it easier for human to maintain and to debug, even if the original package description is tool-generated. That, in my view, rules out XML and JSON. YAML would be nice if it was a bit more "intinctly and intuitively" predictable; that leads me to lean towards TOML. I appreciate there is a certain amount of subjectivity in these views.

[mathstuf]

I'm not too concerned about how we spell things right now. I'd like to focus on what we'd like and need to spell first (see examples from D2800R0) so that we don't end up excluding something because it requires something like 3-deep table nesting in TOML and "that's too hard to do" becomes an excuse.

[mwoehlke-kitware]

I agree with @mathstuf, and again, I think CPS's approach of trying to be transport-agnostic is the right way to start. (And I agree that XML is not an ideal choice of transport; my point was more to contrast that against the much more limited format used by pkg-conf.) Maybe even we never specify the transport format and just let a de facto standard emerge.

A downside of e.g. YAML is that it's harder to consume; in practice, I think any consumer would have to grow a dependency on their language's reference implementation of YAML. By contrast, JSON has many available implementations and a modest likelihood that a consumer is already using one, or linking to something (e.g. Qt) that provides one. That widespread availability was a major factor in suggesting JSON (or even, yes, XML) as the transport.

To be honest, I also don't agree that JSON or even XML are that hard to read, given syntactic coloring and if they've been properly pretty-printed. Sure, they'll have a few more symbol characters compared to a YAML, but the text and visual layout is going to be almost identical.

[mathstuf]

D2800R0 was presented at SG15's Issaquah 2023 meeting. I have some cleanups/details to polish for an R1; hopefully I'll be able to make this month's mailing.