introduce the concept of opt-in package conflicts

[andrewrk]

Extracted from #14265. Depends on #14288.

Motivation: to discover when a dependency tree contains cruft, in the form of fundamentally redundant code.

The idea here is that packages could declare a set of labels which are features that the package provides:

.{
    // ...
    .provides = .{ .compression, .tls },
    // ...
}

Importantly, if two different packages in the dependency tree provide the same functionality, tooling would emit a warning:

$ zig pkg audit
warning: functionality 'tls' is provided by multiple packages:
  tls-gaMwFLV5lzxK8L2S (4 uses)
    networking_utils
    foo.bar
    baz
    quux.asdf.garbo
    lalala.boog
  taylors_cool_crypto-nP9xXden9vAO5Ijq (1 uses)
    ez_request

In this case, despite how cool taylors_cool_crypto is, perhaps it might be better from a bloat perspective to fork ez_request and make it use networking_utils instead, and then send a polite merge request upstream. If the ez_request maintainers don't want to merge it, then the ecosystem forks, ever in search of reducing bloat.

For another example, consider that infamous package, left-pad. We could imagine the underscore library having something like this:

.{
    // ...
    .provides = .{
        .left_pad,
        .right_pad,
        .is_number,
        .is_float,
        // ...
    },
    // ...
}

Here, underscore would have not listed every function it provided, but would list the functionality that it provided that competed with existing, tiny packages. With this proposal implemented, it may have lead to many JavaScript developers asking themselves why they were carrying redundant dependencies in their trees, and worked a little bit to come up with coherent sets of dependencies that reduced bloat. Perhaps more people would have migrated towards utilities such as underscore, or some people might have migrated towards truly independent sets of small packages. Either way, there would have been less redundant code in the dependency tree.

Speaking of underscore, there is another interesting package to bring up: lodash. This package is intentionally competing with underscore. With the provides feature that I propose here, this could actually be codified, by directly putting the other package's ID into the provides field:

.{
    .id = "lodash-X0QPNj321y5Doywn",
    .provides = .{ .left_pad, .@"underscore-8uxWP0tQAeUhBlwM" },
    // ...
}

This would be an aggressive move, explicitly documenting that a package is competing with another package. When an application developer finds such a situation in their tree, it is clear that there could be bloat reduced by picking a lane:

$ zig pkg audit
  warning: dependency tree contains competing packages:
    underscore-SK1WsSlgpGzZWue8 (1 uses)
      foo.bar
    lodash-iV0uo8FxqzJ+7N2h (1 uses)
      baz

Open-source communities need more drama! 🔥 🍿

[rohlem]

Interesting idea! Maybe it's obvious, however I'd like to mention that sometimes this redundancy can be used as a feature.
Say I want my application to be able to utilize one of 3 different "backend" libraries, e.g. for audio output, or GUI Window management.
Each of these 3 projects has had a couple of years already for implementing platform support for their cross-platform abstractions, fixing bugs along the way.

Supporting all of them (=> declaring them as dependencies) allows users to switch the backend they use via build flags, say if they're suffering from a bug in one of them.
However, the [feature flag - platform - backend] support matrix for such a project sounds like it would be complex enough that it would be better-fitted to the domain of user-provided code rather than a declarative format.

But still, for transitive dependencies (shared by backends) it may make sense to deduplicate again.
I guess since it's opt-in, users could just ignore these hints. But maybe just being able to flag specific features in dependencies as "intentionally duplicated" would already be enough to improve this use case.

[matu3ba]

Motivation: to discover when a dependency tree contains cruft, in the form of fundamentally redundant code.

I think this is a local optimum, as to handle all cases of code duplication one needs to look at AST level for each symbol.
I dont see, why the package manager could not have a dedicated mode to "temporary fetch dependencies for analysis"
instead of trying to rely on packagers providing the correct information.