Proposal: Local package management

[presentfactory]

The current package manager is in an unacceptable state as it forces projects to rely on networked dependencies if anything in the Zig ecosystem is to be leveraged. This is a problem as due to the decentralized nature of the system many of these dependencies are hosted on third party sites which introduces privacy risks as well as makes the ecosystem especially fragile due to no guarantee that these dependencies will still be hosted in say 10 years when someone may want to build the project again. Furthermore the system assumes a user will always have access to the internet which may not be true if one wishes to build a local project they have at a time without network access (specifically an archived project where its zig output/cache directory has been removed for space saving).

The ability to build a project totally locally needs to always be an option to mitigate concerns like this, and luckily the anonymousDependency function is a step in the right direction here by allowing you to specify a path for a dependency rather than pulling it from the network. This however is not a full solution as if you for instance use a package that itself has more nested dependencies, you would have to modify its build script to use anonymousDependency and do this every time you update it which is needlessly tedious.

My proposal is to instead have the package manager support some sort of mode in which it pulls information from a local directory rather than from the network. The idea is that it would use the package names found in the various build.zig.zon files it parses over typically and use these to look up folder names in a specified dependency directory indicated by a new field in an upper level build.zig.zon file. These folders would contain the source for each package (so they can just be a cloned Git repository), or potentially they could just be the raw archives as well similar to the networked approach, though personally I think it'd be more useful to be able to do this with source though so these folders can be managed as Git submodules.

An example of the desired directory structure would be something like:

build.zig
build.zig.zon (References my_package)
src
  ...
deps
  my_package
    build.zig
    build.zig.zon (References other_package)
    src
      ...
  other_package
    ...

And the top level build.zig.zon would be something like:

.{
    .name = "my_application",
    .version = "0.1.0",
    .dependency_path = "deps", // If specified this will force the package manager into local mode
    .dependencies = .{
        // This is the name of the folder that it will search in the dependency path for
        .my_package = .{
            .url = "https://link.to/dependency.tar.gz",
            .hash = "...",
        },
    },
}

Whereas a package's build.zig.zon would be more like:

.{
    .name = "my_package",
    .version = "0.1.0",
    .dependencies = .{
        // Will be searched for in the same dependency path as its parent unless
        // otherwise specified by another dependency_path in this file
        .other_package = .{
            .url = "https://link.to/dependency.tar.gz",
            .hash = "...",
        },
    },
}

To accommodate the case of requiring the same package with multiple different versions the package manager could also search the path for a folder with a specific version number on it, e.g. my_package_0.1.0 would be searched for first before the more generic my_package.

Doing this would allow the same benefits of the package manager to carry through (e.g. sharing of dependencies between packages, versioning, etc) but without the reliance on third party hosts or networking in general and greater local control over how the packages are updated.

Of course this option probably wouldn't be a good thing to use for say libraries that you want other people to use as having these local package folders nested in packages would result in potentially duplicated packages, though as long as the system works recursively in parsing new dependency directories found in the build.zig.zon files of dependencies themselves it should at least work regardless. This functionality is instead meant more for monolithic executables not intended to be used as dependencies that want to future-proof their build process so that there is confidence that their build system will always work regardless of external factors.

Also apologies if this has been suggested anywhere yet, I didn't see such a similar proposal other than perhaps some related discussion for bringing file path support in URLs here: #14339.

[figsoda]

Instead of having an option in build.zig.zon, it would probably be better to have an environment variable instead. This is kind-of already possible via $ZIG_GLOBAL_CACHE_DIR if each build is isolated, but having a local option just for p is still probably better here.

The current package manager is in an unacceptable state as it forces projects to rely on networked dependencies if anything in the Zig ecosystem is to be leveraged.

Also see #14281, which is probably a better solution in the long run.