initial support for using fetchGit

[cstrahan]

Generating the gomod2nix.toml appears to work, but I haven't tried building with this just yet.

The idea is that this should allow fetching private repositories (see #3).

fetchGit does not work with short revs, so this:

• Tries to use the GitHub API where possible
• Falls back to cloning the repo in a temporary directory and then uses git rev-parse

[cstrahan]

So, the gomod2nix.toml definitely works. To get the builder using fetchGit, I just did:

diff --git a/builder/default.nix b/builder/default.nix
index bfe3298..87d47a5 100644
--- a/builder/default.nix
+++ b/builder/default.nix
@@ -50,9 +50,9 @@ let
           sources = builtins.toJSON (lib.mapAttrs
             (goPackagePath: meta:
               let
-                src = fetchgit {
-                  inherit (meta.fetch) url sha256 rev;
-                  fetchSubmodules = true;
+                src = builtins.fetchGit {
+                  inherit (meta.fetch) url rev;
+                  submodules = true;
                 };
                 srcPath = "${src}/${meta.relPath or ""}";
               in

The next problem I'm having is that my go tests fail (they want to interact with the docker socket, which isn't going to happen with the sandbox enabled). Do you have an idea of what can be done here, aside from disabling tests? If the tests could be built as a binaries and installed in a different output, perhaps testing could be disabled within the build but executed out of band.

Having something akin to the above would make Nix a much easier sell to my company, so I'd greatly appreciate any suggestions here.

[cstrahan]

Speaking of the tests, maybe the -c flag could be used for what I described above:

        -c
            Compile the test binary to pkg.test but do not run it
            (where pkg is the last element of the package's import path).
            The file name can be changed with the -o flag.

You'd have to crawl the source tree and do something like go test -c some/sub/pkg/*.go for all of the tests (which will drop a pkg.test binary for each). To avoid name clashes between the packages (e.g. foo/bar vs baz/bar -- both would result bar.test), you could mirror the source tree in the output, so there would be e.g. $test/foo/bar/bar.test and $test/baz/bar/bar.test. Testing could then happen in a VM by walking $test and running all of the executibles.

I guess that would look something like:

          for pkg in $(getGoDirs test); do
            mkdir -p "$test/$pkg"
            buildGoDir test -o "$test/$pkg$(basename pkg).test" -c "$pkg/"*.go
          done

Completely untested, but I think that gives an idea of what I'm thinking.

[cstrahan]

Maybe something like go list -f '{{.Name}}' $pkg would be better than $(basename $pkg) -- would work for top level of source tree, and I think reflects what go test -c does to choose the default file name.

[cstrahan]

Made a couple quick hacks to get the tests built in a $test output:

        outputs = [ "out" "test" ];
        checkPhase = attrs.checkPhase or ''
          runHook preCheck

          for pkg in $(getGoDirs test); do
            mkdir -p $test/$pkg
            buildGoDir "test -o $test/$pkg/$(go list -f '{{.Name}}' "$pkg" ).test -c" "$pkg"
          done

          runHook postCheck
        '';

        preFixup = (attrs.preFixup or "") + ''
          find $out/{bin,libexec,lib} -type f 2>/dev/null | xargs -r ${removeExpr removeReferences} || true
          find $test -type f 2>/dev/null | xargs -r ${removeExpr removeReferences} || true
        '';

Works just fine. Tests don't run (so checkPhase is a bit of a misnomer here), this just spits out *.test binaries in $test to be run later. Another phase could be set up to additionally run the tests as part of the build.

I don't have these changes committed on this branch, because I haven't sorted out how this would best work for all users. I kinda like the idea of one phase to build the test binaries and another (if doCheck == true) to run the tests, so people could easily run the tests either in the build or outside of the build (or both), without resorting to bespoke solutions.

[cstrahan]

Ah, I just realized we can pass narHash to fetchGit. I'll see if that's something I can add.

[bahlo]

Hi, having support for private repositories would be amazing! Anything keeping this from being merged?