Private repositories

[pltanton]

Describe the bug
When I try to run gomod2nix it stucks on private dependency, asking username and pssword.
In .config/git/config I've got

[url "[email protected]:"]
	insteadOf = "https://github.com/"

But it does no effect. When I trying to build application locally with go mod it works fine.

[cstrahan]

Should be possible to ditch nix-prefetch-git in favor of something like:

$ nix eval --raw --expr '(builtins.fetchGit { url = https://github.com/some/project; rev = "f672df68b89b808dbf87fb053b10276f1db93f39"; submodules = true; }).narHash' --impure
sha256-1GmlnioIhadMApNcYxOoJPUsrBM4/Zssk6DIdBIHlnc=

That'll use your git config as expected.

---

Would be great to see private repos supported.

[cstrahan]

Tried hacking that up. There's the problem that due to the use of short revs the command line's end up looking like:

nix eval --impure --json --expr '(builtins.fetchGit { url = https://github.com/shurcooL/octicon; rev = "fa4f57f9efb2"; submodules = true; }).narHash'

which fails:

error: hash 'ed9d249f429b' has wrong length for hash type 'sha1'

So options:

1. Find a way to first expand the short rev to the full length rev.
2. Patch fetchGit to allow shorter revs.
3. Have nix provide some first class, builtin means of fetching a git repo, which could be 99% the same code as fetchGit but allowing for short rev.

[cstrahan]

This is what I have so far. Shame about the short rev issue...

diff --git a/fetch/fetch.go b/fetch/fetch.go
index d65cd9f..c50d3b3 100644
--- a/fetch/fetch.go
+++ b/fetch/fetch.go
@@ -1,15 +1,10 @@
 package fetch
 
 import (
+	"encoding/base64"
+	"encoding/hex"
 	"encoding/json"
 	"fmt"
-	log "github.com/sirupsen/logrus"
-	"github.com/tweag/gomod2nix/formats/buildgopackage"
-	"github.com/tweag/gomod2nix/formats/gomod2nix"
-	"github.com/tweag/gomod2nix/types"
-	"golang.org/x/mod/modfile"
-	"golang.org/x/mod/module"
-	"golang.org/x/tools/go/vcs"
 	"io/ioutil"
 	"os"
 	"os/exec"
@@ -17,6 +12,14 @@ import (
 	"regexp"
 	"sort"
 	"strings"
+
+	log "github.com/sirupsen/logrus"
+	"github.com/tweag/gomod2nix/formats/buildgopackage"
+	"github.com/tweag/gomod2nix/formats/gomod2nix"
+	"github.com/tweag/gomod2nix/types"
+	"golang.org/x/mod/modfile"
+	"golang.org/x/mod/module"
+	"golang.org/x/tools/go/vcs"
 )
 
 type packageJob struct {
@@ -157,10 +160,15 @@ func fetchPackage(caches []map[string]*types.Package, importPath string, goPacka
 		return nil, err
 	}
 
+	var attrName string
 	commitShaRev := regexp.MustCompile(`v\d+\.\d+\.\d+-[\d+\.a-zA-Z]*?[0-9]{14}-(.*?)$`)
 	rev := strings.TrimSuffix(sumVersion, "+incompatible")
 	if commitShaRev.MatchString(rev) {
 		rev = commitShaRev.FindAllStringSubmatch(rev, -1)[0][1]
+		attrName = "rev"
+	} else {
+		rev = "refs/tags/" + rev
+		attrName = "ref"
 	}
 
 	goPackagePathPrefix, pathMajor, _ := module.SplitPathVersion(goPackagePath)
@@ -194,27 +202,32 @@ func fetchPackage(caches []map[string]*types.Package, importPath string, goPacka
 	}
 
 	type prefetchOutput struct {
-		URL    string `json:"url"`
-		Rev    string `json:"rev"`
-		Sha256 string `json:"sha256"`
-		Path   string `json:"path"`
-		// date   string
-		// fetchSubmodules bool
-		// deepClone       bool
-		// leaveDotGit     bool
+		LastModified     int    `json:"lastModified"`
+		LastModifiedDate string `json:"lastModifiedDate"`
+		NarHash          string `json:"narHash"`
+		OutPath          string `json:"outPath"`
+		Rev              string `json:"rev"`
+		RevCount         int    `json:"revCount"`
+		ShortRev         string `json:"shortRev"`
+		Submodules       bool   `json:"submodules"`
 	}
 
 	log.WithFields(log.Fields{
 		"goPackagePath": goPackagePath,
 		"rev":           rev,
 	}).Info("Cache miss, fetching")
+
 	stdout, err := exec.Command(
-		"nix-prefetch-git",
-		"--quiet",
-		"--fetch-submodules",
-		"--url", repoRoot.Repo,
-		"--rev", rev).Output()
+		"nix",
+		"eval",
+		"--impure",
+		"--json",
+		"--expr",
+		fmt.Sprintf("(builtins.fetchGit { url = %s; %s = \"%s\"; submodules = true; }).narHash", repoRoot.Repo, attrName, rev),
+	).Output()
+
 	if err != nil {
+		// cstrahan: I have no idea what this case is intended to handle.
 		newRev := fmt.Sprintf("%s/%s", relPath, rev)
 
 		log.WithFields(log.Fields{
@@ -222,12 +235,15 @@ func fetchPackage(caches []map[string]*types.Package, importPath string, goPacka
 			"rev":           newRev,
 		}).Info("Fetching failed, retrying with different rev format")
 		originalErr := err
+
 		stdout, err = exec.Command(
-			"nix-prefetch-git",
-			"--quiet",
-			"--fetch-submodules",
-			"--url", repoRoot.Repo,
-			"--rev", newRev).Output()
+			"nix",
+			"eval",
+			"--impure",
+			"--json",
+			"--expr",
+			fmt.Sprintf("(builtins.fetchGit { url = %s; %s = \"%s\"; submodules = true; }).narHash", repoRoot.Repo, attrName, newRev),
+		).Output()
 		if err != nil {
 			log.WithFields(log.Fields{
 				"goPackagePath": goPackagePath,
@@ -249,8 +265,26 @@ func fetchPackage(caches []map[string]*types.Package, importPath string, goPacka
 		vendorPath = importPath
 	}
 
+	// need to convert SRI sha256 hash to sha256 hex
+	if !strings.HasPrefix(output.NarHash, "sha256-") {
+		log.WithFields(log.Fields{
+			"goPackagePath": goPackagePath,
+		}).Error("Fetching failed")
+		return nil, fmt.Errorf("Error: NarHash didn't begin with sha256- prefix: %s", output.NarHash)
+	}
+
+	b, err := base64.StdEncoding.DecodeString(output.NarHash[7:])
+	if err != nil {
+		log.WithFields(log.Fields{
+			"goPackagePath": goPackagePath,
+		}).Error("Fetching failed")
+		return nil, err
+	}
+
+	sha256 := hex.EncodeToString(b)
+
 	if relPath == "" && pathMajor != "" {
-		p := filepath.Join(output.Path, pathMajor)
+		p := filepath.Join(output.OutPath, pathMajor)
 		_, err := os.Stat(p)
 		if err == nil {
 			fmt.Println(pathMajor)
@@ -262,7 +296,7 @@ func fetchPackage(caches []map[string]*types.Package, importPath string, goPacka
 		GoPackagePath: goPackagePath,
 		URL:           repoRoot.Repo,
 		Rev:           output.Rev,
-		Sha256:        output.Sha256,
+		Sha256:        sha256,
 		// This is used to skip fetching where the previous package path & versions are still the same
 		// It's also used to construct the vendor directory in the Nix build
 		SumVersion: sumVersion,

[cstrahan]

Another issue:

$ nix eval --impure --json --expr '(builtins.fetchGit { url = https://github.com/sanathkr/go-yaml; rev = "ed9d249f429b3f5a69f80a7abef6bfce81fef894"; submodules = true; }).narHash'
error: Cannot find Git revision 'ed9d249f429b3f5a69f80a7abef6bfce81fef894' in ref 'master' of repository 'https://github.com/sanathkr/go-yaml'! Please make sure that the rev exists on the ref you've specified or add allRefs = true; to fetchGit.
(use '--show-trace' to show detailed location information)

that commit is only reachable from the v2 branch. So that needs to be:

$ nix eval --impure --json --expr '(builtins.fetchGit { url = https://github.com/sanathkr/go-yaml; ref = "v2"; rev = "ed9d249f429b3f5a69f80a7abef6bfce81fef894"; submodules = true; }).narHash'
"sha256-A1j6zGFU86vSa8JGgE/PQayurqEgUluUceodV8dp6xA="

or alternatlively pass allRefs = true;.

[cstrahan]

We could fetch the git repos and then cache the short rev to full commit sha.

[cstrahan]

Still messing with this. Something peculiar that I've found:

vcs.RepoRootForImportPath("git.apache.org/thrift.git", false) gives:

(*vcs.RepoRoot)(0xc0001bc540)({
 VCS: (*vcs.Cmd)(0x991c20)(Git),
 Repo: (string) (len=21) "git.apache.org/thrift",
 Root: (string) (len=25) "git.apache.org/thrift.git"
})

The docs say this:

type RepoRoot struct {
	VCS *Cmd

	// Repo is the repository URL, including scheme.
	Repo string

	// Root is the import path corresponding to the root of the
	// repository.
	Root string
}

but that looks wrong in this case. Dunno if that's a bug in golang.org/x/tools/go/vcs or what...

Could maybe work around that for now by doing "https://"+repoRoot.Repo+".git" if the https:// scheme isn't there, but I don't know if that'd work in the general case.

[cstrahan]

Going to work around things for now with this in go.mod:

replace git.apache.org/thrift.git => github.com/apache/thrift v0.0.0-20180902110319-2566ecd5d999

[diogox]

+1 for this 🙏

[adisbladis]

While not a full fix #55 should help.

[takeda]

@adisbladis so I managed to get really close with getting private repos by using

[url "ssh://git@..."]
  insteadOd = "https://...."

As well as creating ~/.netrc for git server that doesn't support ssh.

When I'm calling nix build I'm getting exec: "git": executable file not found in $PATH I'm not sure how I can include it for gomod2nix.buildGoApplication.

When trying to use gomod2nix import I'm getting a different kind of error:

error:
       … while calling the 'filterSource' builtin

         at «string»:2:1:

            1|
            2| builtins.filterSource (name: type: baseNameOf name != ".DS_Store") (
             | ^
            3|   builtins.path {

       … while adding path '/nix/store/cpl6gnw0bvh6r7jr921mzh4cp9hw5a23-go-core_v1.10.0'

         at «none»:0: (source not available)

       error: path '/nix/store/cpl6gnw0bvh6r7jr921mzh4cp9hw5a23-go-core_v1.10.0' is not valid

[frederictobiasc]

Hi, just pinging that there is #123 that solves the issue for me. Should this be merged?

[anthr76]

Hi @frederictobiasc how did you test this? I would like to incorporate as well

[frederictobiasc]

Hi @anthr76,
I'm using niv as dependency manager. So I moved the gomod2nix dependency to juliens branch.
You need to take care that the attribute (sources.*) name exactly matches the dependency format as defined Here, otherwise the override won't have an effect.

Consider adding a trace if in doubt.