Rename of repository not handled well

[matthijskooijman]

This might actually be a bug in go, but I'm reporting it here anyway, since this breaks things here for now.

Recently, this library movied from sourcegraph to github. However, using the old sourcegraph.com name is now broken:

$ go get -d sourcegraph.com/sourcegraph/go-diff
go: sourcegraph.com/sourcegraph/[email protected]: parsing go.mod: unexpected module path "github.com/sourcegraph/go-diff"
go: error loading module requirements

The obvious fix is to just use the new github.com import path, but this is not always trivial. In particular, when go-diff is pulled in by a dependency that has the old url, running go get -u to update dependencies fails with the same error.

I guess that the problem is that go does not gracefully support updating to a new version in this case and updating across a rename (so 0.5.0 -> 0.5.1) needs changing of the import path. I can't see an easy way to fix this on the go side of things, so perhaps the fix here is to have sourcegraph.com not publish the 0.5.1 release (only 0.5.0), though that would also mean that people might never realize they're using an outdated version...

(also, I couldn't actually find the code on sourcegraph.com anymore, http://sourcegraph.com/sourcegraph/go-diff gives a 404, but apparently go get still manages to access the files)

[matthijskooijman]

As a workaround, adding this to my go.mod helps:

replace sourcegraph.com/sourcegraph/go-diff v0.5.1 => github.com/sourcegraph/go-diff v0.5.1

However, that only works for this particular version. On any subsequent update, the line must be updated again...

[sqs]

Sorry for the trouble with this rename. @dmitshur do you have any thoughts here?

[jeanbza]

Relevant: golang/go#30831

@matthijskooijman Could you post a GO111MODULE=on go mod graph so that we can see which library is bringing in the old sourcegraph.com/sourcegraph/go-diff?

[oz]

@jadekler does this help?

$ go mod graph | grep [email protected]  
golang.org/x/[email protected] sourcegraph.com/sourcegraph/[email protected]
golang.org/x/[email protected] sourcegraph.com/sourcegraph/[email protected]
golang.org/x/[email protected] sourcegraph.com/sourcegraph/[email protected]
golang.org/x/[email protected] sourcegraph.com/sourcegraph/[email protected]
golang.org/x/[email protected] sourcegraph.com/sourcegraph/[email protected]
golang.org/x/[email protected] sourcegraph.com/sourcegraph/[email protected]
golang.org/x/[email protected] sourcegraph.com/sourcegraph/[email protected]
sourcegraph.com/sourcegraph/[email protected] github.com/shurcooL/[email protected]
sourcegraph.com/sourcegraph/[email protected] github.com/shurcooL/[email protected]
sourcegraph.com/sourcegraph/[email protected] github.com/gogo/[email protected]
sourcegraph.com/sourcegraph/[email protected] sourcegraph.com/sqs/[email protected]

[oz]

FWIW, There is a similar issue with the golang/lint module golang/lint#446

[jeanbza]

That's useful but unfortunately is not showing me who is pulling in the old version of golang.org/x/build (which itself is pulling in the old sourcegraph.com/sourcegraph/go-diff. Could you post the full go mod graph?

[matthijskooijman]

before running the upgrade, it seems I have not dependencies on sourcegraph/go-diff at all (only sergi/go-diff):

$ GO111MODULE=on go mod graph |grep sourcegraph.go-diff
$ GO111MODULE=on go mod graph |grep go-diff@
github.com/gobuffalo/[email protected] github.com/sergi/[email protected]
github.com/gobuffalo/[email protected] github.com/sergi/[email protected]
github.com/gobuffalo/[email protected] github.com/sergi/[email protected]

after running the upgrade (with the workaround from #34 (comment)), it seems that for me, it is also x/build that references the wrong one. Tracing it back, this seems to come from cloud.google.com/[email protected]:

$ GO111MODULE=on go mod graph |grep \ sourcegraph.com.sourcegraph.go-diff
golang.org/x/[email protected] sourcegraph.com/sourcegraph/[email protected]
$ GO111MODULE=on go mod graph |grep \ golang.org.x.build
cloud.google.com/[email protected] golang.org/x/[email protected]
$ GO111MODULE=on go mod graph |grep \ [email protected]
golang.org/x/[email protected] cloud.google.com/[email protected]

However, there is a more recent version available that no longer depends on x/build since version 0.37.3 (Which makes some other changes to (fix issues around lint](golang/go#30833 (comment)) as well). I'm not quite sure why my mod graph still includes 0.36.0 rather than the latest version (since I did run go get -u -m just now). I'm not entirely sure how the graph is built at all (since I also see cloud.google.com/[email protected] as a dependency on the right side, but never on the left side, so it is not a full graph in the sense that I think it picks one version for each package rather than all of them).

I can trace this a bit further:

$ GO111MODULE=on go mod graph |grep \ [email protected]                                                                                       
golang.org/x/[email protected] cloud.google.com/[email protected]
$ GO111MODULE=on go mod graph |grep \ golang.org/x/[email protected]                                                             
github.com/gobuffalo/[email protected] golang.org/x/[email protected]
$ GO111MODULE=on go mod graph |grep \ github.com/gobuffalo/[email protected]                                                     
github.com/gobuffalo/[email protected] github.com/gobuffalo/[email protected]
$ GO111MODULE=on go mod graph |grep \ github.com/gobuffalo/[email protected]                                                     
github.com/gobuffalo/[email protected] github.com/gobuffalo/[email protected]
github.com/gobuffalo/packr/[email protected] github.com/gobuffalo/[email protected]
github.com/gobuffalo/[email protected] github.com/gobuffalo/[email protected]

I'm not sure how to quickly see the golang.org/x/tools source, but the next step in the list is github.com/gobuffalo/gogen which does not seem to have a go.mod file. AFAIU, this means that an implicit one using the latest versions is used and made explicit in my project's go.mod file (AFAIK these are the // indirect lines?). However, go.mod has a newer version of x/tools than in the graph:

$ grep x.tools go.mod
        golang.org/x/tools v0.0.0-20190411180116-681f9ce8ac52 // indirect

I'm still confused. I put up the full go mod graph output (both before and after the upgrade), perhaps anyone else can make more sense of this? https://gist.github.com/matthijskooijman/54fb7e36f578a4e83211537a77920f12

For completeness:

$ go version
go version go1.12.2 linux/amd64

(Also, AFAIU the underlying problem in go is that it looks at older versions of dependencies too closely, which is the subject of golang/go#30831. Until that is fixed, the workaround is to remove problematic versions out of your mod graph completely, which is what we're trying to do there I suppose...)

[dmitshur]

@matthijskooijman Looks like the latest version of github.com/gobuffalo/buffalo-plugins (v1.14.0) module needs to update its dependencies. Here's the chain of modules being required:

go get: github.com/gobuffalo/[email protected] ->
	github.com/gobuffalo/[email protected] ->
	github.com/gobuffalo/[email protected] ->
	github.com/gobuffalo/[email protected] ->
	github.com/gobuffalo/[email protected] ->
	golang.org/x/[email protected] ->
	cloud.google.com/[email protected] ->
	golang.org/x/[email protected] ->
	sourcegraph.com/sourcegraph/[email protected] ->
	sourcegraph.com/sourcegraph/[email protected]: parsing go.mod: unexpected module path "github.com/sourcegraph/go-diff"

golang.org/x/[email protected] is an old version of x/build, the latest does not have a requirement on sourcegraph.com/sourcegraph/go-diff.

That go get output is from Go tip (1.13 in development).

[dmitshur]

The obvious fix is to just use the new github.com import path, but this is not always trivial. In particular, when go-diff is pulled in by a dependency that has the old url, running go get -u to update dependencies fails with the same error.

I guess that the problem is that go does not gracefully support updating to a new version in this case and updating across a rename (so 0.5.0 -> 0.5.1) needs changing of the import path. I can't see an easy way to fix this on the go side of things, so perhaps the fix here is to have sourcegraph.com not publish the 0.5.1 release (only 0.5.0),

Agree. However, this may not be easy to achieve.

(also, I couldn't actually find the code on sourcegraph.com anymore, http://sourcegraph.com/sourcegraph/go-diff gives a 404, but apparently go get still manages to access the files)

The go get command uses ?go-get=1 query parameter when resolving vanity import paths, as documented here. https://sourcegraph.com/sourcegraph/go-diff?go-get=1 gives 200 OK.

[matthijskooijman]

github.com/gobuffalo/gogen which does not seem to have a go.mod file. AFAIU, this means that an implicit one using the latest versions is used

Ah, just noticed that the latest version of gogen has no go.mod file, but an earlier versions has one (and references an older version of x/tools).

@matthijskooijman Looks like the latest version of github.com/gobuffalo/buffalo-plugins (v1.14.0) module needs to update its dependencies. Here's the chain of modules being required:

Thanks. It indeed seems that the rest of the chain is already updated (or has its go.mod file removed, which should also work I guess). However, I would expect that adding this to my go.mod would then fix things:

replace github.com/gobuffalo/release => github.com/gobuffalo/release v1.4.0

(which is the dependency update that would need to happen in side buffalo-plugins).

However, this does not seem to help, I still get the same error on go get -m -u. Adding the previous workaround still works, but afterwards sourcegraph/go-diff has entirely disappeared from my dep tree. I'm building gotip now to see if I can figure if there is any other problematic path.

[matthijskooijman]

Yup, there were a few more paths. I opened gobuffalo/buffalo-plugins#18 and @markbates has responded like lightning and published a fixed version. With that, I can upgrade as expected again (though I did have to manually upgrade buffalo-plugins once before running go get -u -m).

[oz]

That's useful but unfortunately is not showing me who is pulling in the old version of golang.org/x/build (which itself is pulling in the old sourcegraph.com/sourcegraph/go-diff. Could you post the full go mod graph?

Sorry for the delayed reply. I've checked the dependency graph that referenced the old version of go-diff. In my case, it is related to the lileio/pubsub module, which depends on an old version of Google's GRPC, which (...) depends on an outdated version of go-diff.

I'm re-evaluating some of the dependencies of my little project, to avoid such a graph. 😵

[matthijskooijman]

For anyone looking to solve this problem, @jadekler has written (somewhat extensively) about this issue and how to fix it at https://github.com/golang/go/wiki/Resolving-Problems-From-Modified-Module-Path.