Git remote branch name changes can cause clone to fail

[tianon]

$ docker buildx build https://github.com/tianon/dockerfiles.git#055fe3f3847650f253d9e47fe1145307197d3771:docker-master
...
#1 [internal] load git source https://github.com/tianon/dockerfiles.git#055fe3f3847650f253d9e47fe1145307197d3771:docker-master
fatal: Not a valid object name 055fe3f3847650f253d9e47fe1145307197d3771^{commit}
From [https://github.com/tianon/dockerfiles](https://github.com/tianon/dockerfiles%1B[0m)
   2f5d8fdd..055fe3f3  master     -> origin/master
error: cannot lock ref 'refs/remotes/origin/update/prometheus/blackbox-exporter': 'refs/remotes/origin/update/prometheus' exists; cannot create 'refs/remotes/origin/update/prometheus/blackbox-exporter'
 ! [new branch]        update/prometheus/blackbox-exporter -> origin/update/prometheus/blackbox-exporter  (unable to update local ref)
error: some local refs could not be updated; try running
 'git remote prune origin' to remove any old, conflicting branches
...

Reproducing this is unfortunately really annoying -- you've got to create a repository with a branch name like foo, do a build (so that branch gets pulled into the BuildKit cache for the repo), then delete that remote foo branch and create a new foo/bar branch and do a build again with a new commit (so that it isn't just fully cached right away and tries to fetch again). It will then fail with an error similar to the above.

I'd suggest a two-part remedy to this:

1. most importantly, GitHub (and some others) support direct fetching of commits such that we don't have to do this "fetch all the branches of the remote and hope one of them has the commit we're asking for" dance -- if we're given an explicit commit to fetch, we should start by trying to fetch that exact commit individually and only fetch all branches if that fails (this gets especially important if branches that aren't being fetched are large and unused/unnecessary for the current build 🙈)

2. before we fetch branches but after we've set up the remote, we should probably git remote prune origin as suggested in the error message 👀

Edit: sorry, for context in my above error message, I have an automated process that creates branches like update/foo for updates to the Dockerfile I maintain in directory foo, and it means that at some point I had update/prometheus and now have update/prometheus/blackbox-exporter, but Git doesn't allow those two to co-exist simultaneously because one implies the other must be a directory instead a file because of implementation details in Git itself -- the build that's failing wasn't touching either of those branches, however, so it's honestly surprising for them to be fetched at all, let alone break the build 😄

[tianon]

The following flag to git fetch would likely also be a sufficient solution, but fetching explicitly by the commit provided still seems worthwhile/important:

       -p, --prune
           Before fetching, remove any remote-tracking references that no
           longer exist on the remote. Tags are not subject to pruning if they
           are fetched only because of the default tag auto-following or due
           to a --tags option. However, if tags are fetched due to an explicit
           refspec (either on the command line or in the remote configuration,
           for example if the remote was cloned with the --mirror option),
           then they are also subject to pruning. Supplying --prune-tags is a
           shorthand for providing the tag refspec.

           See the PRUNING section below for more details.

[tianon]

$ docker buildx build ... https://github.com/tianon/dockerfiles.git#f23bacded613589161013e51fce6a06b8a9a350f:docker-master
...
#1 [internal] load git source https://github.com/tianon/dockerfiles.git#f23bacded613589161013e51fce6a06b8a9a350f:docker-master
fatal: Not a valid object name f23bacded613589161013e51fce6a06b8a9a350f^{commit}
From [https://github.com/tianon/dockerfiles](https://github.com/tianon/dockerfiles%1B[0m)
   18c4789a..f23bacde  master            -> origin/master
error: cannot lock ref 'refs/remotes/origin/update/prometheus': 'refs/remotes/origin/update/prometheus/node-exporter' exists; cannot create 'refs/remotes/origin/update/prometheus'
 ! [new branch]        update/prometheus -> origin/update/prometheus  (unable to update local ref)
error: some local refs could not be updated; try running
 'git remote prune origin' to remove any old, conflicting branches
...

(still affected periodically -- eventually my cache expires and the builds succeed 🙈)

[tianon]

1. most importantly, GitHub (and some others) support direct fetching of commits such that we don't have to do this "fetch all the branches of the remote and hope one of them has the commit we're asking for" dance -- if we're given an explicit commit to fetch, we should start by trying to fetch that exact commit individually and only fetch all branches if that fails (this gets especially important if branches that aren't being fetched are large and unused/unnecessary for the current build 🙈)

Thanks to @jedevc and #5441, this half is fixed now! 😄 ❤️