Use rctx.download to fetch go modules

[sluongng]

Currently in go_repository, unless the url attribute is set specifically, a go module is fetched via calling cmd/fetch_repo

bazel-gazelle/internal/go_repository.bzl

Line 147 in d16fc42

elif ctx.attr.urls:

bazel-gazelle/cmd/fetch_repo/main.go

Line 55 in d16fc42

func main() {

fetch_repo will in turn call go mod download to (a) download, (b) extract and (c) verify the content using GOSUMDB.

Because we are not using Bazel's native downloader via rctx.download or rctx.download_and_extract, the downloaded go modules are not cached via Bazel's --repository_cache= or does it benefit from --experimental_remote_downloader=.

We should be able to replicate (a) and (b) of go mod download by reconstructing the download URLs in starlark and feed it to rctx.download_and_extract to expand the zip. The URLs could be a mix of proxy download and direct download to replicate GOPROXY config.

For (c), we can create a custom binary called cmd/verify_zip to validate the downloaded content against GOSUMDB and GONOSUMDB config.

[fmeum]

Just a note: Even if we implement this, we still wouldn't benefit from the repository cache in its current form as we can't provide a hash for the .zip file itself, only for its contents.

[sluongng]

Right. The current hash in go.sum is the hash of the extracted archive directory using https://pkg.go.dev/golang.org/x/mod/sumdb/dirhash#HashDir and not of the .zip archive itself.

I wonder if there is something we can do in Bazel to support rctx.download_directory that is mapped to Remote Asset API FetchDirectory rpc.