go.mod FilePath ReplaceDirective fails copying folder.

[jtszalay]

What version of gazelle are you using?

0.37.0

What version of rules_go are you using?

0.48.0

What version of Bazel are you using?

7.1.2

Does this issue reproduce with the latest releases of all the above?

Yes.

What operating system and processor architecture are you using?

x86_64 Ubuntu

What did you do?

Trying to use go.mod FilePath ReplaceDirective with repo https://github.com/ngrok/ngrok-go

Our repo uses direnv and nix to setup the environment.

What did you expect to see?

I expected to be able to use the local replace.

(If I delete the .direnv folder from my local repo checkout it works.)

What did you see instead?

It fails.

INFO: Repository gazelle~~go_deps~com_ngrok_golang_ngrok instantiated at:
  <builtin>: in <toplevel>
Repository rule go_repository defined at:
  /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~/internal/go_repository.bzl:379:32: in <toplevel>
ERROR: An error occurred during the fetch of repository 'gazelle~~go_deps~com_ngrok_golang_ngrok':
   Traceback (most recent call last):
        File "/var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~/internal/go_repository.bzl", line 279, column 17, in _go_repository_impl
                fail(command)
Error in fail: ["/var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~non_module_deps~bazel_gazelle_go_repository_tools/bin/fetch_repo", "--path", "/home/james/ngrok-go", "--dest", /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~go_deps~com_ngrok_golang_ngrok]
ERROR: <builtin>: fetching go_repository rule //:gazelle~~go_deps~com_ngrok_golang_ngrok: Traceback (most recent call last):
        File "/var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~/internal/go_repository.bzl", line 279, column 17, in _go_repository_impl
                fail(command)
Error in fail: ["/var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~non_module_deps~bazel_gazelle_go_repository_tools/bin/fetch_repo", "--path", "/home/james/ngrok-go", "--dest", /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~go_deps~com_ngrok_golang_ngrok]
ERROR: no such package '@@gazelle~~go_deps~com_ngrok_golang_ngrok//policy': ["/var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~non_module_deps~bazel_gazelle_go_repository_tools/bin/fetch_repo", "--path", "/home/james/ngrok-go", "--dest", /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~go_deps~com_ngrok_golang_ngrok]
ERROR: /home/james/ngrok/go/cmd/ngrok/tung/BUILD.bazel:4:11: //go/cmd/ngrok/tung:tung depends on @@gazelle~~go_deps~com_ngrok_golang_ngrok//policy:policy in repository @@gazelle~~go_deps~com_ngrok_golang_ngrok which failed to fetch. no such package '@@gazelle~~go_deps~com_ngrok_golang_ngrok//policy': ["/var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~non_module_deps~bazel_gazelle_go_repository_tools/bin/fetch_repo", "--path", "/home/james/ngrok-go", "--dest", /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~go_deps~com_ngrok_golang_ngrok]
ERROR: Analysis of target '//go/cmd/ngrok/app:app' failed; build aborted: Analysis failed

Attempting to run the fetch_repo script manually:

└─> /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~non_module_deps~bazel_gazelle_go_repository_tools/bin/fetch_repo --path /home/james/ngrok-go --dest /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~go_deps~com_ngrok_golang_ngrok
fetch_repo: mkdir /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~go_deps~com_ngrok_golang_ngrok/.direnv: file exists

Deleting that dir I then get the following:

└─> /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~non_module_deps~bazel_gazelle_go_repository_tools/bin/fetch_repo --path /home/james/ngrok-go --dest /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~go_deps~com_ngrok_golang_ngrok
fetch_repo: write /var/lib/ngrok/bazel/ngrok-bazel/external/gazelle~~go_deps~com_ngrok_golang_ngrok/.direnv/flake-inputs/kk6d4ip6m8b11k9rm2fw2a03v919pi4v-source: copy_file_range: is a directory

Is there a way to tell it to ignore that .direnv directory. It doesn't need it as a dep.

[fmeum]

It seems reasonable to skip all top-level directories that start with a .. Would you be interested in sending a PR?

[jtszalay]

Yeah! I'm easily able to recreate it so I'll work on making a PR.

[jtszalay]

Ah! I found what the true issue is. It doesn't handle things that are symlinks:

It seems reasonable to skip all top-level directories that start with a ..
I think this is still a fine solution however.

└─> ls -al ../ngrok-go/.direnv/flake-inputs/                                         
total 8
drwxr-xr-x 2 james ngrok 4096 Jun 20 12:26 .
drwxr-xr-x 3 james ngrok 4096 Jun 20 12:26 ..
lrwxrwxrwx 1 james ngrok   50 Jun 20 12:26 kk6d4ip6m8b11k9rm2fw2a03v919pi4v-source -> /nix/store/kk6d4ip6m8b11k9rm2fw2a03v919pi4v-source
lrwxrwxrwx 1 james ngrok   50 Jun 20 12:26 qkig73szmrhgp0qhncxy5vb36lw2g3jj-source -> /nix/store/qkig73szmrhgp0qhncxy5vb36lw2g3jj-source
lrwxrwxrwx 1 james ngrok   50 Jun 20 12:26 sl31dgnpwinzvzhh7ak2rx3h6w7kbb30-source -> /nix/store/sl31dgnpwinzvzhh7ak2rx3h6w7kbb30-source
lrwxrwxrwx 1 james ngrok   50 Jun 20 12:26 yj1wxm9hh8610iyzqnz75kvs6xl8j3my-source -> /nix/store/yj1wxm9hh8610iyzqnz75kvs6xl8j3my-source

Edit: excluding dirs that start with . causes hash mismatch on downloaded dependencies :/

		// Check if this is a top-level directory that starts with a `.`
		if info.IsDir() && filepath.Dir(rel) == "." && filepath.Base(rel)[0] == '.' {
			return filepath.SkipDir
		}

Error in fail: gazelle~~go_deps~org_golang_x_net: fetch_repo: resulting module with sum h1:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=; expected sum h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs=
ERROR: no such package '@@gazelle~~go_deps~org_golang_x_net//idna': gazelle~~go_deps~org_golang_x_net: fetch_repo: resulting module with sum h1:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=; expected sum h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs=
ERROR: /home/james/ngrok/go/cmd/ngrok/config/BUILD.bazel:4:11: //go/cmd/ngrok/config:config depends on @@gazelle~~go_deps~org_golang_x_net//idna:idna in repository @@gazelle~~go_deps~org_golang_x_net which failed to fetch. no such package '@@gazelle~~go_deps~org_golang_x_net//idna': gazelle~~go_deps~org_golang_x_net: fetch_repo: resulting module with sum h1:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=; expected sum h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs=

[fmeum]

Ah, that makes sense. Then let's try to fix symlinks instead :-)

[jtszalay]

#1830
With this our local replace is working with the .direnv directory present.
Ah but the directory internals aren't copied. It just stops. Perhaps we need a more robust solution akin to https://github.com/moby/moby/blob/master/daemon/graphdriver/copy/copy.go