Gazelle: Java GenerateRules seems unable to connect initially to gRPC

[Mercanuis]

My team is using Bazel and Gazelle to manage our CI and help to build our projects. Lately, we've been seeing interesting behavior with Java and gRPC. Running the following configuration for gazelle

# https://github.com/bazelbuild/bazel-gazelle/releases/
http_archive(
    name = "bazel_gazelle",
    integrity = "sha256-dd8ojEsxyB61D1Hi4U9HY8t1SNquEmgXJHBkY3/Z6mI=",
    patch_args = ["-p1"],
    patches = [
        "//third_party/.patches:bazel_gazelle.generate_repo_config_fix_slash.patch",
    ],
    urls = [
        "https://mirror.bazel.build/github.com/bazelbuild/bazel-gazelle/releases/download/v0.36.0/bazel-gazelle-v0.36.0.tar.gz",
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.36.0/bazel-gazelle-v0.36.0.tar.gz",
    ],
)

And this is the BUILD file (potential PII masked just to not get in trouble):

# gazelle:resolve java javax.ws.rs.core @maven//:org_jboss_spec_javax_ws_rs_jboss_jaxrs_api_2_1_spec
# gazelle:resolve java org.keycloak @maven//:org_keycloak_keycloak_server_spi_private
# gazelle:resolve java org.keycloak.models @maven//:org_keycloak_keycloak_server_spi
# gazelle:resolve java org.keycloak.provider @maven//:org_keycloak_keycloak_server_spi
# gazelle:resolve java org.keycloak.provider @maven//:org_keycloak_keycloak_services

load("@rules_java//java:defs.bzl", "java_binary", "java_library")

java_library(
    name = "xxx",
    srcs = [
        "BurrConstants.java",
        "BurrProtocolMapper.java",
    ],
    javacopts = [
        "--release",
        "17",
    ],
    resources = ["//ns/id/keycloak-spi/burr-protocol-mapper/src/main/resources/META-INF/services:resource_files"],
    visibility = ["//:__subpackages__"],
    exports = [
        "@maven//:org_keycloak_keycloak_core",
        "@maven//:org_keycloak_keycloak_services",
    ],
    deps = [
        "@maven//:com_fasterxml_jackson_core_jackson_databind",
        "@maven//:jakarta_ws_rs_jakarta_ws_rs_api",
        "@maven//:org_jboss_logging_jboss_logging",
        "@maven//:org_keycloak_keycloak_core",
        "@maven//:org_keycloak_keycloak_server_spi",
        "@maven//:org_keycloak_keycloak_server_spi_private",  #keep
        "@maven//:org_keycloak_keycloak_services",
    ],
)

java_binary(
    name = "BurrMapper",
    main_class = "com.xxx.BurrProtocolMapper",
    visibility = ["//:__subpackages__"],
    runtime_deps = [":xxx"],
)

We run gazelle with a top level BUILD file with the following:

load("@aspect_rules_js//js:defs.bzl", "js_library")
load("@bazel_gazelle//:def.bzl", "gazelle")
load("@com_github_bazelbuild_buildtools//buildifier:def.bzl", "buildifier")
load("@npm//:commitlint/package_json.bzl", commitlint_bin = "bin")
load("@npm//:defs.bzl", "npm_link_all_packages")
load("@pip//:requirements.bzl", "all_whl_requirements", "entry_point")
load("@rules_python//python:pip.bzl", "compile_pip_requirements")
load("@rules_python_gazelle_plugin//manifest:defs.bzl", "gazelle_python_manifest")
load("@rules_python_gazelle_plugin//modules_mapping:def.bzl", "modules_mapping")
load("//ns/bazel_debian/debian_package_manager:debian_package_manager.bzl", "debian_package_manager")
load("//ns/bazel_gazelle:def.bzl", "GAZELLE_RUNTIME_DEPS")
load("//ns/bazel_go:go_mod.bzl", "go_mod")

#######################################
# Gazelle
#######################################

# gazelle:k8s_kustomize_extension enabled

# gazelle directives: https://github.com/bazelbuild/bazel-gazelle/#directives
# gazelle:prefix github.com/wp-wcm/xxx
# gazelle:build_file_name BUILD,BUILD.bazel

# Excludes
# gazelle:exclude node_modules
# gazelle:exclude .git
# gazelle:exclude **/node_modules/**



# gazelle java directives: https://github.com/bazel-contrib/rules_jvm/tree/main/java/gazelle#configuration-options
# gazelle:java_maven_install_file maven_install.json

# Correct Resolution of Specific Libraries
# gazelle:resolve java com.amazonaws.auth @maven//:com_amazonaws_aws_java_sdk_core
# gazelle:resolve java io.swagger.v3.oas.annotations @maven//:io_swagger_core_v3_swagger_annotations
# gazelle:resolve java org.apache.avro @maven//:org_apache_avro_avro
# gazelle:resolve java org.apache.avro.generic @maven//:org_apache_avro_avro
# gazelle:resolve java org.apache.kafka.clients.consumer @maven//:org_apache_kafka_kafka_clients
# gazelle:resolve java org.apache.kafka.clients.producer @maven//:org_apache_kafka_kafka_clients
# gazelle:resolve java org.apache.kafka.common.config @maven//:org_apache_kafka_kafka_clients
# gazelle:resolve java org.apache.kafka.common.header @maven//:org_apache_kafka_kafka_clients
# gazelle:resolve java org.apache.kafka.common.serialization @maven//:org_apache_kafka_kafka_clients
# gazelle:resolve java org.apache.kafka.common.utils @maven//:org_apache_kafka_kafka_clients
# gazelle:resolve java org.apache.kafka.streams @maven//:org_apache_kafka_kafka_streams
# gazelle:resolve java org.apache.kafka.streams.processor.api @maven//:org_apache_kafka_kafka_streams
# gazelle:resolve java org.hamcrest @maven//:org_hamcrest_hamcrest
# gazelle:resolve java javax.annotation @maven//:javax_annotation_javax_annotation_api
# gazelle:resolve java javax.persistence @maven//:javax_persistence_javax_persistence_api
# gazelle:resolve java javax.validation.constraints @maven//:javax_validation_validation_api
# gazelle:resolve java javax.validation @maven//:javax_validation_validation_api
# gazelle:resolve java org.apache.flink.table.api @maven//:org_apache_flink_flink_table_api_java
# gazelle:resolve java org.apache.flink.table.functions @maven//:org_apache_flink_flink_table_api_java
# gazelle:resolve java org.apache.spark.sql @maven//:org_apache_spark_spark_sql_2_12

# protobuf resolutions
# gazelle:resolve proto proto google/api/annotations.proto @googleapis//google/api:annotations_proto
# gazelle:resolve proto go google/api/annotations.proto @org_golang_google_genproto_googleapis_api//annotations

# java protobuf and grpc-java resolutions
# gazelle:resolve java com.google.protobuf @com_google_protobuf//:protobuf_java
# gazelle:resolve java com.google.protobuf.util @com_google_protobuf//:protobuf_java_util
# gazelle:resolve java io.grpc @io_grpc_grpc_java//api
# gazelle:resolve java io.grpc.alts @io_grpc_grpc_java//alts
# gazelle:resolve java io.grpc.auth @io_grpc_grpc_java//auth
# gazelle:resolve java io.grpc.census @io_grpc_grpc_java//census
# gazelle:resolve java io.grpc.context @io_grpc_grpc_java//context
# gazelle:resolve java io.grpc.googleapis @io_grpc_grpc_java//googleapis
# gazelle:resolve java io.grpc.grpclb @io_grpc_grpc_java//grpclb
# gazelle:resolve java io.grpc.inprocess @io_grpc_grpc_java//inprocess
# gazelle:resolve java io.grpc.netty @io_grpc_grpc_java//netty
# gazelle:resolve java io.grpc.netty.shaded @io_grpc_grpc_java//netty:shaded_maven
# gazelle:resolve java io.grpc.okhttp @io_grpc_grpc_java//okhttp
# gazelle:resolve java io.grpc.protobuf @io_grpc_grpc_java//protobuf
# gazelle:resolve java io.grpc.protobuf.lite @io_grpc_grpc_java//protobuf-lite
# gazelle:resolve java io.grpc.protobuf.services @io_grpc_grpc_java//services:services_maven
# gazelle:resolve java io.grpc.rls @io_grpc_grpc_java//rls
# gazelle:resolve java io.grpc.services @io_grpc_grpc_java//services:services_maven
# gazelle:resolve java io.grpc.stub @io_grpc_grpc_java//stub
# gazelle:resolve java io.grpc.stub.annotations @io_grpc_grpc_java//stub
# gazelle:resolve java io.grpc.testing @io_grpc_grpc_java//testing
# gazelle:resolve java io.grpc.util @io_grpc_grpc_java//util
# gazelle:resolve java io.grpc.xds @io_grpc_grpc_java//xds:xds_maven
# gazelle:resolve java io.grpc.xds.orca @io_grpc_grpc_java//xds:xds_maven

# Custom Annotation Processors for Java
# gazelle:resolve java lombok //tools/languages/java:lombok

gazelle(
    name = "_gazelle",
    data = GAZELLE_RUNTIME_DEPS,
    gazelle = "//ns/bazel_gazelle:gazelle",
)

# The `gazelle` macro defines two binary rules, and the tags apply to both.
# To prevent this issue, we create an alias to the actual gazelle target, so
# there is only 1 tagged binary for gazelle. This is also the binary that is
# executed when running `bazel run //:gazelle`.
# Manual is also added as a tag, as the default rules are tagged with it.
# See here: https://github.com/bazelbuild/bazel-gazelle/blob/master/def.bzl#L150-L155
alias(
    name = "gazelle",
    actual = ":_gazelle",
    tags = [
        "agora_auto_formatter",
        "manual",
    ],
)

So when we run bazel run //:gazelle we get a connection error. The initial connection refuses or times out, and running the same target succeeds a second later:

 ~/Workspaces/xxx/ [yyy] bazel run //:gazelle
Starting local Bazel server and connecting to it...
INFO: Invocation ID: ...
INFO: Streaming build results to: ...
INFO: Analyzed target //:gazelle (490 packages loaded, 17843 targets configured).
INFO: Found 1 target...
Target //:_gazelle up-to-date:
  bazel-bin/_gazelle-runner.bash
  bazel-bin/_gazelle
INFO: Elapsed time: 29.411s, Critical Path: 1.25s
INFO: 574 processes: 573 remote cache hit, 1 internal.
INFO: Build completed successfully, 574 total actions
INFO: Running command line: bazel-bin/_gazelle
INFO: Streaming build results to: https://wavellite.cluster.engflow.com/invocation/2a23188b-bb2f-4d5a-91b3-7b3d79a489bc
8:14AM FTL external/contrib_rules_jvm/java/gazelle/generate.go:79 > Failed to parse package error="Unavailable: connection error: desc = \"transport: Error while dialing: dial tcp [::1]:52706: connect: connection refused\"" package=ns/id/keycloak-spi/burr-protocol-mapper/src/main/java/com/xxx rel=ns/id/keycloak-spi/burr-protocol-mapper/src/main/java/com/xxx step=GenerateRules


 ~/Workspaces/xxx/ [yyy] bazel run //:gazelle
INFO: Invocation ID: ...
INFO: Streaming build results to: ...
INFO: Analyzed target //:gazelle (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:_gazelle up-to-date:
  bazel-bin/_gazelle-runner.bash
  bazel-bin/_gazelle
INFO: Elapsed time: 0.403s, Critical Path: 0.01s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
INFO: Running command line: bazel-bin/_gazelle
INFO: Streaming build results to: https://wavellite.cluster.engflow.com/invocation/d055266d-91b4-4565-a490-29effbf1d6f9
 ~/Workspaces/xxx/ [yyy] 

Asking our team Bazel Experts, this seems to be something related to the java parser that is spawned from gazelle when it runs the target and the client trying to connect to grab the scanned java files. What makes this behavior odd is its not consistent; the second run of the target always runs correctly while the first one shows this issue (and gazelle fails silently)