Should databend meta be attempting to find an active leader if it is started with the join command?

[kkapper]

Databend Metasrv

Version: v1.2.646-nightly-86cd6084b1-simd(1.81.0-nightly-2024-10-16T03:15:57.008930390Z)
Working DataVersion: V003(2024-05-31: Store snapshot in rotbl)

Raft Feature set:
    Server Provide: { append:v0, install_snapshot:v1, install_snapshot:v3, vote:v0 }
    Client Require: { append:v0, install_snapshot:v3, vote:v0 }

On Disk Data:
    Dir: /data/databend-meta/raft
    DataVersion: V003(2024-05-31: Store snapshot in rotbl)
    In-Upgrading: None

Log:
    File: enabled=true, level=INFO, dir=/data/databend-meta/log, format=json
    Stderr: enabled=true, level=WARN, format=text
Id: 1
Raft Cluster Name: databend
Raft Dir: /data/databend-meta/raft
Raft Status: join [
    "plaid-databend-meta-0.plaid-databend-meta.pw-axc01.svc.cluster.local:28004",
]

HTTP API
   listening at 0.0.0.0:28002
gRPC API
   listening at 0.0.0.0:9191
   advertise:  plaid-databend-meta-1.plaid-databend-meta.pw-axc01.svc.cluster.local:9191
Raft API
   listening at 0.0.0.0:28004
Upgrade ondisk data if out of date: version: V003, upgrading: None
    Find and clean previous unfinished upgrading
Upgrade ondisk data finished: version: V003, upgrading: None
   advertise:  plaid-databend-meta-1.plaid-databend-meta.pw-axc01.svc.cluster.local:28004

2024-11-06T16:24:40.498016Z  WARN databend_meta::api::http_service: http_service.rs:117 Http API TLS not set
Wait for 180s for active leader...
2024-11-06T16:24:42.917310Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->2 target=2
2024-11-06T16:24:42.917422Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->0 target=0
2024-11-06T16:24:44.419911Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->2 target=2
2024-11-06T16:24:44.420047Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->0 target=0
2024-11-06T16:24:45.923420Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->2 target=2
2024-11-06T16:24:45.923456Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->0 target=0
2024-11-06T16:24:47.423591Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->2 target=2
2024-11-06T16:24:47.423603Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->0 target=0
2024-11-06T16:24:48.925472Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->2 target=2
2024-11-06T16:24:48.925487Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->0 target=0
2024-11-06T16:24:50.428644Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->0 target=0
2024-11-06T16:24:50.428702Z ERROR openraft::core::raft_core: raft_core.rs:1099 timeout error=timeout after 1s when Vote 1->2 target=2

So we are starting node 0 as the master for sure in this case, and node 1(in this example) is started with the command to join 0. But once started it ends up as a candidate and will never rejoin.

This is not a network permission related error or anything of the sort. And this is much more likely to happen if one were to backup and restore a meta cluster