strange distribution behaviour (Transport endpoint is not connected)

[cinatic]

Hi people,

after a outage of a node (server1) in a 3 node cluster i realized the files are actually not distributed as i expected they would and that some files were missing when they are requested. I have the following setup and thought that the bricks would be. Am i wrong with my assumption that with my setup the data should be distributed across 3 nodes?

gluster 9.0-1 (upgraded now to 9.3)
5.11.1-arch1-1

server3: gluster volume info vol1

Volume Name: vol1
Type: Distributed-Replicate
Volume ID: e2d773e0-1cc0-49d9-9211-ae573290c89f
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x 2 = 6
Transport-type: tcp
Bricks:
Brick1: server1:/srv/bricks/g1/brick
Brick2: server2:/srv/bricks/g1/brick
Brick3: server1:/srv/bricks/g2/brick
Brick4: server3:/srv/bricks/g1/brick
Brick5: server2:/srv/bricks/g2/brick
Brick6: server3:/srv/bricks/g2/brick
Options Reconfigured:
cluster.server-quorum-type: server
cluster.quorum-type: auto
performance.cache-size: 1GB
nfs.trusted-sync: on
performance.client-io-threads: on
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
performance.io-cache: on
performance.io-thread-count: 4
performance.readdir-ahead: on
performance.read-ahead: disable
server.allow-insecure: on
cluster.lookup-optimize: on
client.event-threads: 4
server.event-threads: 4
performance.stat-prefetch: on
cluster.readdir-optimize: on
performance.write-behind-window-size: 8MB

peer status shows that the peer is offline

server3# gluster peer status
Number of Peers: 2

Hostname: server3
Uuid: 476055b7-e2d6-400b-a4a5-974f8099c12d
State: Peer in Cluster (Connected)

Hostname: server1
Uuid: 02d3b7d9-562d-4711-b059-7a71bb134e18
State: Peer in Cluster (Disconnected)

but volume status does not show the bricks of the disconnected (i would expect it to show as offline)

server3# gluster volume status vol1
Status of volume: vol1
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick server2:/srv/bricks/g1/brick      49154     0          Y       3779886
Brick server3:/srv/bricks/g1/brick    49152     0          Y       484520
Brick server2:/srv/bricks/g2/brick      49155     0          Y       3779893
Brick server3:/srv/bricks/g2/brick    49153     0          Y       484527
Self-heal Daemon on localhost               N/A       N/A        Y       484044
Self-heal Daemon on server2             N/A       N/A        Y       3780390

heal status looks also very strange, as server2 is online, stated by peer status

server3# gluster volume heal vol1 info
Brick server1:/srv/bricks/g1/brick
Status: Transport endpoint is not connected
Number of entries: -

Brick server2:/srv/bricks/g1/brick
Status: Connected
Number of entries: 0

Brick server1:/srv/bricks/g2/brick
Status: Transport endpoint is not connected
Number of entries: -

Brick server3:/srv/bricks/g1/brick
Status: Transport endpoint is not connected
Number of entries: -

Brick server2:/srv/bricks/g2/brick
Status: Connected
Number of entries: 0

Brick server3:/srv/bricks/g2/brick
Status: Transport endpoint is not connected
Number of entries: -

[cinatic]

ok after the outage with all nodes everything is running as it should. i used the opportunity to upgrade each node to gluster 9.3 and tried to reproduce the issue, so i shutdown server1 again and have exact the same problem. i checked if the file is actually available on any of the bricks left and actually the brick on server3 has the file:

server3$ ll /srv/bricks/g2/brick/data/020e753ec3b54652865c995b95f7dde5.jpg
54731242 -rw-r--r-- 2 root root 19K Nov 19  2020 /srv/bricks/g2/brick/data/020e753ec3b54652865c995b95f7dde5.jpg

but the FUSE drive still seems to request server1 which is down:

server2$ ll /mnt/gluster_vol1/data/020e753ec3b54652865c995b95f7dde5.jpg
'/mnt/gluster_vol1/data/020e753ec3b54652865c995b95f7dde5.jpg': Transport endpoint is not connected

[cinatic]

i read some documentation and it turned out that it might be an configuration issue. After switching back the default value for the following options FUSE client will do a proper failover:

gluster volume set vol1 performance.client-io-threads off
gluster volume set vol1 cluster.lookup-optimize off
gluster volume set vol1 readdir-ahead off
gluster volume set vol1 performance.readdir-ahead off
gluster volume set vol1 cluster.readdir-optimize off