volume info shows the volume state as started but volume status shows it as offline

[PrasadDesala]

Observed behavior

volume info shows the volume as started but volume status shows it as offline.

[root@gluster-kube1-0 bricks]# glustercli volume status pvc-46967f93-0e6e-11e9-af0b-525400f94cb8
Volume : pvc-46967f93-0e6e-11e9-af0b-525400f94cb8
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+------+-----+
| BRICK ID | HOST | PATH | ONLINE | PORT | PID |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+------+-----+
| 4112110f-4443-4d85-9bd8-1914efcee897 | gluster-kube3-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-46967f93-0e6e-11e9-af0b-525400f94cb8/subvol1/brick1/brick | false | 0 | 0 |
| 6927f476-a7e8-40cd-8205-a7668d667ada | gluster-kube2-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-46967f93-0e6e-11e9-af0b-525400f94cb8/subvol1/brick2/brick | false | 0 | 0 |
| 2584d0c9-c6d7-4d3d-94b6-6b909fbf4b67 | gluster-kube1-0.glusterd2.gcs | /var/run/glusterd2/bricks/pvc-46967f93-0e6e-11e9-af0b-525400f94cb8/subvol1/brick3/brick | false | 0 | 0 |
+--------------------------------------+-------------------------------+-----------------------------------------------------------------------------------------+--------+------+-----+
[root@gluster-kube1-0 bricks]# glustercli volume info pvc-46967f93-0e6e-11e9-af0b-525400f94cb8

Volume Name: pvc-46967f93-0e6e-11e9-af0b-525400f94cb8
Type: Replicate
Volume ID: d5f6bb15-8d2d-4b27-a700-96e86ad972e4
State: Started
Capacity: 1.0 GiB
Transport-type: tcp
Options:
performance/io-cache.io-cache: off
performance/write-behind.write-behind: off
cluster/replicate.self-heal-daemon: on
debug/io-stats.count-fop-hits: on
performance/open-behind.open-behind: off
performance/quick-read.quick-read: off
performance/read-ahead.read-ahead: off
performance/readdir-ahead.readdir-ahead: off
debug/io-stats.latency-measurement: on
performance/md-cache.md-cache: off
Number of Bricks: 3
Brick1: gluster-kube3-0.glusterd2.gcs:/var/run/glusterd2/bricks/pvc-46967f93-0e6e-11e9-af0b-525400f94cb8/subvol1/brick1/brick
Brick2: gluster-kube2-0.glusterd2.gcs:/var/run/glusterd2/bricks/pvc-46967f93-0e6e-11e9-af0b-525400f94cb8/subvol1/brick2/brick
Brick3: gluster-kube1-0.glusterd2.gcs:/var/run/glusterd2/bricks/pvc-46967f93-0e6e-11e9-af0b-525400f94cb8/subvol1/brick3/brick

And volume start on that volume is failing with response "Volume already started"

[root@gluster-kube1-0 bricks]# glustercli volume start pvc-46967f93-0e6e-11e9-af0b-525400f94cb8
volume start failed

Response headers:
X-Gluster-Cluster-Id: 98ef5bef-583c-41ee-b594-50f3d4784679
X-Gluster-Peer-Id: 4e752b45-aa0a-4784-83f7-6b487e886b4d
X-Request-Id: 16694478-f0d7-4488-a440-6f744e476bad

Response body:
volume already started

Expected/desired behavior

volume status shows the volume as online.

Details on how to reproduce (minimal and precise)

1. Create a 3 node gcs system using vagrant.
2. with brick-mux enabled, create 100 pvc.
3. Stop and start all the volumes.
   glustercli volume stop pvc-381a5faf-0e6e-11e9-af0b-525400f94cb8
   glustercli volume stop pvc-38485e48-0e6e-11e9-af0b-525400f94cb8
   glustercli volume stop pvc-38853d16-0e6e-11e9-af0b-525400f94cb8
   glustercli volume stop pvc-38c06414-0e6e-11e9-af0b-525400f94cb8
   glustercli volume stop pvc-38f92094-0e6e-11e9-af0b-525400f94cb8
   glustercli volume stop pvc-39337ec5-0e6e-11e9-af0b-525400f94cb8
   glustercli volume stop pvc-396d7779-0e6e-11e9-af0b-525400f94cb8
   ....
   ..
   .
   Same way start all the volumes.
   glustercli volume start pvc-381a5faf-0e6e-11e9-af0b-525400f94cb8
   glustercli volume start pvc-38485e48-0e6e-11e9-af0b-525400f94cb8
   glustercli volume start pvc-38853d16-0e6e-11e9-af0b-525400f94cb8
   glustercli volume start pvc-38c06414-0e6e-11e9-af0b-525400f94cb8
   ..
   .

Information about the environment:

Glusterd2 version used (e.g. v4.1.0 or master): v6.0-dev.94.git601ba61
Operating system used: Centos 7.6
Glusterd2 compiled from sources, as a package (rpm/deb), or container:
Using External ETCD: (yes/no, if yes ETCD version): yes; version 3.3.8
If container, which container image:
Using kubernetes, openshift, or direct install:
If kubernetes/openshift, is gluster running inside kubernetes/openshift or outside: Kubernetes

• Output of statedump from any one of the node

[vpandey-RH]

@PrasadDesala I will need more information than just volume status as there could be numerous reasons as to why the Port shows as 0. One of them could be that the bricks need some time to Sign In after glusterfsd has been spawned. Can you get me the output of -
''' ps aux | grep glusterfsd ''' after you start the PVCs.

Also, can you tell me if the volume stop and start requests are sequential. I mean the start requests are sent after all stop requests have been sent or the requests don't have a particular ordering ?

Also, After all start requests have been sent and returned successful, can you give the bricks sometime to sign in and check after that if you have the same observations ?

[PrasadDesala]

@PrasadDesala I will need more information than just volume status as there could be numerous reasons as to why the Port shows as 0. One of them could be that the bricks need some time to Sign In after glusterfsd has been spawned. Can you get me the output of -
''' ps aux | grep glusterfsd ''' after you start the PVCs.

[root@gluster-kube1-0 bricks]# ps aux | grep glusterfsd
root 8113 0.0 0.0 9088 672 pts/2 S+ 12:54 0:00 grep --color=auto glusterfsd
root 21733 8.5 2.6 13825056 870656 ? Ssl 11:34 6:52 /usr/sbin/glusterfsd --volfile-server gluster-kube1-0.glusterd2.gcs --volfile-server-port 24007 --volfile-id pvc-381a5faf-0e6e-11e9-af0b-525400f94cb8.4e752b45-aa0a-4784-83f7-6b487e886b4d.var-run-glusterd2-bricks-pvc-381a5faf-0e6e-11e9-af0b-525400f94cb8-subvol1-brick1-brick -p /var/run/glusterd2/4e752b45-aa0a-4784-83f7-6b487e886b4d-var-run-glusterd2-bricks-pvc-381a5faf-0e6e-11e9-af0b-525400f94cb8-subvol1-brick1-brick.pid -S /var/run/glusterd2/3c5c17b3422e2a07.socket --brick-name /var/run/glusterd2/bricks/pvc-381a5faf-0e6e-11e9-af0b-525400f94cb8/subvol1/brick1/brick -l /var/log/glusterd2/glusterfs/bricks/var-run-glusterd2-bricks-pvc-381a5faf-0e6e-11e9-af0b-525400f94cb8-subvol1-brick1-brick.log --xlator-option *-posix.glusterd-uuid=4e752b45-aa0a-4784-83f7-6b487e886b4d

Also, can you tell me if the volume stop and start requests are sequential. I mean the start requests are sent after all stop requests have been sent or the requests don't have a particular ordering ?

Volumes are started once all the stop requests are completed.
Stop 100 volumes {1..100} --> wait till volume stop completes on all the volumes --> Start 100 volumes {1..100}

Also, After all start requests have been sent and returned successful, can you give the bricks sometime to sign in and check after that if you have the same observations ?

Its been more than an hour I hit this issue. Still the volume status shows as not online.

I see this error in glusterd2 logs when I tried to start the volume. path "/var/run/glusterd2/bricks/pvc-46967f93-0e6e-11e9-af0b-525400f94cb8/subvol1/brick3/brick" is present though.
[root@gluster-kube1-0 bricks]# ll /var/run/glusterd2/bricks/pvc-46967f93-0e6e-11e9-af0b-525400f94cb8/subvol1/brick3/brick
total 0

time="2019-01-02 11:49:10.525084" level=error msg="registry.SearchByBrickPath() failed for brick" brick=/var/run/glusterd2/bricks/pvc-46967f93-0e6e-11e9-af0b-525400f94cb8/subvol1/brick3/brick error="SearchByBrickPath: port for brick /var/run/glusterd2/bricks/pvc-46967f93-0e6e-11e9-af0b-525400f94cb8/subvol1/brick3/brick not found" source="[rpc_prog.go:104:pmap.(*GfPortmap).PortByBrick]"

@vpandey-RH Let me know if you need any other information, I have the system in the same state.

[atinmu]

@vpandey-RH did we figure out what's the cause of this state?