Continuous "0-epoll: Failed to dispatch handler" errors in brick logs

[PrasadDesala]

Observed behavior

Brick logs are getting spammed with continuous "0-epoll: Failed to dispatch handler" error.

[2019-01-02 09:58:49.149423] E [MSGID: 101191] [event-epoll.c:759:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler

The above events are continuously getting generated in the brick logs. On a brick log till now it has logged "4553" times and is getting increased.

Expected/desired behavior

No errors in brick logs.

Details on how to reproduce (minimal and precise)

1. Create a 3 node gcs system using valgrind.
2. Create 100 pvc.
3. Enabled brick-mux.
4. Stop/start the volume.
   Check brick logs.

Information about the environment:

Glusterd2 version used (e.g. v4.1.0 or master): v6.0-dev.94.git601ba61
Operating system used: Centos 7.6
Glusterd2 compiled from sources, as a package (rpm/deb), or container:
Using External ETCD: (yes/no, if yes ETCD version): yes; version 3.3.8
If container, which container image:
Using kubernetes, openshift, or direct install:
If kubernetes/openshift, is gluster running inside kubernetes/openshift or outside: Kubernetes

[PrasadDesala]

var-run-glusterd2-bricks-pvc-381a5faf-0e6e-11e9-af0b-525400f94cb8-subvol1-brick3-brick.log.gz
var-run-glusterd2-bricks-pvc-381a5faf-0e6e-11e9-af0b-525400f94cb8-subvol1-brick2-brick.log.gz
var-run-glusterd2-bricks-pvc-381a5faf-0e6e-11e9-af0b-525400f94cb8-subvol1-brick1-brick.log.gz
kube3-glusterd2.log.gz
kube2-glusterd2.log.gz
kube1-glusterd2.log.gz

[gpavinteractiv]

I see something similar with a newly provisioned ovirt 4.3 cluster.
Lots of :

[2019-01-15 10:07:53.646643] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler

[atinmu]

I see something similar with a newly provisioned ovirt 4.3 cluster.

What gluster version are you using? Are you using GD2 in any case? If not then it's unrelated.

@PrasadDesala Are you still seeing this happening? I tried to reproduce this couple of times but failed in replicating the problem.

[gpavinteractiv]

oVirt is using Gluster 5.2 on redhat 7.6 but you are right, that is not GD2. This bug report was one of the only result when googling the "0-epoll: Failed to dispatch handler" line

[PrasadDesala]

I see something similar with a newly provisioned ovirt 4.3 cluster.

What gluster version are you using? Are you using GD2 in any case? If not then it's unrelated.

@PrasadDesala Are you still seeing this happening? I tried to reproduce this couple of times but failed in replicating the problem.

Yes, I am still seeing this issue on the latest nightly build.
glusterd version: v6.0-dev.109.gitdfb2462
git SHA: dfb2462
go version: go1.11.2
go OS/arch: linux/amd64

[emerzon]

Seems to be same issue as reported here: https://bugzilla.redhat.com/show_bug.cgi?id=1651246
However there was no activity from devs there since November.