Creating multiple duplicate instance of shovel in case of pod restart.

[nilaytiwari]

I am creating shovel plugin in rabbit mq, that is working fine with one pod, However, We are running on Kubernetes cluster with multiple pods and in case of pod restart, it is creating multiple instance of shovel on each pod independently, which is causing duplicate message replication on destination.

Detail steps are below

1. We are deploying rabbit mq on Kubernetes cluster using helm chart.
2. After that we are creating shovel using Rabbit MQ Management UI. Once we are creating it from UI, shovels are working fine and not replicating data multiple time on destination.
3. When any pod get restarted, it create separate shovel instance. That start causing issue of duplicate message replication on destination from different shovel instance.
4. When we saw shovel status on Rabbit MQ UI then we found that, there are multiple instance of same shovel running on each pod.
5. When we start shovel from Rabbit MQ UI manually, then it will resolved this issue and only once instance will be visible in UI.

So problem which we concluded that, in case of pod failure/restart, shovel is not able to sync with other node/pod, if any other shovel is already running on node/pod. Since we are able to solve this issue be restarting of shovel form UI, but this not a valid approach for production. This issue we are not getting in case of queue and exchange.

Can anyone help us here to resolve this issue

[michaelklishin]

I will convert this issue to a GitHub discussion. Currently GitHub will automatically close and lock the issue even though your question will be transferred and responded to elsewhere. This is to let you know that we do not intend to ignore this but this is how the current GitHub conversion mechanism makes it seem for the users :(

[michaelklishin]

Duplicate of #3154. We could not reproduce this behavior and explained the only scenario where multiple instances of a Shovel can possible create duplicates: if the source is an exchange. Using queue(s) as sources won't produce any duplicates because the shovels will be competing consumers.

Without any RabbitMQ information or logs from all nodes there isn't much we can do beyond trying to reproduce again.

[michaelklishin]

A member of our team who works a lot on/with Kubernetes Operators will try to reproduce as well. I must be missing something since my latest attempt still doesn't reproduce what you observe #3154 (comment)

[michaelklishin]

@mkuratczyk helped me reproduce, moving to #3260