-
Community Support Policy
RabbitMQ version used4.0.3 How is RabbitMQ deployed?Community Docker image Steps to reproduce the behavior in questionIssue Details: Cluster Configuration - 3 node cluster running AKS Issue Description: At some point of time a message in a quorum queue is moving into a limbo state. In that state we cannot consume that message. The message is there in the queue and when we start a consumer, it will show as Unacked in the management console, but the consumer is not receiving the message.
When we tried to get the message via management console, we are getting
We tried to peek the message using Given below is the queue's quorum status, The issue is not just able to consume the message, it is the concern of growing segment files. Since this particular message is not acknowledged, RabbitMQ wont compact the segment files for this queue. So if we keep publishing messages into this queue and consume every other messages except this one (as the consumer is not getting that message), the segment files keep growing and we eventually ran out of the disk space (which happened in our production environment). It is not that we are rejecting this message and requeuing it, we cannot consume it. We did rejected messages (this and others) and requeued as part of testing all other messages came back and we consumed them. We don’t have exact count on how many time we rejected this message, but we had the same issue happened in another queue for message which is rejected 2 times only. The only solution as of now to recover from growing segment files is to purge the queue. |
Beta Was this translation helpful? Give feedback.
Answered by
dipint
Jan 15, 2025
Replies: 3 comments 11 replies
-
Are you 100% certain this is the case? Please provide a Wireshark trace showing your consumer start-up. I should be able to see the AMQP handshake, and the Almost every time I have assisted a RabbitMQ user who claims that messages are not reaching their application, it's one of these issues:
I realize that |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
Question - do you see |
Beta Was this translation helpful? Give feedback.
-
|
We are not seeing any logs like that specifically, but we see a log like |
Beta Was this translation helpful? Give feedback.
-
|
One more thing we observed, once a queue reaches into the limbo state, the leader is not stick on a single node. If we perform a read operation in the queue, then leader switches to a different node (not always though). I think that have some correlation with the above error as whenever we see that error, then looks like a restart of the queue happens with a new leader election. |
Beta Was this translation helpful? Give feedback.
-
|
this is a known bug in 4.0.2 - please upgrade to 4.0.5 |
Beta Was this translation helpful? Give feedback.
-
|
Sure, we just tuning our test case to make sure that we can can recreate the issue near consistently in the current environment, so that we can do the same testing with 4.0.5. Thanks for all the input @kjnilsson. |
Beta Was this translation helpful? Give feedback.
-
|
@dipint please try to reproduce against |
Beta Was this translation helpful? Give feedback.
4 replies
-
|
Sure @michaelklishin, we are planning to do the same, upgrade RMQ to 4.0.3 or above. Since the issue is very peculiar as we dont have any specific pattern yet to recreate the issue, the plan is to try all different things we applied in our 4.0.2 testing. Will keep you updated. |
Beta Was this translation helpful? Give feedback.
-
|
I mean specifically Our current hypothesis that the very first core server Bug Fixes issues in the |
Beta Was this translation helpful? Give feedback.
-
|
Ya, we are also going with 4.0.5. We are able to kind of recreate the issue, but there are lot of un-answered questions between us itself. So we are doing the same steps couple of more times in the current environment to kind of consistently re-create the issue and the going to upgrade to 4.0.5 and test again. |
Beta Was this translation helpful? Give feedback.
-
|
I'm fairly confident this is issue is fixed in 4.0.3 - if you can reproduce in 4.0.5 it would be good to see some logs. |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin, @kjnilsson, @lukebakken Just give an update, we updated one of our environment to 4.0.5 and did some testing. As of now with all the tests we performed we are not able to recreate the issue in 4.0.5. We are continuing our testing before we make a call to apply this version to production. I will keep you all posted. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Thank you - we appreciate your efforts and your follow-up. |
Beta Was this translation helpful? Give feedback.
Answer selected by
michaelklishin
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
@michaelklishin, @kjnilsson, @lukebakken Just give an update, we updated one of our environment to 4.0.5 and did some testing. As of now with all the tests we performed we are not able to recreate the issue in 4.0.5. We are continuing our testing before we make a call to apply this version to production. I will keep you all posted.