job-status-consumer: does not reconnect when RabbitMQ connection fails #436
Labels
Comments
VMois
pushed a commit
to VMois/reana
that referenced
this issue
Mar 25, 2022
- set job-status-consumer RabbitMQ prefetch value to 10; - this should reduce the risk of getting 406 PRECONDITION error on message ack. addresses reanahub/reana-workflow-controller#436
VMois
pushed a commit
to VMois/reana
that referenced
this issue
Mar 25, 2022
- set job-status-consumer RabbitMQ prefetch value to 10; - this should reduce the risk of getting 406 PRECONDITION error on message ack, and possibly consumer disconnection issue. addresses reanahub/reana-workflow-controller#436
VMois
pushed a commit
to VMois/reana
that referenced
this issue
Mar 25, 2022
- set job-status-consumer RabbitMQ prefetch value to 10; - this should reduce the risk of getting 406 PRECONDITION error on message ack, and possibly consumer disconnection issue. addresses reanahub/reana-workflow-controller#436
VMois
pushed a commit
to VMois/reana
that referenced
this issue
Mar 29, 2022
- set job-status-consumer RabbitMQ prefetch value to 10; - this should reduce the risk of getting 406 PRECONDITION error on message ack, and possibly consumer disconnection issue. addresses reanahub/reana-workflow-controller#436
VMois
pushed a commit
to VMois/reana-workflow-controller
that referenced
this issue
Mar 30, 2022
VMois
pushed a commit
to VMois/reana-workflow-controller
that referenced
this issue
Mar 31, 2022
VMois
pushed a commit
to VMois/reana
that referenced
this issue
Mar 31, 2022
- set job-status-consumer RabbitMQ prefetch value to 10; - this should reduce the risk of getting 406 PRECONDITION error on message ack, and possibly consumer disconnection issue. addresses reanahub/reana-workflow-controller#436
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Originated from reanahub/reana-job-controller#357 (comment)
After connection to RabbitMQ fails in
job-status-consumer, sometimes, it does not automatically reconnect. So manual intervention is needed to restart ther-w-controllerpod.This is unusual behavior because Kombu-based consumers should reconnect on failure.
Some of the errors that might cause the issue:
I found this comment that might explain why the consumer is disconnecting, and also offers a possible solution to reduce
PREFETCH_COUNT.But, it might also indicate some deeper problems with consumer logic. For example, when the workflow is finished/failed it can take up to a few minutes for the consumer to process this message because of recalculating quotas for the workflow.
The text was updated successfully, but these errors were encountered: