Ack message for time consuming jobs

[amindadgar]

After a talk with @scientiststwin and @cyri113, I found out that we would send an ack message when the job was ended. For jobs that would need time I thought it wouldn't be a wise solution to send the ack messages since we could get a time-out error.

The possible solutions we talked about was

1. Increase the timeout for a specific queue that its job is taking long (Maybe up to 6 hours)
2. Send the ack message before doing the job.

For more information, in time consuming jobs we would publish a message after their job was ended so I think we can go with the second solution, but it could change the integrity of the program which some parts are sending ack message after the job was done and some other parts are sending ack messages before doing the job (We couldn't be sure whether the job is done successfully or not).

• If we want to make all acks to be sent before the job, Then for error handling and job ending I think we would need to Publish a message.
• In the miro board for a time consuming job like analytics we had assumed to fire an event (publish a message), when the job was successful or an error occurred (Which I'm not sure with the described problem what to do with its acks).

@mehrdadmms what's your thoughts about that?

[cyri113]

Based on 3 conversations I had today. We have version system components that will need to implement job queues.

So the question is, should we use RabbitMQ or another solution.

If we implement a job queue, the message consumer would receive the message, store it in the job queue and respond with ack. As mentioned by @Behzad-rabiei, RabbitMQ would be the postman.

Then the message would be picked up by a worker in the job queue.

Similar issue: TogetherCrew/discord-bot#23

Node: BullMQ
Python: Celery, RQ, Huey

Other question is whether we make a shared redis instance or one per system component.