Current step implementation is not "scalable"

[kirillsalykin]

I've been considering the current steps approach, and it seems potentially limiting for scalability.

What do I mean?

Take a standard example: generating a PDF and, once ready, sending an email with it. Suppose you already have a job specifically for sending emails. To send this PDF email, you’d need a unique send email step just for this scenario.

Another example:
If you want to calculate a job_queue_length metric, the steps approach makes it a bit trickier. For example, the queue length of send-email jobs is hidden within the steps structure, making it harder to track directly.

This is why I think having dependencies between tasks might be a better fit.

For instance, a task_dep table could look something like this:

task_id
depends_on_task_id

When the system dequeues the next task, it would check to ensure that all dependencies are completed before proceeding.

WDYT?

UPDATE:
I actually like the syntax of the steps approach, so perhaps we could find a middle ground, where adding dependencies doesn't complicate things too much.

[maxcountryman]

Suppose you already have a job specifically for sending emails.

This is beyond the scope of Underway directly and therefore left out of the examples, but generally the approach that I take and would recommend is to implement generalized business logic independently of the job or task.

For example, I have a "service" (sorry, this is an overloaded term, here I mean the design pattern and not e.g. a Tower service) that encapsulates sending emails. This service handles sending emails and can be called from anywhere the service is made available in my application--that could be in a job, different jobs, outside a job, etc.

In its simplest form, I think having a set of functions that manage email sending is a good start, then each job that sends emails wraps such a function. This is a more general code reuse pattern but applies here as well.

Regarding queue length, that actually should be straightforward: it's just len of the steps Vec. It's not currently exposed, but it would be trivial to provide a method on Job that returns this if it's useful to you.

[kirillsalykin]

Ok, so there is one task, which is happen to be multistep, it is not represented as related tasks; do I get semantics correctly?

Based on famous example with two steps: pdf and email - there will be one task with input containing both pdf and email related data, and not two related tasks (one for pdf generation and one for email delivery)?

[maxcountryman]

there will be one task with input containing both pdf and email related data

Exactly: Task is implemented for Job and internally Job is wrapping input in serde_json::Value along with other state. So yes, there's only one input type. Externally it appears as though there's two separate tasks and in fact the Job implementation of Task will enqueue itself twice in this example: one time for the first step (PDF) and another time for the second step (email). Again internally there's some simple state tracking happening that indicates which step should be executed, but the reality is there's only one Task here. The result of all this is that we get independent executions of steps but a concrete type that the queue manages: this is basically a fancier version of the multitask example that allows for type safe steps without having to define an enum input type upfront.

[kirillsalykin]

Thank you for explanation!