TaskId and JobId

[kirillsalykin]

Hi,

I am still trying to implement enqueue_multi :)

But I ran into an question(s):
https://github.com/maxcountryman/underway/blob/main/src/job.rs#L907

   pub async fn cancel(&self) -> Result<bool> {
        let mut tx = self.queue.pool.begin().await?;
        let in_progress_tasks = sqlx::query_as!(
            InProgressTask,
            r#"
            select
              id as "id: TaskId",
              task_queue_name as "queue_name",
              input,
              retry_policy as "retry_policy: RetryPolicy",
              timeout,
              heartbeat,
              concurrency_key
            from underway.task
            where input->>'job_id' = $1
              and state = $2
            "#,
            self.id.to_string(),
            TaskState::Pending as TaskState
        )
        .fetch_all(&mut *tx)
        .await?;

        let mut cancelled = false;
        for in_progress_task in in_progress_tasks {
            if in_progress_task.mark_cancelled(&mut tx).await? {
                cancelled = true;
            }
        }
        tx.commit().await?;

        Ok(cancelled)
    }
}

What is job_id here? how it is different from the task_id? why do we need both?

I think it would make things a little simpler if there would be just the id which would identify a task.
Should I need a queue name in order to change task?
(sorry, I am again with this question/request :) )

1. you also probably want to have select for update to lock records, because they are about to be updated.
2. how it can be multiple tasks?
3. mark_cancelled probably should accept multiple tasks as input, cancelling tasks one by one - not the most efficient approach
4. this makes me curious how do you see relation of tasks and queues?
5. (I dont see how task can be in multiple queues?) what is the scenario(s) when tasks with the same task_id can belong to different queues?

[maxcountryman]

What is job_id here? how it is different from the task_id? why do we need both?

It's an ID of the job, which is shared amongst all tasks.

I think it would make things a little simpler if there would be just the id which would identify a task.

Jobs are composed of steps which are executed as independent tasks, so unfortunately there's no single task ID in that way.

I think part of what's tripping you up here is the fact that underway.task is implemented as a single table but the actual data model is not. Does it help if you think about it as a set of virtual tables? For an even more concrete example, take a look at pg_boss, which uses table partitions to literally create separate tables (this is something that we most likely do want to do in the future for performance reasons).

[maxcountryman]

This log output, from the step example, might also help you visualize how jobs work in practice:

2024-11-16T16:51:00.717987Z  INFO process_next_task{queue.name="example-step" task.id="019335e2-8fcf-671e-858a-11aac1be2237"}:execute{job.id=019335e2-8fcf-3e1f-3b8b-5ff879f3bdd9 step=1 steps=3}: example_step: Starting computation with n = 42
2024-11-16T16:51:00.725515Z  INFO process_next_task{queue.name="example-step" task.id="019335e2-8fee-0a49-5407-ee3293301ed3"}:execute{job.id=019335e2-8fcf-3e1f-3b8b-5ff879f3bdd9 step=2 steps=3}: example_step: Squared value: 42^2 = 1764
2024-11-16T16:51:00.728310Z  INFO process_next_task{queue.name="example-step" task.id="019335e2-8ff5-e339-ca65-6ed2568cb0d2"}:execute{job.id=019335e2-8fcf-3e1f-3b8b-5ff879f3bdd9 step=3 steps=3}: example_step: Modulo 10 of 1764 is 4

Notice how there's one job ID but each step has a unique task ID. This allows you to identified all the steps a particular enqueue of a job took as well as e.g. cancel all related steps.

[kirillsalykin]

Ok, I think i understand it.
Huge thanks for explaination!

Would it make sense to make job_id, step and steps a separate column? wdyt?

[kirillsalykin]

another question on queue name and cancellation - it seems that cancel fn cancels tasks in any queue,
but AFAIK job = multiple tasks in a specific queue, so I would expect that it should filter out tasks by queue name as well, am I being wrong?

[kirillsalykin]

(assuming each task can be in different queues and the same task can be in more than queue)

[maxcountryman]

Would it make sense to make job_id, step and steps a separate column?

I don't believe so, no.

Fundamentally the "job" concept is layered on top of the lower level primitives: task, queue, worker, and scheduler. Put another way, the lower level task queue is entirely unaware of the job concept.

That's a good and bad thing. It's less efficient to embed the job data model in JSON when querying. But the upside is the task queue implementation doesn't need to know anything about higher level concepts like jobs--they're entirely decoupled.