WIP: ability to kick failed jobs back onto queue. #3
Conversation
|
Scenario with job timing out: Fresh job: |
|
Scenario with job failing: Fresh job: |
| t, err := job.Timeouts() | ||
| if err != nil { | ||
| b.log.Panic(err) | ||
| } | ||
| if t >= TimeoutTries { | ||
| b.log.Printf("job %d has %d timeouts, burying", job.Id, t) | ||
| if (t - kicks) >= TimeoutTries { |
There was a problem hiding this comment.
i reckon this condition and the one on L130 should be pulled out into predicate methods as intent isn't super clear
There was a problem hiding this comment.
Yep definitely. WIP :)
That said, the problem was already present, this WIP commit just makes it worse.
There was a problem hiding this comment.
Ah sorry, should have paid more attention to subject line.
|
So, the only weird case here is if a task has been buried a few times because it was failing, and it then times out: New job: Fails until buried: Is kicked: Fails again, kicked again: It now times out: Reserved again: Job runs, eventually either timeouts or releases reaches the threshold. Is this even a big deal though? (edited by @pda) |
Side-note: if we use this approach, some things need to be treated as signed integers. |
Background: #2
TL;DR: kicking failed jobs causes them to be instantly re-buried based on their stats.
This change means each
kickbuys a job one morereleaseortimeoutbefore it's auto-buried./cc @lwc.
For reference, here's a sample
stats-joboutput: