[Feature] Prioritize queues when agents poll for new jobs

[bladernr]

Not sure if this is a function of connectors, or the api server, or both, but it would be very useful to specify queues by priority.

Currently an agent config can have this:

job_queues:
- doubletusk
- egx
- baremetal
- hpe
- 202101-28672
- charmed-ceph
- micro-ceph

and it's equally likely that a job pending for ANY of those queues will be picked up "next" after a current job is run.

But there are cases where we want to share machines but still need them for a primary purpose. In the above case, that machine is primarily a contractual obligation for NVIDIA testing. So it would be quite useful if we could do something like this:

priority_queues:
- egx
- doubletusk
low_priority_queues:
- 202101-28672
- baremetal
- hpe
- charmed-ceph
- micro-ceph

And then the behaviour would be the agent would poll first for jobs in the egx or doubletusk queues, and if none exist, move on to the rest in sequence.

The possible scenario is:
Person 1 has a job running on it now
Person 2 has submitted a job for baremetal
Person 3 has submitted a job for hpe
Person 4 has submitted a job for baremetal
I have submitted a job for egx (or doubletusk) to run a regression test on a new update to linux-nvidia

With priority queues, as soon as Person 1's currently executing job completes, my egx/doubletusk job which is already in queue alongside the three 'baremetal' and 'hpe' jobs will get picked up first.

And I'd imagine that the polling would need bounce to better ensure priority enable this... so:

Poll priority_queues/egx
Poll priority_queues/doubletusk
Poll low_priority_queues/202101-28672
Poll priority_queues/egx
Poll priority_queues/doubletusk
Poll low_priority_queues/baremetal
Poll priority_queues/egx
Poll priority_queues/doubletusk
Poll low_priority_queues/hpe
...

Perhaps that already quietly happens simply by order of how the queues are listed in the agent config, and if so this would just make that more obvious but if not, then it would implement a way to jump the line for priority jobs.

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CERTTF-247.

This message was autogenerated