From 84fb9cb3ca7fc25c39a9e39552ea743231aa024f Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Wed, 3 Apr 2024 13:10:25 +0200 Subject: [PATCH 1/6] Propose queue changeTaskRunPriority --- README.md | 1 + rfcs/0190-queue-change-task-run-priority.md | 55 +++++++++++++++++++++ rfcs/README.md | 1 + 3 files changed, 57 insertions(+) create mode 100644 rfcs/0190-queue-change-task-run-priority.md diff --git a/README.md b/README.md index 39d0bad..e663e44 100644 --- a/README.md +++ b/README.md @@ -68,3 +68,4 @@ See [mechanics](mechanics.md) for more detail. | RFC#180 | [Github cancel previous tasks](rfcs/0180-Github-cancel-previous-tasks.md) | | RFC#182 | [Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub](rfcs/0182-taskcluster-yml-remote-references.md) | | RFC#189 | [Batch APIs for task definition, status and index path](rfcs/0189-batch-task-apis.md) | +| RFC#190 | [Queue change task run priority](rfcs/0190-queue-change-task-run-priority.md) | diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md new file mode 100644 index 0000000..38903c7 --- /dev/null +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -0,0 +1,55 @@ +# RFC 0190 - Queue change task run priority +* Comments: [#0190](https://github.com/taskcluster/taskcluster-rfcs/pull/190) +* Proposed by: @lotas + +## Summary + +Since we refactored queue internals it became possible to see what tasks are pending for a particular +worker pool / task queue. + +This RFC proposes new API method that would allow to change the priority of the existing task run. + +## Motivation + +There might be several use-cases when some worker pool has a lot of pending tasks and there is a need +to move some more important task to the top of the waiting list, or also move down less important ones. + +Instead of cancelling existing tasks we can instead choose to change task run priorities. + +## Details + +I propose we implement this on a task run level and not the task itself: + +* tasks are supposed to be immutable +* changing the priority makes more sense in the context of a specific run, not the whole task + +Queue service will expose new method `queue.changeTaskRunPriority(taskId, runId, newPriority)`. + +This will try to do the following: + +* update `task.runs[runId]` in the database to reflect priority change (possibly extra field to keep the original) +* update `priority` field in the `queue_pending_tasks` table + +If any of this fails (for example task is being changed by some other process), this method will also fail. + +This will also require introduction of the new scopes (anyOf): + +* `queue:change-task-run-priority-in-queue:` +* `queue:change-task-run-priority:` + + +Current order for picking up tasks is based on the task priority and insertion time (FIFO). +This RFC proposes to change the priority only, and leave the insertion time as is. + + +### Other considerations + +Chain of Trust validation might be broken if we change too much of the task definition or run. +We need to make sure we don't break it with this change. + + + + +## Implementation + +_pending_ \ No newline at end of file diff --git a/rfcs/README.md b/rfcs/README.md index 7d005a2..50d283f 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -56,3 +56,4 @@ | RFC#180 | [Github cancel previous tasks](0180-Github-cancel-previous-tasks.md) | | RFC#182 | [Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub](0182-taskcluster-yml-remote-references.md) | | RFC#189 | [Batch APIs for task definition, status and index path](0189-batch-task-apis.md) | +| RFC#190 | [Queue change task run priority](0190-queue-change-task-run-priority.md) | From a80bc75484c048199c69b6907607085c12b9022e Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Tue, 28 May 2024 14:49:29 +0200 Subject: [PATCH 2/6] Include more-granular scopes, typo fix --- rfcs/0190-queue-change-task-run-priority.md | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index 38903c7..4a7a18d 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -12,7 +12,7 @@ This RFC proposes new API method that would allow to change the priority of the ## Motivation There might be several use-cases when some worker pool has a lot of pending tasks and there is a need -to move some more important task to the top of the waiting list, or also move down less important ones. +to move some more important tasks to the top of the waiting list, or also move down less important ones. Instead of cancelling existing tasks we can instead choose to change task run priorities. @@ -36,20 +36,17 @@ This will also require introduction of the new scopes (anyOf): * `queue:change-task-run-priority-in-queue:` * `queue:change-task-run-priority:` - +* `queue:lower-task-run-priority:` to only allow lowering the priority +* `queue:raise-task-run-priority:` to only allow raising the priority Current order for picking up tasks is based on the task priority and insertion time (FIFO). This RFC proposes to change the priority only, and leave the insertion time as is. - ### Other considerations Chain of Trust validation might be broken if we change too much of the task definition or run. We need to make sure we don't break it with this change. - - - ## Implementation _pending_ \ No newline at end of file From 24e74fa919a515fe6695c8b1dc54baa7cb62c30f Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Tue, 28 May 2024 15:16:20 +0200 Subject: [PATCH 3/6] Update the proposal from task-run level to the task and task group levels --- README.md | 2 +- rfcs/0190-queue-change-task-run-priority.md | 56 ++++++++++++++------- rfcs/README.md | 2 +- 3 files changed, 41 insertions(+), 19 deletions(-) diff --git a/README.md b/README.md index e663e44..954f386 100644 --- a/README.md +++ b/README.md @@ -68,4 +68,4 @@ See [mechanics](mechanics.md) for more detail. | RFC#180 | [Github cancel previous tasks](rfcs/0180-Github-cancel-previous-tasks.md) | | RFC#182 | [Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub](rfcs/0182-taskcluster-yml-remote-references.md) | | RFC#189 | [Batch APIs for task definition, status and index path](rfcs/0189-batch-task-apis.md) | -| RFC#190 | [Queue change task run priority](rfcs/0190-queue-change-task-run-priority.md) | +| RFC#190 | [Queue change task / task group priority](rfcs/0190-queue-change-task-run-priority.md) | diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index 4a7a18d..29a9982 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -1,4 +1,4 @@ -# RFC 0190 - Queue change task run priority +# RFC 0190 - Queue change task / task group priority * Comments: [#0190](https://github.com/taskcluster/taskcluster-rfcs/pull/190) * Proposed by: @lotas @@ -7,45 +7,67 @@ Since we refactored queue internals it became possible to see what tasks are pending for a particular worker pool / task queue. -This RFC proposes new API method that would allow to change the priority of the existing task run. +This RFC proposes new API method that would allow to change the priority of the existing task run / task group. ## Motivation There might be several use-cases when some worker pool has a lot of pending tasks and there is a need to move some more important tasks to the top of the waiting list, or also move down less important ones. -Instead of cancelling existing tasks we can instead choose to change task run priorities. +Instead of cancelling some existing tasks we can change task run priorities. + +Use-case examples: + +* We have a high-priority patch running tasks on Try. We want to bump the priority of all tasks in this task group +* We have a chemspill release or high priority nightly to get out. We want to bump the priority of all tasks in one or more groups +* We are waiting for a specific task in a low resource pool (eg: mac hardware). We want to bump the priority of that specific task +* There are several tasks with the same highest priority and one of them is more important. We want to lower the priority of the less important task ## Details -I propose we implement this on a task run level and not the task itself: +Queue service will expose new methods: + +* `queue.changeTaskPriority(taskId, newPriority)` +* `queue.changeTaskGroupPriority(taskGroupId, newPriority)` + +New priority would be stored along the task definition for a given task or all tasks within the task group. -* tasks are supposed to be immutable -* changing the priority makes more sense in the context of a specific run, not the whole task +The process to change single task will be as follows: -Queue service will expose new method `queue.changeTaskRunPriority(taskId, runId, newPriority)`. +* new task priority will be stored in the database as `task.priority_override` column +* if task was scheduled already, its priority in the `queue_pending_tasks` table will be updated +* if the tasks run fails and new one is created, it will check if `task.priority_override` is set and use it instead of the original priority -This will try to do the following: +The process to change the whole task group will be as follows: -* update `task.runs[runId]` in the database to reflect priority change (possibly extra field to keep the original) -* update `priority` field in the `queue_pending_tasks` table +* all tasks in the task group will be updated to store the new priority in the `task.priority_override` column +* all currently scheduled tasks would be attempted to be updated in the `queue_pending_tasks` table +* all unscheduled tasks and tasks that will be restarted will use the new priority -If any of this fails (for example task is being changed by some other process), this method will also fail. +### New scopes This will also require introduction of the new scopes (anyOf): -* `queue:change-task-run-priority-in-queue:` -* `queue:change-task-run-priority:` -* `queue:lower-task-run-priority:` to only allow lowering the priority -* `queue:raise-task-run-priority:` to only allow raising the priority +* `queue:change-task-priority-in-queue:` +* `queue:change-task-priority:` +* `queue:lower-task-priority:` to only allow lowering the priority +* `queue:raise-task-priority:` to only allow raising the priority + +And for task group: + +* `queue:change-task-group-priority:/` (for consistency with existing task group scopes) +* `queue:lower-task-group-priority:` to only allow lowering the priority +* `queue:raise-task-group-priority:` to only allow raising the priority + +### Priority calculation Current order for picking up tasks is based on the task priority and insertion time (FIFO). This RFC proposes to change the priority only, and leave the insertion time as is. ### Other considerations -Chain of Trust validation might be broken if we change too much of the task definition or run. -We need to make sure we don't break it with this change. +To insure that Chain of Trust validation is not affected, we aim to keep the original task definition +and store the new priority in a separate column. ## Implementation diff --git a/rfcs/README.md b/rfcs/README.md index 50d283f..2b6b41c 100644 --- a/rfcs/README.md +++ b/rfcs/README.md @@ -56,4 +56,4 @@ | RFC#180 | [Github cancel previous tasks](0180-Github-cancel-previous-tasks.md) | | RFC#182 | [Allow remote references to .taskcluster.yml files processed by Taskcluster-GitHub](0182-taskcluster-yml-remote-references.md) | | RFC#189 | [Batch APIs for task definition, status and index path](0189-batch-task-apis.md) | -| RFC#190 | [Queue change task run priority](0190-queue-change-task-run-priority.md) | +| RFC#190 | [Queue change task / task group priority](0190-queue-change-task-run-priority.md) | From 4da263ad7a4657d4463af1378b5b12e3478b34bb Mon Sep 17 00:00:00 2001 From: Yarik Date: Tue, 4 Jun 2024 11:09:08 +0200 Subject: [PATCH 4/6] Update rfcs/0190-queue-change-task-run-priority.md Co-authored-by: Ben Hearsum (he/him) --- rfcs/0190-queue-change-task-run-priority.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index 29a9982..0a063db 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -66,7 +66,7 @@ This RFC proposes to change the priority only, and leave the insertion time as i ### Other considerations -To insure that Chain of Trust validation is not affected, we aim to keep the original task definition +To ensure that Chain of Trust validation is not affected, we aim to keep the original task definition and store the new priority in a separate column. ## Implementation From 34535c44a86a3091de12be46f4ab710bebde034b Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Tue, 4 Jun 2024 11:30:50 +0200 Subject: [PATCH 5/6] Include info about cot ignore_keys, update phrasing --- rfcs/0190-queue-change-task-run-priority.md | 23 +++++++++------------ 1 file changed, 10 insertions(+), 13 deletions(-) diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index 0a063db..b82d3d2 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -7,7 +7,7 @@ Since we refactored queue internals it became possible to see what tasks are pending for a particular worker pool / task queue. -This RFC proposes new API method that would allow to change the priority of the existing task run / task group. +This RFC proposes a new API method that would allow changing the priority of the existing task run / task group. ## Motivation @@ -30,19 +30,21 @@ Queue service will expose new methods: * `queue.changeTaskPriority(taskId, newPriority)` * `queue.changeTaskGroupPriority(taskGroupId, newPriority)` -New priority would be stored along the task definition for a given task or all tasks within the task group. +New priority would replace the original one for a given task or all tasks within the task group. The process to change single task will be as follows: -* new task priority will be stored in the database as `task.priority_override` column -* if task was scheduled already, its priority in the `queue_pending_tasks` table will be updated -* if the tasks run fails and new one is created, it will check if `task.priority_override` is set and use it instead of the original priority +* new task priority will be stored in the database as `task.priority` (overriding existing value) +* additionally, if task was scheduled already, its priority will be updated in the `queue_pending_tasks` table The process to change the whole task group will be as follows: -* all tasks in the task group will be updated to store the new priority in the `task.priority_override` column -* all currently scheduled tasks would be attempted to be updated in the `queue_pending_tasks` table -* all unscheduled tasks and tasks that will be restarted will use the new priority +* all tasks in the task group will be updated to store the new priority in the `task.priority` column +* additionally, all currently scheduled tasks would be attempted to be updated in the `queue_pending_tasks` table + +Tasks would only be updated if they are not resolved yet and before their deadlines. + +> **Note**: To allow changing the priority of the task without breaking CoT validations, it would be necessary to add `"priority"` to the [`ignore_keys`](https://github.com/mozilla-releng/scriptworker/blob/454c4dd0bae7958140ea8d19adf3670e705ace09/src/scriptworker/cot/verify.py#L910) ### New scopes @@ -64,11 +66,6 @@ And for task group: Current order for picking up tasks is based on the task priority and insertion time (FIFO). This RFC proposes to change the priority only, and leave the insertion time as is. -### Other considerations - -To ensure that Chain of Trust validation is not affected, we aim to keep the original task definition -and store the new priority in a separate column. - ## Implementation _pending_ \ No newline at end of file From 276a15216e8ebe3b7976f6770ad10813a32c55fb Mon Sep 17 00:00:00 2001 From: Yaraslau Kurmyza Date: Tue, 4 Jun 2024 11:46:00 +0200 Subject: [PATCH 6/6] Include pulse events --- rfcs/0190-queue-change-task-run-priority.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/rfcs/0190-queue-change-task-run-priority.md b/rfcs/0190-queue-change-task-run-priority.md index b82d3d2..b071011 100644 --- a/rfcs/0190-queue-change-task-run-priority.md +++ b/rfcs/0190-queue-change-task-run-priority.md @@ -46,6 +46,17 @@ Tasks would only be updated if they are not resolved yet and before their deadli > **Note**: To allow changing the priority of the task without breaking CoT validations, it would be necessary to add `"priority"` to the [`ignore_keys`](https://github.com/mozilla-releng/scriptworker/blob/454c4dd0bae7958140ea8d19adf3670e705ace09/src/scriptworker/cot/verify.py#L910) +### Events + +Since Taskcluster doesn't have any built-in auditing tools, it would be beneficial to have events for the priority change. + +Following events would be emitted: + +* `task-priority-changed` with `taskId`, `existingPriority` and `newPriority` +* `task-group-priority-changed` with `taskGroupId`, `newPriority` + +When the task group priority is changed, the event will be emitted for each task in the group additionally. + ### New scopes This will also require introduction of the new scopes (anyOf):