Improve throughput for immediate tasks (1/3) #5841
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pedro-psb
force-pushed
the
feat/run-immediate-tasks-on-parent
branch
from
7fb9d32
to
d18b071
Compare
pedro-psb
changed the title
gerrod3
reviewed
Sep 27, 2024
pulpcore/tasking/worker.py
Outdated
Comment on lines
349
to
352
| .first() | ||
| ) | ||
| with contextlib.suppress(AdvisoryLockError), task: | ||
| task.refresh_from_db() |
There was a problem hiding this comment.
.first() can return None. Will this not error trying to call a method on None?
There was a problem hiding this comment.
Yes, nice catch.
But anyway, this was a sketch of the approach I had in mind, which won't be used.
I force pushed with the ideas I discussed with @mdellweg yesterday.
* Refatored handle_available_tasks for more clarity of the flow * Refactored iter_tasks for more flow clarity and to prepare prioritizaton of immediate tasks. * Added/changed misc comments, docstrings and names
pedro-psb
force-pushed
the
feat/run-immediate-tasks-on-parent
branch
from
d18b071
to
9ab7302
Compare
This defines tasks of the type "immediate" by adding a Task.immediate(bool) field to the Task model. Non-immediate are called "long". Then, these type of task is put in front of the line when the worker queries for new available tasks. Closes pulp#5767
pedro-psb
force-pushed
the
feat/run-immediate-tasks-on-parent
branch
from
9ab7302
to
589053b
Compare
|
(there is a refator and a impl commit) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In busy tasking environments, "immediate" running tasks (which are only dispatched to workers due to resource locking requirements) can wait for very long behind long-running tasks, such as syncs and publishes.
This tries to solve this implementing the solution proposed in #5767, as discussed in the Pulp team.