Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update module to configure good job for background workers #10

Merged
merged 5 commits into from
Sep 30, 2023

Conversation

robbevp
Copy link
Member

@robbevp robbevp commented Sep 10, 2023

This PR matches accentor/api#478 and allows the user to configure both good job and delayed job. A next PR can remove the delayed job config.

I opted to have the user pass the names of queues instead of simply the amount of workers to allow more finegrained control.
The default still uses 4 workers (though now with 5 threads each). One of the workers is reserved for the most urgent items. The others will work off all queues in the order they are specified in the string.

@robbevp robbevp added the enhancement New feature or request label Sep 10, 2023
@robbevp robbevp self-assigned this Sep 10, 2023
@robbevp robbevp requested a review from chvp September 10, 2023 12:42
@robbevp
Copy link
Member Author

robbevp commented Sep 10, 2023

All the checks failing is unrelated to this PR

@chvp chvp force-pushed the enhc/add-support-for-good-job branch from b0ddbdb to f57bb06 Compare September 16, 2023 10:43
default.nix Outdated Show resolved Hide resolved
default.nix Outdated Show resolved Hide resolved
@robbevp robbevp requested a review from chvp September 17, 2023 07:14
@robbevp
Copy link
Member Author

robbevp commented Sep 17, 2023

Coming back to this: Our default workers for Good job might be a bit too much for most setups. I went for 4 processes (since this was the default with delayed job), but since these processes are now multi-threaded (and have 5 threads by default), that means that we can process up to 20 jobs at once.
I don't have a good view on the amount of jobs and concurrency of them in production, so can't say for sure.

Copy link
Member

@chvp chvp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would consider that too much even for our setup. In the worst case we're transcoding 20 files at the same time, which is a load I wouldn't want the server to have. I'd start with what we have now, but only one process for all queues instead of three. If it turns out that this results in issues, we can always adjust later.

@robbevp
Copy link
Member Author

robbevp commented Sep 18, 2023

Updated default queues to be 1 process (with the queues in order of priority)

@robbevp robbevp requested a review from chvp September 18, 2023 06:36
@chvp chvp merged commit 9d3d648 into main Sep 30, 2023
3 checks passed
@chvp chvp deleted the enhc/add-support-for-good-job branch September 30, 2023 09:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants