Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve submission speed for many recipes #103

Open
jbusecke opened this issue Sep 7, 2023 · 3 comments
Open

Improve submission speed for many recipes #103

jbusecke opened this issue Sep 7, 2023 · 3 comments

Comments

@jbusecke
Copy link
Contributor

jbusecke commented Sep 7, 2023

Over at https://github.com/leap-stc/cmip6-leap-feedstock I am picking up the pace and am currently running through several hundreds of datasets a day.

I find that the main limitation for scaling this at the moment is -runner submitting jobs in serial.

I am running submission from here and see a submitted job about every ~30 seconds:
image

Is there any way to increase the submission speed here? Could we submit batches of jobs at the same time?

@yuvipanda
Copy link
Collaborator

We will probably have to implement a threadpool here to make concurrent submission work:

for name, recipe in recipes.items():

@jbusecke
Copy link
Contributor Author

jbusecke commented Sep 8, 2023

And then maybe pass an option to limit concurrency? That would be ideal to not fail out due to quota issues.

@yuvipanda
Copy link
Collaborator

@jbusecke correct! I think concurrency should just be 1 by default, as that's the most common use case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants