New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize data transfers within pipeline #47

Open

sallyom opened this issue Oct 2, 2024 · 1 comment

Collaborator

sallyom commented Oct 2, 2024

S3 should only be utilized to store results - functional data back & forth should be done via PVCs as much as possible.

Collaborator

leseb commented Oct 2, 2024

If this is just to pick the best score, we could also read the job logs and read the scores from here, even less back and forth.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment