Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make the python loading scripts exit gracefully upon failure #30

Open
2 of 4 tasks
jkleiber opened this issue Apr 22, 2024 · 1 comment
Open
2 of 4 tasks

Make the python loading scripts exit gracefully upon failure #30

jkleiber opened this issue Apr 22, 2024 · 1 comment
Labels
python Tasks that update Python code

Comments

@jkleiber
Copy link
Owner

jkleiber commented Apr 22, 2024

Summary
If the data loader python script fails to an exception, it will simply hang in the console and need to be force killed. This is not good for GitHub actions usage as the script will sit there for 6 hours until GitHub times out the workflow. This is partially due to the supabase client not destructing gracefully, and also due to a lack of exception handling throughout the code.

The objective of this ticket is to address the problem in the short and long terms

Short term solution

  • Make the github action workflow timeout shorter in order to protect against script runaway

Long term solution

  • Add exception handling to joblib parallelization
  • Add exception handling to any third party API calls
  • Figure out how to destruct the supabase API client correctly
@jkleiber jkleiber added this to the Minimum Viable Product milestone Apr 24, 2024
@jkleiber jkleiber added the python Tasks that update Python code label Apr 24, 2024
@jkleiber jkleiber removed this from the Minimum Viable Product milestone May 22, 2024
@jkleiber
Copy link
Owner Author

#39 fixed the issues where supabase would hang due to polling for the JWT refresh token, and shortened the job timeouts to 30 mins. However, we still need to add better exception handling to make the scripts more robust, so leaving this open for now. These robustness changes are not MVP though

@jkleiber jkleiber added this to the Technical Debt Clean-up milestone Jun 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
python Tasks that update Python code
Projects
None yet
Development

No branches or pull requests

1 participant