Spike - Investigate celery worker terminating prematurely

[ADPennington]

Description:
We have multiple open tickets that make reference the celery worker terminating prematurely during data processing, including:

• Discussion & Spike - As an STT user, I need the ability to access error reports, even when my report has > 200K errors #3026
• [bug] OFA unable to export data to csv by record type and fiscal period #3137
• The screenshot below was captured during qasp-related testing for Parse Exception Handling #3080

This is a dedicated spike ticket to track investigations into the root cause of this issue, which is currently impacting generation of error reports for STTs and OFA's ability to export data from the postgres db.

Open Questions:
Please include any questions, possible solutions or decisions that should be explored during work

• Root cause(s)?
• Does planned work to refactor TDP validation checks by priority help reduce the incidence of this?

Deliverable(s):
Create a list of recommendations or proofs of concept to be achieved to complete this issue

• root cause(s) documented
• possible solutions documented
• Outcome 3

Supporting Documentation:

Note: DAC time is in UTC. cloud.gov logs time in EST.

•