Reparse Command Fails when Queryset is Large

[elipe17]

Thank you for taking the time to let us know about the issue you found. The basic rule for bug reporting is that
something isn't working the way one would expect it to work. Please provide us with the information requested
below and we will look at it as soon as we are able.

Description

• While testing the Reparse Django command the selected options generated a queryset too large to fit into memory thus causing the OS to kill the process and a subsequently cause the Reparse to fail.

Command and output below:

shell=
vcap@9eda436c-8a8f-4409-442c-5963:~$ python manage.py clean_and_reparse -y 2023 -q Q1

You have selected to reparse datafiles for FY 2023 and Q1. The reparsed files will NOT be stored in new indices and the old indices
These options will delete and reparse (87) datafiles.
Continue [y/n]? y
Previous reparse has exceeded the timeout. Allowing execution of the command.
2024-08-26 20:12:28,462 INFO clean_and_reparse.py::__backup:L47 :  Beginning reparse DB Backup.
Beginning reparse DB Backup.
2024-08-26 20:12:28,466 INFO db_backup.py::get_system_values:L51 :  Using postgres client at: /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/
Using postgres client at: /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/
2024-08-26 20:12:28,469 INFO db_backup.py::backup_database:L86 :  Executing backup command: /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/pg_dump -Fc --no-acl -f /tmp/reparsing_backup_FY_2023_Q1_rpv3.pg -d postgres://u9oc318z26941vlu:p2wtjxap7i30tjpg2gef0hfwv@cg-aws-broker-prodmezsouuuxrb933l.ci7nkegdizyy.us-gov-west-1.rds.amazonaws.com:5432/tdp_db_qasp
Executing backup command: /home/vcap/deps/0/apt/usr/lib/postgresql/15/bin/pg_dump -Fc --no-acl -f /tmp/reparsing_backup_FY_2023_Q1_rpv3.pg -d postgres://u9oc318z26941vlu:p2wtjxap7i30tjpg2gef0hfwv@cg-aws-broker-prodmezsouuuxrb933l.ci7nkegdizyy.us-gov-west-1.rds.amazonaws.com:5432/tdp_db_qasp
2024-08-26 20:13:11,678 INFO db_backup.py::backup_database:L91 :  Successfully executed backup. Wrote pg dumpfile to /tmp/reparsing_backup_FY_2023_Q1_rpv3.pg
Successfully executed backup. Wrote pg dumpfile to /tmp/reparsing_backup_FY_2023_Q1_rpv3.pg
2024-08-26 20:13:11,687 INFO db_backup.py::backup_database:L101 :  Pg dumpfile size in bytes: 286097754.
Pg dumpfile size in bytes: 286097754.
2024-08-26 20:13:11,687 INFO db_backup.py::upload_file:L173 :  Uploading /tmp/reparsing_backup_FY_2023_Q1_rpv3.pg to S3.
Uploading /tmp/reparsing_backup_FY_2023_Q1_rpv3.pg to S3.
2024-08-26 20:13:14,146 INFO db_backup.py::upload_file:L186 :  Successfully uploaded /tmp/reparsing_backup_FY_2023_Q1_rpv3.pg to s3://cg-178858c2-2794-44ac-a18a-c8f6efe4197a/backup/tmp/reparsing_backup_FY_2023_Q1_rpv3.pg.
Successfully uploaded /tmp/reparsing_backup_FY_2023_Q1_rpv3.pg to s3://cg-178858c2-2794-44ac-a18a-c8f6efe4197a/backup/tmp/reparsing_backup_FY_2023_Q1_rpv3.pg.
2024-08-26 20:13:14,148 INFO db_backup.py::main:L326 :  Deleting /tmp/reparsing_backup_FY_2023_Q1_rpv3.pg from local storage.
Deleting /tmp/reparsing_backup_FY_2023_Q1_rpv3.pg from local storage.
2024-08-26 20:13:14,175 INFO backup_db.py::handle:L36 :  Cloud backup/restore job complete.
Cloud backup/restore job complete.
2024-08-26 20:13:14,175 INFO clean_and_reparse.py::__backup:L49 :  Backup complete! Commencing clean and reparse.
Backup complete! Commencing clean and reparse.
Killed