Handle S3 incomplete uploads

[becky-gilbert]

Summary

Now that we're uploading recordings directly to S3 as multi-part uploads, we need to decide how to handle incomplete uploads. Incomplete uploads occur when the upload is started but not completed (or aborted). These video file parts will sit in storage indefinitely unless we handle them somehow. Currently they are not visible to the lookit-api or researchers at all unless they are completed in S3.

We already have a considerable number of incomplete file uploads. I don't have any information yet as to how many parts are associated with each file or how much storage space these incomplete uploads are taking up.

Possible solutions

• Set up a lookit-api task to get the list of all incomplete multi-part uploads, get the ones that are older than some date/time, and handle them. Should we try to complete (some of) them, or always abort/delete them? Moved to #1438
• If we decide to recover some incomplete videos: add a column to the video table so that we can flag videos as incomplete/recovered/possibly corrupt.
• Add logging about upload problems to the experiment data, so that researchers have some indication of why a video isn't listed in a given response (EFP task): Add error catching/logging for recording start and stop/upload problems #402
• Improve our handling of in-progress recordings by trying to complete the recording upload when the participant ends the experiment early (EFP task). Moved to Leaving an experiment early: confirmation message and in-progress recordings  #397.
• Improve Sentry logging for uploading problems (EFP task): Add error catching/logging for recording start and stop/upload problems #402
• Add AWS rule for deleting partial/incomplete uploads after 30 days. Moved to#1438

[becky-gilbert]

Transferring this issue from lookit-api to EFP, since we've decided against the lookit-api solutions to this problem.

[becky-gilbert]

Closing as all are either completed or moved to new issues.