fix: Implement s3 cache integrity checking on submission time.

[benl-2023]

What was the problem/requirement? (What/Why)

• When assets in S3 get deleted, it can result in the local S3 cache becoming invalid. That is, assets may fail to be uploaded if the client believes an asset is already uploaded, when in reality it is no longer present in S3.
• This results in failed jobs, as workers will not be able to find these assets in S3.

What was the solution? (How)

• The solution is to implement sampled cached asset inspections to verify that the contents in the scope of a submission that are cached, are also present in S3.
  • When assets are cached but not present in S3, the client resets its cache.
• When the cache is reset, all assets will be uploaded.

What is the impact of this change?

• Reduces the likelihood of jobs failing due to missing assets in S3.

How was this change tested?

The following setup was done:

• The local submission host was cleared of its caches.
• The S3 job attachments bucket was cleared of all objects.
• 2 jobs were submitted with 999 assets uploaded as part of Job No. 1 and a single asset uploaded as part of Job No.2.
  • A total of 1000 assets were uploaded to S3.

Then

• The asset required job Job No.2 was manually deleted from S3.
• Job No.2 was resubmitted. Job submission information provided by the submission GUI confirmed the asset was re-uploaded.
  • Also manually confirmed in S3 that the asset was re-uploaded.
• The Job completed successfully.

Was this change documented?

No. This is a bug fix / improvement.

Is this a breaking change?

No.

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[mwiebe]

When the cache is reset, all assets will be uploaded.

I think you mean that all assets will be re-checked with s3:HeadObject, right? Assets that are on S3 will only get this check, not be re-uploaded.