Logic to batch up compaction commits

[patchwork01]

Background

Split from:

• Unordered batch commit queue for compaction jobs #3686

Description

We'd like to write the logic to sit in a lambda that will take a batch of compaction commits, organise them by Sleeper table, write them to S3 in one file for each Sleeper table, then send those files to the state store committer queue.

Analysis

The lambda will take a batch of compaction commit SQS messages and pass it to this new class.

Because we split up the batch into Sleeper tables and send each table's batch independently, that means that each Sleeper table can fail independently. We'll need to report which messages did and did not fail, with some way for the lambda to know which SQS messages it refers to (e.g. a message handle like in ingest and compaction tasks).

We can assume the serialisation/deserialisation of the requests and batches will be covered in the following issue:

• State store commit request type with transaction object held directly or in S3 #3901

We will follow up with the Lambda, SQS and S3 code in a separate issue:

• Lambda handler to batch up compaction commits #3905