State store commit request type with transaction object held directly or in S3

[patchwork01]

Background

Split from:

• Unordered batch commit queue for compaction jobs #3686

Description

We'd like to introduce a version of StateStoreCommitRequest that holds the transaction object either directly in the message, or with a pointer to S3.

This should let us simplify the flow of the state store committer lambda, and potentially avoid the need for StateStoreCommitter as a separate class. To apply this commit request we would pass it directly to the state store after deserialisation.

Analysis

The commit request could use the same model as the state store method added in the following issue:

• State store method to add a transaction from S3 #3899

We can use that directly in StateStoreCommitter for now, and anticipate removing most of the logic in that class such that it can be removed in the future.

We could include serialisation/deserialisation code that will be necessary for the following issue, or split it out:

• Logic to batch up compaction commits #3904