State store method to add a transaction from S3

[rtjd6554]

Background

Split from:

• Unordered batch commit queue for compaction jobs #3686

Related to:

• Add transaction type to DynamoDB item for a state store transaction held in S3 #3679

Description

We'll need to add some method on the state store to be able to add a transaction that's held in S3.

Analysis

This method could take the metadata that we expect to be in the SQS message and the transaction log, and also a pointer to the transaction body in S3. The state store can then read the body from S3 and apply it to the log.

An alternative would be to take the body of the transaction and a pointer to where it's held in S3. We can then validate the transaction and add it to the log without including the body of the transaction in the log.

If we pass the transaction object and a pointer to it in S3, this would rely on the deserialisation being done elsewhere, but would allow any other operations based on the details of the transaction to be done before calling the state store.

This may be necessary if we want to keep the job tracker update in the state store commit lambda. We probably do want that for now, since a solution for the tracker updates is a bit larger, and it's more flexible if we can leave our options open.

We could add a method signature something like this:

stateStore.addFileTransactionFromDataBucket(String key, FileReferenceTransaction transaction);

This will also mean adjusting the call to TransactionLogStore, because that's currently where the upload to S3 happens, and we need to avoid it being repeated. We've split out a separate issue for that that will need to be done first:

• Split transaction upload to S3 out of transaction log store #4041