Check for double counting of transactions in state store committer throughput system test

[patchwork01]

Background

Follows on from:

• State store committer throughput is occasionally above expected range in system test #3528

We've had a number of runs of the state store committer throughput system test where it found throughput higher than should be possible with AWS Lambda with SQS FIFO. We expect that it should only process at most 10 invocations of 10 SQS messages each in a second, per message group.

We expect at most 100 invocations per second for each Sleeper table, but we've seen up to 150 per second or more for a single table, and up to 400 per second for one of 10 tables.

Description

We'd like to check whether there's anything about how we measure throughput that could be producing an incorrectly high throughput.

Analysis

The entry point for a system test checking throughput of the committer is in SystemTestStateStore, in commitsPerSecondForTable or commitsPerSecondByTable. This uses AwsStateStoreCommitterLogsDriver to query the CloudWatch logs of the committer, and computes throughput in StateStoreCommitterLogEntries.computeOverallCommitsPerSecondByTableId.