You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We've had a number of runs of the state store committer throughput system test where it found throughput higher than should be possible with AWS Lambda with SQS FIFO. We expect that it should only process at most 10 invocations of 10 SQS messages each in a second, per message group.
We expect at most 100 invocations per second for each Sleeper table, but we've seen up to 150 per second or more for a single table, and up to 400 per second for one of 10 tables.
Description
We'd like to check whether there's anything about how we measure throughput that could be producing an incorrectly high throughput.
Analysis
The entry point for a system test checking throughput of the committer is in SystemTestStateStore, in commitsPerSecondForTable or commitsPerSecondByTable. This uses AwsStateStoreCommitterLogsDriver to query the CloudWatch logs of the committer, and computes throughput in StateStoreCommitterLogEntries.computeOverallCommitsPerSecondByTableId.
The text was updated successfully, but these errors were encountered:
Background
Follows on from:
We've had a number of runs of the state store committer throughput system test where it found throughput higher than should be possible with AWS Lambda with SQS FIFO. We expect that it should only process at most 10 invocations of 10 SQS messages each in a second, per message group.
We expect at most 100 invocations per second for each Sleeper table, but we've seen up to 150 per second or more for a single table, and up to 400 per second for one of 10 tables.
Description
We'd like to check whether there's anything about how we measure throughput that could be producing an incorrectly high throughput.
Analysis
The entry point for a system test checking throughput of the committer is in
SystemTestStateStore
, incommitsPerSecondForTable
orcommitsPerSecondByTable
. This usesAwsStateStoreCommitterLogsDriver
to query the CloudWatch logs of the committer, and computes throughput inStateStoreCommitterLogEntries.computeOverallCommitsPerSecondByTableId
.The text was updated successfully, but these errors were encountered: