Handle missing arrangements in S3

[cavis]

Seeing a small number of logged errors, where the counts lambda gets bytes downloaded for a digest. But gets a 404 from S3 trying to get the arrangement json.

Appears to be a race condition where the stitch-lambda just PUT the arrangement json. But the bytes came in too fast for eventual read consistency. Which seems odd to me, since on the first getObject request S3 is supposed to guarantee read consistency. Should probably re-read those docs again.

In any case - come up with some sort of a strategy to address this. Maybe throw records back on kinesis like the analytics DDB lambda does? Or use S3 listObjects requests before getting? Or wait a very short time in the same lambda execution then retry? Or slow the bytes lambda kinesis down?

Crazy ideas!