Enable Distributed mode for Step Function maps (v2)

[bh-perennial]

This PR introduces a way to utilize the "Distributed" type of Map (vs "Inline") to enable Step Functions to scale to larger sizes (issue: #1216).

How it works

• Add in a new flag --use-distributed-map when creating a step function (ie python helloworld.py step-functions create --use-distributed-map)
• All map steps are then turned to use Distributed mode (vs the Inline) mode
• Distributed Map values are written to S3 and then we need to run some additional steps (fetch the S3 manifest and run a second Distributed Map step to extract the value)
• The Run ID of the initial step function is passed down to maps so that it can be accessed on child Step Functions so data can be found in the S3 bucket. This is achieved by passing it into the Dynamo table.
• Adds a retry (with jitter) for creating too many batch requests as it only permits 50 req/sec

Infra changes

To support the new mode you need to ensure you have updated the step_functions_role (from https://github.com/outerbounds/terraform-aws-metaflow) to have states:StartExecution and kms:GenerateDataKey

Additional Notes

• This PR replaces EnableDistributed mode for Step Function maps #1576 and addresses the issues from issue Support DistributedMap within AWS Step Functions for higher throughput #1216.

[mvarma-caris]

Thank you for this change! Do you have an estimate of when it'll be merged and available in a new minor version?

[savingoyal]

@bh-perennial thank you for your patience. I think we can clean up the implementation in this PR a bit. For example, we don't need to rely on DynamoDB to pass along run ids. They can be set at the start step and moved along, as in this PR - #1720.

Next up, I am reviewing the need to interface with S3 and whether right now is the opportune time to rip dynamodb away from the setup.

[savingoyal]

#1720 supersedes this