Improve batch rate limiting logic

[tsmith023]

Currently, the batch rate limit logic does not take into account the number of objects sent in the previous request when calculating whether it should backoff. To improve throughput and maximise loading times, this should be factored into the logic.

[0saurabh0]

will raise a PR for this

[0saurabh0]

@tsmith023 Is there a typical or maximum batch size that we should consider when designing the backoff logic?