object_store: Retry on connection duration timeouts (retry / recover after partially reading a streaming response)

[flokli]

**Is your feature request related to a problem or challenge?
I'm using object_store to stream large(r) files to in this case, the body of a HTTP .

I essentially do a store.get(path).await?.into_stream() to get the data stream.

When using it with the Azure backend, I noticed that Azure reliably closes the connection after 30 seconds. Other providers (S3) also explicitly inject errors, but keep most of the error handling in their SDKs.

I know there's some retry logic for some error cases in object_store, but the "connection closed while getting data as a stream" part doesn't seem to be covered. I think there should be a high-level function that retries receiving the remaining data in these error cases (verifying the etag is still the same).

Describe alternatives you've considered
Manually dealing with error handling and retries in all object_store consumers

[cbrewster]

The golang GCS client supports automatic retrying here via keeping track of how many bytes the client has already seen and creating a new ranged read request to resume the read where the client left off:

https://github.com/googleapis/google-cloud-go/blob/9afb797d75499807b29c372ec375668be4d2995e/storage/http_client.go#L1268-L1275

[flokli]

So apparently object_store::buffered::BufReader might be the answer to this, at least it prevents long-running connections to the backend.

I feel like the docstring of into_stream() should be extended to point it out that you might want to use object_store::buffered::BufReader instead.

I'm not that happy yet object_store::buffered::BufReader doesn't do any "readahead", meaning we'd have to wait the summed up round-trip times whenever the buffer gets empty, but that's probably also tricky to do in the general case.

I'm also not sure if the calls to get_range it does are actually retried, in case of temporary connectivity issues / rate-limiting.

[itsjunetime]

take

[erratic-pattern]

take

[erratic-pattern]

I've made a PR apache/arrow-rs#6519 similar to the closed PR apache/arrow-rs#5383, but only permits reqwest::Error::Decode to be retried, since this is the error type that we see associated with this issue (see #272)

EDIT: I am not sure this is exactly the same issue that @flokli is seeing, but it is the one being seen in the #272 issue so maybe I should associate this PR with that issue instead?