add clean_file_from_url #165
Conversation
| .await?; | ||
|
|
||
| loop { | ||
| let bytes = reader.read(&mut read_buf).await?; |
There was a problem hiding this comment.
But I'm not sure how does the RetryMiddleware works with stream reading. If an error occurs within this read() does the RetryMiddleware retries this read or the entire GET? If it retries the entire GET what does the next successful read return? If it returns bytes from the beginning of the file then that's problematic.
There was a problem hiding this comment.
I believe at this point it will not retry, though I also think we should not retry plainly at that moment since if we could have submitted some content to the cleaner already. The behavior for an error from the stream (e.g. in the highlighted line) should be the same behavior as if we had a file read return an error, which I think we just bubble it up in this case.
There was a problem hiding this comment.
Let's test to confirm the behavior of RetryMiddleware interacting with error reading from a stream (the error may be internally handled by retry and not propagated to this level).
Adds a utility function to clean a file given a url to use to "GET" the file, i.e. an S3 presigned url.
The contents are then read incrementally and passed into the cleaner handle to chunk, dedup and upload.
Intended potential use for streaming files to migrate them.