Uploading file in chunks using range header

[triplef]

Basic Info

• Faraday Version: 1.9.3

Issue description

I’m trying to use Faraday to upload a file to Azure Blob Storage, which has a limit (of 268435456 bytes) for the body of each request. Larger requests must be sent in chunks using a Range or x-ms-range header. Is there a way to do that with Faraday using multipart / FilePart or otherwise?

I looked at the implementation of FilePart, which seems to support passing an IO object instead of a file path, but I’m not sure if there's a way to limit such an object to just a subsection of a file?

Here is the code I’m currently using, which works fine for files below that limit:

Faraday.new(headers: {'x-ms-blob-type': 'BlockBlob'}) {|c|
  c.request :multipart
  c.response :raise_error
  c.adapter Faraday.default_adapter
}.put(fileUploadUrl, {file: Faraday::FilePart.new(filePath, 'application/octet-stream')})

Any help would be appreciated!

[iMacTia]

Hi @triplef, I'm afraid this might not be in the multipart middleware capabilities at the moment.
Just to understand, is the server expecting multiple individual requests (each on a separate connection), or is this supposed to be a streaming request (single connection reused for many requests)?

Also, I'll move this ticket under the new faraday-multipart repository since that's where the middleware lives now, you might get a few notifications from GitHub 😄

[triplef]

Thanks for your feedback! I’m not sure there’s a requirement regarding reusing the connection, I’m guessing it should work either way but I’d probably try to reuse first.

Do you happen to know of any other project that might be able to do this?

[iMacTia]

I mostly work with AWS and use their sdk when I need something plug-and-play. So I made a quick search and found that Azure offers something similar in Ruby: https://github.com/azure/azure-storage-ruby

It would make sense to use that instead of re-implementing it using Faraday. What do you think?

[triplef]

Yeah that makes a lot of sense and I looked into this, unfortunately that SDK doesn’t seem to have the necessary initializers to work with an externally provided URL for our use case (see Azure/azure-storage-ruby#211).

I’ll see if I can get more info there or find some other way. Thanks again!

[iMacTia]

I may be wrong, but I suspect they're simply lacking documentation around it. Looking at the BlobService code, I found a reference to a storage_sas_token. It seems like the initializer accepts many more options than what is shown in the README.md. Is that still not enough to achieve what you need? (Sorry, I'm just trying to help but I know close to zero about Azure 😅)

[iMacTia]

I forgot the link to the reference obviously 🤦 : https://github.com/Azure/azure-storage-ruby/blob/master/blob/lib/azure/storage/blob/blob_service.rb#L56

[triplef]

I still don’t see a way to give it a full URI (the URI seems to be always built from individual components, see blob_uri), but I found another solution: Setting a header like x-ms-version: 2021-02-12 increases the maximum blob size to 5 GiB, which is more than enough for our purposes.

So I ended up with the following:

filePart = Faraday::FilePart.new(filePath, 'application/octet-stream')
Faraday.new(headers: {
  'x-ms-blob-type': 'BlockBlob',
  'x-ms-version': '2021-02-12'
}) {|c|
  c.response :raise_error
  c.adapter Faraday.default_adapter
}.put(fileUploadUrl, filePart, {'Content-Length': filePart.size.to_s})

Thanks again for your help!

[iMacTia]

Thanks for coming back and sharing your solution! I'm sure that will help others in future ❤️