Callback on multipart upload?

[Secrus]

Hi!
First of all, thanks for the great work on this package.

Now to the question.
In the documentation, you give an example of tracking the download of a file with tqdm. Is it possible to reverse this and track the upload? The reason I want this is I want to migrate from requests and requests-toolbelt which together offer this:
(code from requests-toolbelt docs)

import requests
from requests_toolbelt.multipart import encoder

def my_callback(monitor):
    # Your callback function
    pass

e = encoder.MultipartEncoder(
    fields={'field0': 'value', 'field1': 'value',
            'field2': ('filename', open('file.py', 'rb'), 'text/plain')}
    )
m = encoder.MultipartEncoderMonitor(e, my_callback)

r = requests.post('http://httpbin.org/post', data=m,
                  headers={'Content-Type': m.content_type})

If this is not currently possible, I would be happy to work on adding this, however, I would love some guidelines on where the code should be added since I have never worked on "lower level" networking code.
Thanks in advance!

[Secrus]

@tomchristie sorry to call by name, but is there any chance you could provide some pointers here?

[tomchristie]

Sure, so...

What you'd need here is a way of monitoring when request.stream is iterated over.
One option might a wrapper over the stream interface.

I'd probably start out just by trying this kind of thing...

class MonitorUploadStream:
    def __init__(stream, content_length):
        self.stream = stream
        self.content_length = content_length

    def __iter__(self):
        complete = 0
        for chunk in self.stream:
            complete += len(chunk)
            print(complete, self.content_length)
            yield chunk

    def close(self):
        self.stream.close()


request = client.build_request(...)
request.stream = MonitorUploadStream(request.stream, request.headers.get('Content-Length'))
response = client.sent(request)

That'd at least give you a bit of a handle onto seeing where you need to be hooking in, although it doesn't provide the interface you're looking for.

If you're aiming to replace the requests toolbelt multipart encoder, then you'd want to replicate the same approach they take.

I'd suggest you're aiming for this kind of interface...

multipart = MonitorMultipartStream(data=data, files=files, boundary=boundary)
headers = multipart.get_headers()

httpx.post(stream=multipart, headers=multipart.get_headers()

You'd be able to clone the multipart implementation from https://github.com/encode/httpx/blob/master/httpx/_multipart.py

The __iter__ call is where you'd want to hook in you additional monitoring/callback logic.

You'll notice that the multipart implementation in httpx supports both sync and async interfaces. You might not need that for your use-case.

Hopefully there's enough there to get you started.

Let us know how you get on...

[Secrus]

Thanks! I will try to hack something with this. If I were to make something concrete out of it, would you accept it as PR to httpx itself? I think this could be a nice addition, either as _utils or _multipart addition.

[tomchristie]

If I were to make something concrete out of it, would you accept it as PR to httpx itself?

I'd suggest that we don't pre-determine any outcomes there.
A discussion and seeing where you get too would be our first step.

[yannsartori]

@Secrus I was also curious about this ability. Were you able to get a working solution?

[Secrus]

@yannsartori didn't really have time to get into it.