Add param for max_download_size for request

[codejunction]

Can we add new parameter to the get and post request to limit the max_download_size so as to limit the request timeout.

This can be achieved through stream but would be nice to have as a default options for the Client and AsyncClient in general.

It might return a status code of 413 and an empty content and text field.

My scenario

i have a bunch of url from a crawler. But few of them are file links (500mb+). Example : www.dl.example.com/35/23/patch its an executable linux binary.

i have check for file types but since this link doesn’t have application header and content length param, my container runs out of memory on this request.

[tomchristie]

Ok, so...

We've got a bit of discussion on this here.

If we did want to add this to, then I'd expect that we'd want to add it here...

limits = httpx.Limits(max_download_size=...)  # currently accepts `max_connections` and `max_keepalive_connections`.
client = httpx.Client(limits=limits)
...

I also think we'd probably want:

• To raise an exception when the download limit is exceeded.
• For the download limit to apply to non-streaming responses only.
• Have some reasonable default.

Example : www.dl.example.com/35/23/patch its an executable linux binary.

Did you mean to provide a different URL there?

[codejunction]

I completely agree to your analysis and i think it should be good to have it in limits. Also the url was just an example for the explanation.

[codejunction]

My current alt solution is as follows:

from httpx import Client


def httx_get(self, link:str) -> int:
    _client = Client(timeout=120)
    truncate_after = 1 * 1024 * 1024  # 1MB
    with _client.stream("GET", link) as response:
        body = b""
        for chunk in response.iter_bytes():
            body += chunk
            if response.num_bytes_downloaded >= truncate_after:
                self._logger.warning(f"Size Exceeded for {link.url}")
                return 413
        return response.status_code

it would me good to have a limit for download size as well