Getting '[Errno 9] Bad file descriptor'

[igorghi]

Hello!

I am using the httpx client through context manager inside a class for some internal service in Django, and after a large number of requests I am getting '[Errno 9] Bad file descriptor'. I suspect that for some reason the connections after being closed are not being cleaned from system for some reason, but this is a wild guess from someone not really experienced in this part.

class ServiceClient:


    def __init__(self, *args, **kwargs):
        self.__session = httpx.Client(base_url=BASE_URL, headers={
            'Authorization': API_KEY,
            'Content-Type': 'application/json'
        })
        
     def create_batch(self, nome:str):

        with self.__session as c:
            req = c.post('batch', json={
                'name': nome
            })
            req.raise_for_status()

        json = req.json()
   
        return json

Any help or ideas are appreciated. I am not sure if this is the correct way to use inside a class with the context manager.

[tomchristie]

Possibly depends on how/where you're instantiating ServiceClient?

You only want to use a context manager if you're being precise about the expected lifetime of the client connection pool. In the sync case most of the time you actually just want a single global client instance shared across the entire codebase. The connection pool will be closed properly on shutdown.

If you're in a context where there is a clean startup/shutdown sequence that you can hook into, then it's a little neater to be precise about the client lifespan, and pass the instance around wherever it's needed.

We could probably do with some docs on recommended client usage centred around the client lifespan.

[igorghi]

So if I understood it correctly, I can instantiate a single global client and use the context manager to manage opening/closing the connections of this client instance? Theoretically I should inject this client inside the service then.

Right now I am creating a new instance of ServiceClient every request to the Django view, consequently, the init method creates a new httpx client every request.

[tomchristie]

I can instantiate a single global client

Correct, yes.

and use the context manager to manage opening/closing the connections of this client instance?

I'm not quite sure what you mean here. You can use a client in a context manager if you want to be strict about making sure that the connection pool is closed off at the end of the lifetime. However in a WSGI web app such as Django or Flask you can't actually do that, because there isn't any API around "perform these actions when the web app has starts, perform these actions when the web app stops". Actually that's okay tho, we don't need to. We can just get away with being a little fuzzy, and let the connection pool be closed on program termination, when __del__ is called. This is the same as the behaviour you'd get with requests.Session().

The practical upshot of this is that:

• When using httpx.Client() the simplest thing to do will tend to be to have a single global instance, and just allow it to be automatically closed on program termination.
• When using httpx.AsyncClient() you want to be more precise about resource lifetimes, and you'll want an instance that is explicitly opened/closed with a context. You can either pass this around explicitly, or perhaps use a ContextVar to allow access to it throughout the program. (In contrast to WSGI, ASGI does have startup/shutdown events that we can hook into to support this.)

We really ought to get this well documented. 😬😀

[igorghi]

I'm not quite sure what you mean here ... However in a WSGI web app such as Django or Flask you can't actually do that, because there isn't any API around "perform these actions when the web app has starts, perform these actions when the web app stops".

When the code exits the context manager, is the connection pool not closed?

Alternatively, you can explicitly close the connection pool without block-usage using .close():

client = httpx.Client()
try:
    ...
finally:
    client.close()

My question is if after the close() being called after exiting the context, I can reuse the client instatianted on the ServiceClient.__init__ or I need to instantiate a new client.

I guess my Django knowledge is a little fuzzy and risking sounding dumb, instantianting globally the client, the client does persist between requests (I am guessing that there is a thread that remains alive and new threads are spawned each request)?

Thanks for your replies :) I am trying to understanding the correct usage of httpx and the context manager flow of it.

[tomchristie]

I guess my Django knowledge is a little fuzzy and risking sounding dumb, instantianting globally the client, the client does persist between requests (I am guessing that there is a thread that remains alive and new threads are spawned each request)?

It's not a dumb question in the slightest.

instantianting globally the client, the client does persist between requests

Yes, the client will persist between requests.

I am trying to understanding the correct usage of httpx and the context manager flow of it.

The short answer is... unless you've got a good reason to do something different, then just use a single client instance, and don't bother with the context manager.

(You really only need it in the async case.)