Scratch space on connection class

[bluetech]

I'd like to add a scratch attribute to connection which is by default None with type Any but which may be set by 3rd parties for private use.

This must sound like an XY problem so let me explain why I'm asking for this.

I'm using Postgres with Django and have a need for a proper (process-level) connection pool rather than Django's persistent connections. I know that psycopg3 has a nice new connection pool, better than the psycopg2's ones. However, Django doesn't support psycopg3 yet, so I decided to port psycopg3's ConnectionPool to psycopg2 (minus the asyncio stuff). The port is very straightforward, the only real problem I ran into is that ConnectionPool sets a couple of attributes on the connection instance (_pool and _expire_at), but psycopg2's connection class is slotted and doesn't allow extra attributes.

The workaround I'm using now is to use a connection subclass which has a __dict__ so allows extra attributes, but this wrecks the types somehow since connection_class is given dynamically and mypy doesn't much like subclassing this. Another possible workaround is to use a weakref dict. But I thought I'd ask if psycopg2 could add some "scratch space" on the connection so I can use that directly instead.

BTW, from playing with the connection pool so far, it works well!

[dvarrazzo]

Hey hello,

Thank you for playing with this, I figured out the redesigned connection class would have been useful for psycopg2.

...but no, I prefer to not add "scratch attributes". The solution of subclassing works adequately well. Mypy not liking it is not really a big issue, there are so many things that mypy doesn't like (yet) and the right way to work around them is with a # type: ignore. You are working around a mypy limitation, subclassing seems the right thing to do in a models object, whereas an untyped scratch argument is not really type safe.

You can also drop the dynamic connection_class argument from your pool reimplementation, or even better using your connection subclass as the lower bound of your argument.

[bluetech]

I've uploaded the backport to a repo here: https://github.com/bluetech/psycopg-pool-backport

@dvarrazzo Would you be OK with me publishing it to PyPI under the name psycopg-pool-backport? If not, I can come up with a less official-sounding name.

[dvarrazzo]

@bluetech that seems very useful! I will take a look, thank you

[dvarrazzo]

By the way, how did you work around the non-writable connection attributes? I was thinking that a solution might be to have a map connection_id -> data in the pool itself.

[bluetech]

I used a subclass. This also ended up having a few other uses:

1. Added a (useless) generic Row - useful for not having to rewrite away the generic from the code.
2. Added execute like in psycopg3 - useful for not having to rewrite all of the tests to use cursor.execute instead.
3. Added __dict__ - for test which mocks connection.rollback, not possible with slots.