Enhanced custom name resolution

[Allgot]

This is the follow-up discussion of #2592.

As the result of the last discussion, the new extension, sni_hostname has been added.
However, I have identified an opportunitiy for further improvement in the custom name resolution process: add another extension that allows users to specify the pseudo-header, :authority.

Currently, if the user utilizes the sni_hostname extension to connect to an IP address directly, then the :authority header is automatically set as the specified IP address. This behavior is fine, but users could face difficulties when using some systems (e.g. Cloudflare's DoH resolver) that require an exact :authority header field rather than just the IP address. However, with this newly suggested extension, they would be able to easily access these systems.

Here's an example of how the code would look like after the implementation:

headers = {"Host": "www.google.com"}
extensions = {"sni_hostname": "www.google.com", "authority": ""www.google.com"}
response = httpx.get("https://142.250.200.36:80/", headers=headers, extensions=extensions)

[tomchristie]

Hi,

Thanks for the well-framed discussion. We actually handle this already, although I don't think we have it properly documented. When sending a request over HTTP/2, the Host header is automatically replaced with the :authority pseudo-header.

A good starting point for improving our documentation would be to note this in https://www.encode.io/httpcore/extensions/#sni_hostname

[Allgot]

Great to hear that it already exists!

Thanks a lot :)

[toppk]

also, I'm back. after the new backend change broke my monkey patching, I finally uplifted what I have to the new code. although I haven't looked at the current subsequent sni_hostname bits, to see if that's all that I need (it isn't).

The scope of my effort is not just to control name resolution for odd cases where dns != http host != sni hostname, but also to prevent dns queries under normal circumstances. IMO dns queries shouldn't occur unless you get a timeout/5xx/3xx for not immediately breaking the world when DNS is out (see facebook 2021 outage). So my transport implementation allows you to control to say, do dns after certain http responses, and allows you to cycle through a list of IP's if there was more then just one address in response to a query.

my current work was rushed, and has a lot of issues, but has a "working" async interface this time, as that's what I needed (also has a sync because I need that too ;)

Obviously this is beyond what is needed in httpcore/httpx, so I will be cleaning up my interface, and looking to fix http-core and httpx if there's still a capability gap are warranted to simplify my solution. Of course, if we want to get some dns caching in httpcore (not sure if it's still missing), i'd be willing to clean up my implementation (and make it typed properly in pre-3.11 python).

So I cannot say that i've studied httpcore much beyond what I needed to get my hacks up to date, but I can give my opinion on socket_options, which is that to say that abstractions are leaky, but I think there should be a funnel on them. I was surprised (but not surprised) that socket_options was plumbed into httpx but network_backend was not. I rather replace socket_option with httpcore_options that a niced typed object that has:

class httpcore.HttpCoreLeakyOptions:
   socket_options: typing.Optional[typing.Iterable[SOCKET_OPTION]] = None
   network_backend: NetworkBackend = None

this way httpcore decides what is plumbed, and we don't have to keep going back to httpx to plumb another variable. Said another way, IMNSHO the leaks belong to httpcore.

I'm motivated this time to see this though, as my hacks have made it's way to a codebase it doesn't belong. I had to do something similar to websockets, but that's another story. Again, at this point all I know that I want to do is add network_backend (hopfully via a httpcore_options, as I mentioned) but there may be other things that would still be needed. and I will try to see what the story is with dns caching and if there's appetite for something there too..

And on another note, I used httpbin.org for (the limitted) testing this stuff which is great because it has a strict sni, two IP addresses but relaxed host checking. Unfortunately it's often offline and slow if it's online, and the closest public replacement that I've found (httpbun.org) doesn't come close. I've been slowly porting httpbin into uvicorn in the hopes that this capability can stay alive and even improve. I'm sure httpbin.org has been a victim of being a public service, but perhaps there was other reasons.

(disclaimer: as described above these gists are the crude implementation of the use-case I want httpx to serve and I cannot say they are fully necessary or leverage httpx/http-core properly)

the new gist (monkeypatching the backend, since it's not currently exposed to httpx)
https://gist.github.com/toppk/346e7ec85c956e6608cdadd6e1edae1f

the original gist (monkeypatching connect_tcp)
https://gist.github.com/toppk/a5838c9727d806c7845f7be0e2eacf2f

[toppk]

happy to move this comment to somewhere else. just hoping I get @tomchristie attention :)

[tomchristie]

I was surprised (but not surprised) that socket_options was plumbed into httpx but network_backend was not.

Yep. The network backends only recently became public API... encode/httpcore#699

If you want to open an issue on httpx for us to add a network_backend=... parameter to the HTTPTransport class that would be worthwhile. (I could also potentially imagine us deprecating the socket_options at that layer.)

IMO dns queries shouldn't occur unless you get a timeout/5xx/3xx for not immediately breaking the world when DNS is out (see facebook 2021 outage).

It'd make a lot of sense for httpcore to provide DNS caching over the lifespan of the client. Perhaps you'd like to open a discussion on httpcore?

[karpetrosyan]

To avoid having socket_options in only one version, we can wait for that and revert existing socket_options commit.