`getaddrinfo` fails on Illumos, inconsistent elsewhere if port is, but type & protocol are not specified

[zyv]

Bug report

Bug description:

I have discovered that

1. On Solaris OpenIndiana, if you don't specify at least socket type or protocol for the service name, address resolution fails (see below).
2. Linux & MacOS happily ignore this and continue with the resolution anyway.
3. Specifying the socket family doesn't help.

I've filed a PR to fix the problem at the call site of the software that exposed it (miguelgrinberg/Flask-SocketIO#2088), but the maintainer argues that it's a bug in CPython and is unwilling to accept a patch that specifies the protocol.

His position that Python's high-level APIs should provide consistent behavior across platforms is understandable, although one could argue that getaddrinfo("127.0.0.1", 5000) is a borderline invalid use of the API. However, the documentation as it stands suggests that None can be passed, but doesn't explain why you would want to do so, and the paragraph below suggests that keyword arguments act as AND filters and can be safely omitted.

Unfortunately, I don't see any fantastic way to fix this in CPython that would provide a consistent interface across platforms. I think we could do a configure check to detect this getaddrinfo behavior (might also be the case on AIX, Tru64, etc., although it's just a speculation) and then call the C function twice at Python level with (type=socket.SOCK_STREAM and type=socket.SOCK_DGRAM) to merge the results (which is already done for another reason):

cpython/Lib/socket.py

Line 964 in beee91c

def getaddrinfo(host, port, family=0, type=0, proto=0, flags=0):

Or, if you don't mind, we can just always do it where protocol AND socket type are unspecified, but port is specified and is numeric. This is very simple, the overhead (I think) is negligible, and it is consistent across platforms.

I would appreciate an opinion from Python developers and/or Solaris engineers before I put more work into this.

OpenIndiana

$ python3
Python 3.9.19 (main, Mar 26 2024, 20:30:24) [GCC 13.2.0] on sunos5
Type "help", "copyright", "credits" or "license" for more information.

>>> socket.getaddrinfo("127.0.0.1", 5000)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 9] service name not available for the specified socket type

>>> socket.getaddrinfo("127.0.0.1", 5000, family=socket.AF_INET)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 9] service name not available for the specified socket type

>>> socket.getaddrinfo("127.0.0.1", None)
[(<AddressFamily.AF_INET: 2>, 0, 0, '', ('127.0.0.1', 0))]

>>> socket.getaddrinfo("127.0.0.1", 5000, type=socket.SOCK_STREAM)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 2>, 6, '', ('127.0.0.1', 5000))]

>>> socket.getaddrinfo("127.0.0.1", 5000, proto=socket.IPPROTO_TCP)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 2>, 6, '', ('127.0.0.1', 5000))]

macOS

% python3
Python 3.12.5 (main, Aug  6 2024, 19:08:49) [Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> socket.getaddrinfo("127.0.0.1", 5000)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.0.1', 5000)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 5000))]

AIX

-bash-5.1$ python3
Python 3.9.19 (main, Apr  5 2024, 05:08:51) 
[GCC 10.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.

>>> socket.getaddrinfo("127.0.0.1", 5000)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.0.1', 5000))]

>>> socket.getaddrinfo("127.0.0.1", 5000, type=socket.SOCK_STREAM)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 5000))]

Solaris

$ python3
Python 3.9.19 (main, Sep 23 2024, 16:24:38) 
[GCC 13.2.0] on sunos5
Type "help", "copyright", "credits" or "license" for more information.

>>> socket.getaddrinfo("127.0.0.1", 5000)
[(<AddressFamily.AF_INET: 2>, 0, 0, '', ('127.0.0.1', 5000))]
>>> socket.getaddrinfo("127.0.0.1", 5000, family=socket.AF_INET)
[(<AddressFamily.AF_INET: 2>, 0, 0, '', ('127.0.0.1', 5000))]
>>> socket.getaddrinfo("127.0.0.1", None)
[(<AddressFamily.AF_INET: 2>, 0, 0, '', ('127.0.0.1', 0))]
>>> socket.getaddrinfo("127.0.0.1", 5000, type=socket.SOCK_STREAM)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 2>, 6, '', ('127.0.0.1', 5000))]
>>> socket.getaddrinfo("127.0.0.1", 5000, proto=socket.IPPROTO_TCP)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 2>, 6, '', ('127.0.0.1', 5000))]

FreeBSD 14.1

[zaytsev@freebsd ~]$ python3
Python 3.11.9 (main, Oct  5 2024, 11:59:33) [Clang 18.1.5 (https://github.com/llvm/llvm-project.git llvmorg-18.1.5-0-g617a15 on freebsd14
Type "help", "copyright", "credits" or "license" for more information.

>>> socket.getaddrinfo("127.0.0.1", 5000)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.0.1', 5000)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 5000)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_SEQPACKET: 5>, 132, '', ('127.0.0.1', 5000))]

>>> socket.getaddrinfo("127.0.0.1", None)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.0.1', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_SEQPACKET: 5>, 132, '', ('127.0.0.1', 0))]

Linux (Fedora)

zaytsev@fedora:~$ python3
Python 3.12.4 (main, Jun  7 2024, 00:00:00) [GCC 14.1.1 20240607 (Red Hat 14.1.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> socket.getaddrinfo("127.0.0.1", 5000)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 5000)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.0.1', 5000)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('127.0.0.1', 5000))]

Summary

• Linux: STREAM, DGRAM, RAW
• macOS: STREAM, DGRAM
• OpenBSD: STREAM, DGRAM
• FreeBSD with Cheri: STREAM, DGRAM, SEQPACKET
• AIX: DGRAM (but STREAM supported, obviously!)
• Solaris/Illumos: exception

/cc @kulikjak

CPython versions tested on:

3.9

Operating systems tested on:

Other

Linked PRs

• gh-123832: Adjust socket.getaddrinfo docs for better POSIX compliance #126182
• [3.12] gh-123832: Adjust socket.getaddrinfo docs for better POSIX compliance (GH-126182) #126824
• [3.13] gh-123832: Adjust socket.getaddrinfo docs for better POSIX compliance (GH-126182) #126825

[zyv]

PoC

--- socket.py.orig      Tue Mar 26 20:30:27 2024
+++ socket.py   Sun Sep  8 07:37:32 2024
@@ -948,6 +948,11 @@
     narrow the list of addresses returned. Passing zero as a value for each of
     these arguments selects the full range of results.
     """
+    # Provide consistent behavior across UNIX platforms (see issue #123832)
+    if isinstance(port, int) and type == 0 and proto == 0:
+        return getaddrinfo(host, port, family, SOCK_DGRAM, proto, flags) + \
+            getaddrinfo(host, port, family, SOCK_STREAM, proto, flags)
+
     # We override this function since we want to translate the numeric family
     # and socket type values to enum constants.
     addrlist = []

$ python3
Python 3.9.19 (main, Mar 26 2024, 20:30:24)
[GCC 13.2.0] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getaddrinfo("127.0.0.1", 5000)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 1>, 17, '', ('127.0.0.1', 5000)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 2>, 6, '', ('127.0.0.1', 5000))]

[zyv]

I have just checked AIX 7.1 and AIX 7.3, and my speculation that they also exhibit inconsistent behavior is confirmed. In the case of AIX, if the socket type is not specified, it doesn't return all supported socket types, but only SOCK_DGRAM instead of the more commonly used SOCK_STREAM, even though it supports it:

-bash-5.1$ python3
Python 3.9.19 (main, Apr  5 2024, 05:08:51) 
[GCC 10.3.0] on aix
Type "help", "copyright", "credits" or "license" for more information.

>>> socket.getaddrinfo("127.0.0.1", 5000)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.0.1', 5000))]

>>> socket.getaddrinfo("127.0.0.1", 5000, type=socket.SOCK_STREAM)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 5000))]

I guess this supports my idea of just calling the function twice at the socket.py level to provide consistent behavior across UNIX platforms.

[zyv]

Just checked on Linux, and also returns RAW in addition to STREAM & DGRAM:

zaytsev@fedora:~$ python3
Python 3.12.4 (main, Jun  7 2024, 00:00:00) [GCC 14.1.1 20240607 (Red Hat 14.1.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> socket.getaddrinfo("127.0.0.1", 5000)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 5000)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.0.1', 5000)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('127.0.0.1', 5000))]

So even the behavior of two major platforms (Linux and macOS) is inconsistent.

[picnixz]

So even the behavior of two major platforms (Linux and macOS) is inconsistent.

Can you check whether the behaviour is still inconsistent if you use the same Python version (AFAICT, one is 3.9 for Solaris and the other is 3.12)?

[zyv]

Can you check whether the behaviour is still inconsistent if you use the same Python version (AFAICT, one is 3.9 for Solaris and the other is 3.12)?

Yes, I can confirm that Python 3.9 behaves inconsistently on all platforms:

• Linux: STREAM, DGRAM, RAW
• macOS: STREAM, DGRAM
• OpenBSD: STREAM, DGRAM
• FreeBSD with Cheri: STREAM, DGRAM, SEQPACKET
• AIX: DGRAM (but STREAM supported, obviously!)
• Illumos: exception

I can easily confirm that this behavior persists in Python 3.12 on Linux and macOS, but it will take me a huge effort to build Python 3.12 on AIX and Solaris from scratch to confirm that it's the same.

However, I don't think it's absolutely necessary, because to my knowledge no relevant changes happened between 3.9 and 3.12, and for 3.12 the behavior remains inconsistent for at least 2 of the 4 platforms in question.

[picnixz]

I don't know if it's inconsistent or not actually. The docstring tells me:

Passing zero as a value for each of these arguments selects the full range of results.

Is RAW supported on macOS in general? On the other hand, we need to dig into https://github.com/python/cpython/blob/main/Modules/getaddrinfo.c to see how (and possibly why) the RAW family is not found.

[zyv]

I don't know if it's inconsistent or not actually. The docstring tells me:

Passing zero as a value for each of these arguments selects the full range of results.

Well, throwing an exception on Solaris and returning only SOCK_DGRAM when SOCK_STREAM is definitely supported on AIX doesn't sound like "the full range of results" to me.

If you define consistency as each platform having its own interpretation of what "the full range of results" actually is, then yes, it might be consistent :)

Is RAW supported on macOS in general?

I guess not, because when I explicitly request SOCK_RAW, it still only returns SOCK_STREAM and SOCK_DGRAM (although I would have expected an empty list):

% python3.9              
Python 3.9.20 (main, Sep  6 2024, 19:03:56) 
[Clang 15.0.0 (clang-1500.3.9.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

>>> socket.getaddrinfo("127.0.0.1", 5000, type=socket.SOCK_RAW)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.0.1', 5000)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 5000))]

Perhaps another argument to iterate over SOCK_STREAM, SOCK_DGRAM and SOCK_RAW, making a set and returning a list.

[picnixz]

If you define consistency as each platform having its own interpretation of what "the full range of results" actually is, then yes, it might be consistent :)

Yes that's what I had in mind. I think we rely on the libc implementation of getaddrinfo(3) so the results are platform-dependent and we cannot really do more I guess.

Well, throwing an exception on Solaris and returning only SOCK_DGRAM when SOCK_STREAM is definitely supported on AIX doesn't sound like "the full range of results" to me.

This in particular seems a bug if you can use SOCK_STREAM.

I guess not, because when I explicitly request SOCK_RAW, it still only returns SOCK_STREAM and SOCK_DGRAM (although I would have expected an empty list):

We'll need to investigate whether it's what is being returned by the OS or if we incorrectly fetch the results. I am not very familiar with this part of the code and cannot test it so someone more qualified will need to do it.

[zyv]

We'll need to investigate whether it's what is being returned by the OS or if we incorrectly fetch the results. I am not very familiar with this part of the code and cannot test it so someone more qualified will need to do it.

My understanding is that getaddrinfo(3) is a low-level interface, and C-level users never really need to omit both ai_socktype and ai_protocol at the same time. Usually, if you really don't care about the protocol, you at least set the socket type, otherwise how are you supposed to handle it when you want stream and get datagram instead?

The Python interface, however, has only two mandatory parameters (where the service name is actually not very mandatory, as you can pass None instead), and has the semantics that you have posted. Unfortunately, the low-level interface just doesn't match it. If you look at the CPython implementation of getaddrinfo wrapper, you will see tons of weird hacks like setting the service name to 00 for macOS, or the Tru64+ stuff. Obviously, the intent was to abstract these differences and provide platform-independent semantics to high-level interface users.

So if that's still the goal, I don't see a way around enumerating socket types to provide a consistent interface for high-level users. As you can see from the reaction of the Flask-SocketIO maintainer, he doesn't see a problem with doing getaddrinfo("127.0.0.1", 5000) without setting the protocol or socket type because the documentation says it's fine, and expects Python to abstract the low-level getaddrinfo differences from him.

The Gevent people were bitten by the same bug and ended up adding hacks to their code: gevent/gevent#1256. Eventlet is still broken on Solaris and AIX.