OTP: reuseaddr option is ignored on Windows

[essen]

Sounds like SO_REUSEADDR on Linux is equivalent to SO_REUSE_ADDR + SO_EXCLUSIVEADDR on Windows. Might be worth patching OTP to support it.

Alternatively the new socket NIF should be made to work properly with it from scratch. But since it wants to be low-level perhaps it wants both options exposed instead?

• https://github.com/erlang/otp/blob/master/erts/emulator/drivers/common/inet_drv.c#L6478
• https://stackoverflow.com/questions/14388706/socket-options-so-reuseaddr-and-so-reuseport-how-do-they-differ-do-they-mean-t
• https://docs.microsoft.com/en-us/windows/desktop/WinSock/using-so-reuseaddr-and-so-exclusiveaddruse

[juhlig]

There has been some work done in OTP 25 and 26 around this, see the new reuseaddr, reuseport, reuseport_lb and exclusiveaddruse options in the inet:setopts/2docs.
But it looks like it has been forgotten to adapt them for the socket backend :(

Even leaving the socket backend issue aside, implications are not all good in respect to ranch. The implementation changed a few times in the course of OTP 25.
A thing I see as critical is the release window of OTP >=25 up to <25.2. In those releases, setting {reuseaddr, true} will set the underlying SO_REUSEADDR socket option on Windows without setting SO_EXCLUSIVEADDRUSE, which is tl;dr unsafe. The thing is that ranch_tcp and ranch_ssl set this option implicitly and disallow setting (ie, disabling) it explicitly by the user. So if ranch is used with OTP >=25 up to <25.2 on Windows, the listening socket is set up in this unsafe manner and the user can't even do anything about it even if he is aware of the problem.
In OTP >=25.2 up to <26, {reuseaddr, true} sets SO_REUSEADDR on Windows only for UDP sockets and ignores it for other socket types, so we are safe again as far as ranch is concerned, the behavior is tl;dr the same as before OTP 25.
In OTP >=26, {reuseaddr, true} will set SO_REUSEADDR on Windows only iff {reuseport, true} is set at the same time, and it also introduced the exclusiveaddruse option that can be used to make sockets "safe" (if that is even a term in the Windows world).

[essen]

As long as nobody complains and the behavior is correct moving forward, which sounds like it is, then there is no problem.

[juhlig]

Ah, I remember that view on things of yours from bygone times 😄 Good old times 👴

[hazardfn]

@juhlig / @essen I believe we're experiencing a problem that may be tangentially related to what you're saying but I'm struggling to connect the dots.

After updating to OTP 26 the following ranch option fails to work with the socket backend despite working on previous OTP versions (at least on my darwin dev machine, haven't tested linux):

  @spec reuse_port() :: {:raw, any(), any(), any()} | nil
  defp reuse_port do
    case :os.type() do
      {:unix, :linux} -> {:raw, 1, 15, <<1::32-native>>}
      {:unix, :darwin} -> {:raw, 0xFFFF, 0x0200, <<1::32-native>>}
      _ -> nil
    end
  end

When switching back to the inet backend on OTP 26 this works as before. On at least OTP 24 and 25 this also works with the socket backend. I also found this so it could be a regression: erlang/otp#5122.

EDIT: For clarification this is for enabling SO_REUSEPORT on darwin/linux systems.

[essen]

Could be a regression or at least an unwanted side effect of the new developments. I would recommend writing a small snippet using gen_tcp directly to demonstrate the problem and opening a new ticket in OTP's repository.

[juhlig]

fails to work with the socket backend

@hazardfn What does "fail to work" mean, ie what exactly happens or does not happen when?

[hazardfn]

fails to work with the socket backend

@hazardfn What does "fail to work" mean, ie what exactly happens or does not happen when?

@juhlig Sorry for being unclear, basically SO_REUSEPORT is never set as intended and you end up with :eaddrinuse. Switching back to OTP 25 the same code works fine with the socket backend (we don't get the :eaddrinuse error because SO_REUSEPORT is set as expected). Using the inet backend the same code works in both OTP 25 and 26.

This occurs when creating a child_spec through ranch and starting it as part of the application supervision tree with the raw option as specified above.

Put another way in OTP 25 the {:raw, 0xFFFF, 0x0200, <<1::32-native>>} socket option works with the socket backend, in OTP 26 it seemingly does nothing. I strongly suspect this would also be the case for Linux but I don't have a testing machine to verify that.

[juhlig]

Hm, does setting the {reuseport, true} instead of the raw option work for you, on 26?

[juhlig]

Oh wait, what was I thinking... That option does not work with the socket backend, that was what the ticket I opened at OTP was about 🤪

[juhlig]

I strongly suspect this would also be the case for Linux but I don't have a testing machine to verify that.

You're right, same on Linux. I added it to erlang/otp#7764 (comment)

[essen]

I will do a small 2.2 release soon. Is there anything to be done for that release in Ranch relating to this ticket?

[juhlig]

I don't think there is anything to add right now 🤷‍♂️

FYI, status of erlang/otp#7764 is that the new options are not at all available for the socket backend right now (notices were put up in the docs), and regarding the raw option issue that @hazardfn pointed out, bmk just found out that they work as expected on 26.0 but not on 26.1 and will investigate.

[Maria-12648430]

Nothing from me either 🤷‍♀️

I mean, we could do some fiddling with the default and disallowed listen options, like for Windows setting {reuseaddr, false} on OTP<26 and {reuseaddr, true}, {reuseport, true}, {exclusiveaddruse, true} on OTP>=26 (and disallow changing them) (unless the socket backend is used, where those options are not supported yet). Stuff like that. Not sure if it's worth the trouble.

[juhlig]

FYI: erlang/otp#7764 (comment)

[essen]

You could send a PR with a test case for this and it should be succeeding in about a week when CI recompiles OTP master.