Port reuse problems / SOREUSEPORT support

[ungps]

When spamming Unikraft HTTPserver with lots of request using curl, the server eventually gets stuck. By inspecting the pcap in Wireshark, we would see last connection would try to "TCP Port reuse" in this case.

Alternatively, instead of spaming with connections, we have a more consistent way to replicate the bug. By using --local-port 42000 (or whatever port) option in curl and just do three requests we would see a similar behaviour (but without all the clutter before Unirkaft gets stuck): first curl would work just fine. The second one will fail with the message "bind failed with errno 98: Address already in use" (or errno 7 "Failed to connect to 172.44.0.2 port 8123: Connection refused", depending on Unikraft version). After that point all other requests would get stuck.

The cause could be that LWIP does not have support for SO_REUPOSRT flag and tries to reuse the TCP port by default. Something that helped a bit was enabling LWIP_SO_LINGER and setting the SO_LINGER with timeout 0 on the socket. By doing so, we'de get back the response, but the connection is still weird. After receiving the response, the connection would be force reset: curl: (56) Recv failure: Connection reset by peer.