Problems with rabbitmq and current okd Version (4.11.0-0.okd-2023-01-14-152430)

[ffppmm]

Hello,

I started a discussion because rabbitmq started eating memory after the update. We experience this behavior in all clusters we updated from 4.11.0-0.okd-2022-11-19-050030 to 4.11.0-0.okd-2023-01-14-152430.

What we know so far:

• It's not a rabbitmq memory leaking
• Memory problem comes from epmd spawning every minute
• normally the erlang VM ensures epmd is running, but at this point something goes wrong (rabbitmq spawning every minute epmd (in okd version 4.11.0-0.okd-2023-01-14-152430) rabbitmq/rabbitmq-server#7160 (comment))
• the erlang VM can't determind the "old" epmd running and spawns a new empd with a random port

Are there any changes between 4.11.0-0.okd-2022-11-19-050030 and 4.11.0-0.okd-2023-01-14-152430 that could cause this "problem"/behavior. Maybe some Security Constraints have changed?

Anyone an idea why this happens? Do we need to "grant" more rights? Any help/hint is very appreciated!

Thanks in advance,
Philipp

[ffppmm]

Hi,

we did an strace of the network binding:

[pid 376100] socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 376100] setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
[pid 376100] fcntl(3, F_GETFL)          = 0x2 (flags O_RDWR)
[pid 376100] fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 376100] bind(3, {sa_family=AF_INET, sin_port=htons(4369), sin_addr=inet_addr("0.0.0.0")}, 16) = 1
[pid 376100] listen(3, 128)             = 0
[pid 376100] socket(AF_INET6, SOCK_STREAM, IPPROTO_IP) = 4
[pid 376100] setsockopt(4, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0
[pid 376100] setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
[pid 376100] fcntl(4, F_GETFL)          = 0x2 (flags O_RDWR)
[pid 376100] fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 376100] bind(4, {sa_family=AF_INET6, sin6_port=htons(4369), sin6_flowinfo=htonl(0), inet_pton(AF_INET6, "::", &sin6_addr), sin6_scope_id=0}, 28) = 1
[pid 376100] listen(4, 128)             = 0

But there is already a running empd on this port:

beam.smp 2523294 10007900   18u  IPv4  96422974      0t0  TCP 127.0.0.1:34714->127.0.0.1:4369 (ESTABLISHED)
epmd     2523372 10007900    3u  IPv4  96418170      0t0  TCP *:4369 (LISTEN)
epmd     2523372 10007900    4u  IPv6  96418171      0t0  TCP *:4369 (LISTEN)
epmd     2523372 10007900    5u  IPv4  96418197      0t0  TCP 127.0.0.1:4369->127.0.0.1:34714 (ESTABLISHED)

We see that the new epmd runs on a different port not on 4369 as from the strace.

# nsenter --net=/var/run/netns/8a1b8c8c-ceda-4b43-ab81-4508f1493e28 -- lsof -P -n -i 2> /dev/null | grep 376100
epmd      376100 10007900    3u  IPv4 105157026      0t0  TCP *:38539 (LISTEN)
epmd      376100 10007900    4u  IPv6 105157027      0t0  TCP *:34725 (LISTEN)

We would expect the following behavior:

[pid 2781306] socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
[pid 2781306] setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
[pid 2781306] fcntl(3, F_GETFL) = 0x2 (flags O_RDWR)
[pid 2781306] fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
[pid 2781306] bind(3, {sa_family=AF_INET, sin_port=htons(4369), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)
[pid 2781306] exit_group(0) = ?
[pid 2781306] +++ exited with 0 +++

Any ideas?

Regards Philipp

[ffppmm]

Hi,

any thoughts on this? We really need to find out why we get a random port for:
[pid 376100] bind(3, {sa_family=AF_INET, sin_port=htons(4369), sin_addr=inet_addr("0.0.0.0")}, 16) = 1

Regards Philipp

[poreklo]

Hi.

I think this is a bug in the kernel function bind. If the port is busy it returns success instead of error.
epmd expect an error "Address already in use" when calling bind function if port is busy.
While no complete solution yet we have applied a workaround with a sidecar container.
It terminate empd processes that are listening not on port 4369.

spec:
  template:
    spec:
      shareProcessNamespace: true
      containers:
        - name: watchdog
          image: busybox:1.36.1
          args:
            - sh
            - '-c'
            - >-
              while true; do A=$(netstat -nlp 2>/dev/null | grep epmd | grep -v
              4369 | grep -o "[0-9]*/epmd" | grep -o "[0-9]*" | sort | uniq);
              C=$(echo $A | wc -w); echo -n $(date "+%F %T"); if [ $C -eq 0
              ];then echo " Nothing to terminate."; else kill $A; echo "
              Terminated $C processes."; fi; sleep 1200; done
          resources:
            limits:
              cpu: 50m
              memory: 32Mi
            requests:
              cpu: 30m
              memory: 32Mi

[vrutkovs]

Perhaps its https://bugzilla.redhat.com/show_bug.cgi?id=2159802?

[ffppmm]

Sounds like