Goldy segmentation faults and malloc assertion failures

[benalexau]

I have been testing Goldy Client as an option to fronting Aeron, which uses UDP to transport and reassemble user messages. Configuration:

./goldy -l 127.0.0.1:60123 -b 127.0.0.1:40123 -c localhost.crt -k localhost.key --log=INFO
./goldy-client -l 127.0.0.1:50123 -b 127.0.0.1:60123 -c rootCA.crt --log=INFO

Simple (low volume) tests worked fine, but as I stressed the system Aeron started receiving corrupt UDP messages via the Goldy tunnel.

In an effort to verify correct operation of individual components, I used nc -ul 40123 > file to capture a file sent through Goldy with cat file | nc -u 127.0.0.1 50123. If this file was small, the shasum proved correct delivery. However larger files resulted in either corrupt delivery, Goldy Client segmentation faults, or Goldy Client assertion errors.

It is possible to produce the following segmentation fault using openssl rand 4027 |nc -u 127.0.0.1 50123. A random size of 4026 does not produce the fault.

$./goldy-client -l 127.0.0.1:50123 -b 127.0.0.1:60123 -c rootCA.crt --log=DEBUG
2017-03-15 14:17:17.266015 INFO  Goldy 0.2 starting up
2017-03-15 14:17:17.266154 DEBUG Binded UDP 127.0.0.1:50123
2017-03-15 14:17:17.266232 DEBUG Loaded server cacert file
2017-03-15 14:17:17.266271 DEBUG Seeded random number generator
2017-03-15 14:17:17.266300 INFO  Proxy is ready, listening for connections on UDP 127.0.0.1:50123
2017-03-15 14:17:17.266328 INFO  main_loop - start
2017-03-15 14:17:17.266335 DEBUG start_listen_io - 3
2017-03-15 14:17:19.612061 DEBUG global_cb fds: 3,3 revents: 0x01 count: 0
2017-03-15 14:17:19.612149 DEBUG connect_to_new_client: connected on fd 6
2017-03-15 14:17:19.612206 INFO  (127.0.0.1:41015) Client connected
*** Error in `./goldy-client': free(): invalid next size (fast): 0x0000000001fca6e0 ***
======= Backtrace: =========
/usr/lib/libc.so.6(+0x70c4b)[0x7f13430a7c4b]
/usr/lib/libc.so.6(+0x76fe6)[0x7f13430adfe6]
/usr/lib/libc.so.6(+0x777de)[0x7f13430ae7de]
/usr/lib/libc.so.6(freeaddrinfo+0x28)[0x7f134310c048]
./goldy-client[0x407288]
./goldy-client[0x404986]
./goldy-client[0x404a29]
./goldy-client[0x405da2]
./goldy-client[0x4488a3]
./goldy-client[0x44ca21]
./goldy-client[0x403886]
./goldy-client[0x405e72]
./goldy-client[0x405f34]
/usr/lib/libc.so.6(__libc_start_main+0xf1)[0x7f1343057291]
./goldy-client[0x40379a]
======= Memory map: ========
00400000-00477000 r-xp 00000000 09:00 530546                             /home/bpa/projects/goldy-client/goldy-client
00676000-00677000 r--p 00076000 09:00 530546                             /home/bpa/projects/goldy-client/goldy-client
00677000-00678000 rw-p 00077000 09:00 530546                             /home/bpa/projects/goldy-client/goldy-client
00678000-0067b000 rw-p 00000000 00:00 0 
01fbf000-01fe0000 rw-p 00000000 00:00 0                                  [heap]
7f133c000000-7f133c021000 rw-p 00000000 00:00 0 
7f133c021000-7f1340000000 ---p 00000000 00:00 0 
7f1342e20000-7f1342e36000 r-xp 00000000 09:00 1314335                    /usr/lib/libgcc_s.so.1
7f1342e36000-7f1343035000 ---p 00016000 09:00 1314335                    /usr/lib/libgcc_s.so.1
7f1343035000-7f1343036000 r--p 00015000 09:00 1314335                    /usr/lib/libgcc_s.so.1
7f1343036000-7f1343037000 rw-p 00016000 09:00 1314335                    /usr/lib/libgcc_s.so.1
7f1343037000-7f13431cc000 r-xp 00000000 09:00 1313965                    /usr/lib/libc-2.24.so
7f13431cc000-7f13433cb000 ---p 00195000 09:00 1313965                    /usr/lib/libc-2.24.so
7f13433cb000-7f13433cf000 r--p 00194000 09:00 1313965                    /usr/lib/libc-2.24.so
7f13433cf000-7f13433d1000 rw-p 00198000 09:00 1313965                    /usr/lib/libc-2.24.so
7f13433d1000-7f13433d5000 rw-p 00000000 00:00 0 
7f13433d5000-7f13434d8000 r-xp 00000000 09:00 1314023                    /usr/lib/libm-2.24.so
7f13434d8000-7f13436d7000 ---p 00103000 09:00 1314023                    /usr/lib/libm-2.24.so
7f13436d7000-7f13436d8000 r--p 00102000 09:00 1314023                    /usr/lib/libm-2.24.so
7f13436d8000-7f13436d9000 rw-p 00103000 09:00 1314023                    /usr/lib/libm-2.24.so
7f13436d9000-7f13436fc000 r-xp 00000000 09:00 1313964                    /usr/lib/ld-2.24.so
7f13438d5000-7f13438d9000 rw-p 00000000 00:00 0 
7f13438fa000-7f13438fb000 rw-p 00000000 00:00 0 
7f13438fb000-7f13438fc000 r--p 00022000 09:00 1313964                    /usr/lib/ld-2.24.so
7f13438fc000-7f13438fd000 rw-p 00023000 09:00 1313964                    /usr/lib/ld-2.24.so
7f13438fd000-7f13438fe000 rw-p 00000000 00:00 0 
7ffdc1798000-7ffdc17b9000 rw-p 00000000 00:00 0                          [stack]
7ffdc17c8000-7ffdc17ca000 r--p 00000000 00:00 0                          [vvar]
7ffdc17ca000-7ffdc17cc000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Aborted (core dumped)

In addition, running openssl rand 4026 |nc -u 127.0.0.1 50123 with varying delays between each invocation will occasionally lead to:

2017-03-15 14:00:38.066473 INFO  Created socket to backend UDP 127.0.0.1:60123
2017-03-15 14:00:38.388375 INFO  (127.0.0.1:58484) Session closed
2017-03-15 14:00:38.388453 INFO  (127.0.0.1:52739) Client connected
2017-03-15 14:00:38.388483 INFO  Created socket to backend UDP 127.0.0.1:60123
2017-03-15 14:00:41.828988 INFO  (127.0.0.1:52739) Session closed
2017-03-15 14:00:41.829031 INFO  (127.0.0.1:53196) Client connected
2017-03-15 14:00:41.829047 INFO  Created socket to backend UDP 127.0.0.1:60123
goldy-client: malloc.c:2403: sysmalloc: Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed.

The above was compiled using Goldy Client 4f70c9f.