SONiC-VPP T1 Topology: syncd crashing every 3-4 hours

[nikunjdh]

Investigation notes from Shashi:
The crash seems to be happening in bignumbers arithmetic operation. The code is from gmp library I think and is used mostly in cryptographic algorithms implementation.

PFA syncd core dump stack trace generated with gdb:

cisco@cisco-ubuntu:/tmp/syncd.core$ gdb syncd syncd.1694514037.66.core
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from syncd...
(No debugging symbols found in syncd)
[New LWP 96]
[New LWP 66]
[New LWP 87]
[New LWP 166]
[New LWP 88]
[New LWP 95]
[New LWP 135]
[New LWP 138]
[New LWP 140]
[New LWP 92]
[New LWP 89]
[New LWP 91]
[New LWP 97]
[New LWP 99]
[New LWP 100]
[New LWP 134]
[New LWP 136]
[New LWP 139]
[New LWP 90]
[New LWP 86]
[New LWP 101]
[New LWP 98]
[New LWP 103]
[New LWP 102]
[New LWP 137]
[New LWP 141]

warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/libuuid.so.1" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/libnl-route-3.so.200" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/lib/x86_64-linux-gnu/libnl-3.so.200" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/lib64/ld-linux-x86-64.so.2" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/libbsd.so.0" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/libsodium.so.23" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/lib/x86_64-linux-gnu/libpcre.so.3" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/libkrb5.so.3" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/lib/x86_64-linux-gnu/libcom_err.so.2" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/lib/x86_64-linux-gnu/libkeyutils.so.1" is not at the expected address (wrong library or version mismatch?)

warning: .dynamic section for "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1" is not at the expected address (wrong library or version mismatch?)

warning: Could not load shared library symbols for 18 libraries, e.g. /usr/lib/x86_64-linux-gnu/libsaimetadata.so.0.
Use the "info sharedlibrary" command to see the complete listing.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/syncd -u -s -p /usr/share/sonic/hwsku/sai.profile'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007ff7cf545ce1 in __mpn_impn_mul_n (prodp=0x7ff7c5ffb700, up=0x0, vp=0x8, size=0, tspace=) at mul_n.c:115
115 mul_n.c: No such file or directory.
[Current thread is 1 (Thread 0x7ff7c5ffb700 (LWP 96))]
(gdb) bt full
#0 0x00007ff7cf545ce1 in __mpn_impn_mul_n (prodp=0x7ff7c5ffb700, up=0x0, vp=0x8, size=0, tspace=) at mul_n.c:115
esize = 8
cy_limb = 0
#1 0x0000000000000000 in ?? ()
No symbol table info available.
(gdb)