Default linker settings incompatible with musl

[donob4n]

Hi, I'm trying to build on Alpinelinux with musl. The buil process seems fine but when I run pidgin with debug mode I get:

(11:52:25) plugins: /usr/lib/purple-2/libwhatsmeow.so is not loadable: Error relocating /usr/lib/purple-2/libwhatsmeow.so: sqlite3_autovacuum_pages: initial-exec TLS resolves to dynamic definition in /usr/lib/purple-2/libwhatsmeow.so
Any idea?

[hoehermann]

It looks like sqlite3 assumes some dynamic linking, but musl needs static linkage. I have no experience with musl whatsoever. This article suggests adding -ldflags "-linkmode external -extldflags -static" to CMakeLists.txt could work. Feel free to try that. Unfortunately, I cannot possibly support all the different flavours of Linux by myself. Your input is welcome.

[donob4n]

Hi, I tried it without success. On #alpinelinux someone commented that with "-buildmode=c-archive" it will be ignored.

As an alternative I tried to build it with system sqlite3 lib, but it ended:

(21:57:48) plugins: probing /usr/lib/purple-2/libwhatsmeow.so
Pidgin 2.14.8 has segfaulted and attempted to dump a core file.
This is a bug in the software and has happened through
no fault of your own.

If you can reproduce the crash, please notify the developers
by reporting a bug at:
https://pidgin.im/development/simpleticket/

Please make sure to specify what you were doing at the time
and post the backtrace from the core file.  If you do not know
how to get the backtrace, please read the instructions at
https://pidgin.im/development/wiki/GetABacktrace`

Do you know what should be exactly the way for using system lib?

[hoehermann]

You can get some more information by running pidgin in the debugger: gdb pidgin. Enter r to start Pidgin, wait for the crash, then enter bt to obtain a backtrace with more information about where things went wrong.

This issue could also be overcome by using another SQlite database driver which does not need an external library – such as https://github.com/glebarez/go-sqlite. The configuration syntax seems to be a bit different so purple-whatsmeow needs to be adapted in places, but it could work. I am quite hesitant to do this step because it could break things for users on all the other Linux flavours.

[donob4n]

I'm playing with gdb since yesterday. The problem is that the backtrace on the segfault is not very helpful:

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7fb6f0e in memchr () from /lib/ld-musl-x86_64.so.1
(gdb) bt
#0  0x00007ffff7fb6f0e in memchr () from /lib/ld-musl-x86_64.so.1
#1  0x00007ffff7fb79d1 in strnlen () from /lib/ld-musl-x86_64.so.1
#2  0x00007fffffffdce0 in ?? ()
#3  0x0000000000000000 in ?? ()
(gdb) 

I'm pretty sure that the segfault is produced inside the plugin code but I added some printfs on plugin_init() and I don't get any output.

[hoehermann]

That looks like stack corruption to me (especially the null address in frame).

Maybe it is not the CGO options we need to change, but the final linker pass:

cd build
rm CMakeCache.txt
env LDFLAGS="-static" cmake -DCMAKE_BUILD_TYPE=Debug ..
make

For me, with glibc, this will not build, but maybe it is needed for use with musl. Maybe you need to change the CMakeLists.txt and/or link some libraries statically and others dynamically. This question might be of help.

[donob4n]

Sorry, I was pretty busy. I tried and get this link error:

[100%] Linking C shared library libwhatsmeow.so
/usr/lib/gcc/x86_64-alpine-linux-musl/11.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: /usr/lib/gcc/x86_64-alpine-linux-musl/11.2.1/crtbeginT.o: relocation R_X86_64_32 against hidden symbol `__TMC_END__' can not be used when making a shared object
/usr/lib/gcc/x86_64-alpine-linux-musl/11.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: failed to set dynamic section sizes: bad value
collect2: error: ld returned 1 exit status
make[2]: *** [src/c/CMakeFiles/whatsmeow.dir/build.make:322: src/c/libwhatsmeow.so] Error 1
make[1]: *** [CMakeFiles/Makefile2:169: src/c/CMakeFiles/whatsmeow.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
>>> ERROR: purple-gowhatsapp: build failed

[hoehermann]

Sorry but I am out of ideas. :( I will not install Alpine Linux to investigate this issue directly.

[donob4n]

No problem, I'm out until next week. When I have some time I will try to build it with go-sqlite version.

[hoehermann]

Does https://github.com/hoehermann/purple-gowhatsapp/tree/dev do anything for you?

[donob4n]

I just tested it. I noticed that it uses something like 'modernc.org/sqlite/lib: modernc.org/[email protected]' but unfortunately after building it produces same segfault with same backtrace.

Maybe I need to modify something when building it?

[hoehermann]

Since I am not familiar with the musl environment, I have no idea. I am sorry.

[donob4n]

Hi, I did some progress here. There are two overlapping problems:

golang/go#13492 which I could workaround using golang/go#52541

And that golang compiler generates instructions for TLS access that are invalid in dynamic-loaded libraries (on musl). This can be bypassed using LD_PRELOAD but a nice solution would be to build it using gccgo instead cgo.

I will try to patch the CMakeList but it doesn't seem simple. Would you want to support both compilers?

[hoehermann]

Hi donoban. Nice to hear you made some progress and insightful discoveries.

The current build procedure is a mess. Usually, C sources and go sources reside in one directory. CGO will notice the C sources and compile them with the default compiler, then go will link all objects. Unfortunately, I could not figure out how to set it up correctly. Which is why I need some dummy objects, the go sources are compiled separately and packed into an archive. The C sources are also built on their own and a final external linker pass puts it all together. If you can find an alternative, I would be really happy. So far, I tried two times but did not succeed.

I consider the top-level CMakeLists.txt and win32 configurator to be fine. Please do not touch them.
In contrast, the C CMakeLists.txt is totally ready to be thrown out the window and go CMakeLists.txt in particular.

Up to today, I did not even know gccgo existed. :D I think, supporting two compilers is a hassle. I would like to switch to gccgo completely if that made things easier for linking against musl. However, win32 builds are also requested frequently. I want to be able to provide them. Either by cross-compilation from a Debian-based system or by executing the compiler in a Windows environment provided by wine. As far as I know, there is no gccgo for win32. :(

[donob4n]

Yeah I also discovered it yesterday, and few minutes after wrote here I discovered also that it has a go binary that replicates official go behaviour. So there is no real need to "maintain" both separately.

I've tried to built and failed due some redefitions errors on linkage http://ix.io/4b9z , maybe it's something wrong with cmake or it's directly a gccgo problem. I will open a issue on Alpine gitlab since there is a developer interested on fix gcc-go issues.

[hoehermann]

When building with gcc-go, you must not include purple-dummy.c in the build. The dummy is a bodge which is only necessary due to a quirk in the way cgo deduces function signatures. The dummy may be build, but the resulting object file must never be linked into the final shared object.

[donob4n]

WoW Yeah!! It builds fine after removing it!

[donob4n]

Ouch, I was too soon glad:

plugins: /usr/lib/purple-2/libwhatsmeow.so is not loadable: Error relocating /usr/lib/purple-2/libwhatsmeow.so: runtime.mapaccess2__fast64: symbol not found

[hoehermann]

Hm… Maybe the go runtime shipped with gcc-go is not as fully fledged as the google-go one?

[donob4n]

Well, finally I got it working with gcc-go and a small patch https://gitlab.alpinelinux.org/alpine/aports/-/merge_requests/39618#252cf8dc5c6b53ea319ac57bbf1f640f4e7f49cf

Maybe it is not accepted but at least purple-gowhatsapp is now working for me :)

I'm closing this. Thanks for the help and write the plugin @hoehermann 😃

[hoehermann]

Well done. Nice work. Can you share your changes to the CMakeLists.txt? Many others have tried compiling with gcc-go (for example #86), but did not succeed. I would like to see how it is done. :)

[donob4n]

Yeah, here is remove-dummy.txt

I need to explicit link to '-lgo' which is probably wrong but probably is related with the splitted c/go sources.

[hoehermann]

I like to provide as much compatibility as I can. I tried your patch, but on Ubuntu 22.04 gccgo version 11.2.0 fails with "unrecognized command-line option ‘-buildmode=c-archive’". It looks like gccgo is not a drop-in replacement for google-go, henceforth the build commands need more system-specific adjustments. :( This will remain on my "nice to have, but too much effort"-list.

[donob4n]

Maybe it's due gcc version? I tried on alpinelinux edge which has gcc 12 and ubuntu seems on gcc 11.