Crash under glibc 2.41 due to libopenlibm having an executable stack

[kpobrien]

With glibc-2.41,

ldd --version                                                      
ldd (GNU libc) 2.41
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

I get the following error trying to run Julia 1.9.4, 1.10.8, and 1.11.3.

ERROR: Unable to load dependent library /home/kevin/.julia/juliaup/julia-1.11.3+0.x64.linux.gnu/bin/../lib/julia/libopenlibm.so
Message:/home/kevin/.julia/juliaup/julia-1.11.3+0.x64.linux.gnu/bin/../lib/julia/libopenlibm.so: cannot enable executable stack as shared object requires: Invalid argument

Nightly (1.12.0-DEV.1971) starts without an error.

Downgrading to glibc-2.40 and lib32-glibc-2.40 fixes the error on all versions.

ldd --version
ldd (GNU libc) 2.40
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

I assume it's the same issue as ValveSoftware/portal2#451 which as the author points out, likely results from this change in glibc-2.41:

https://lists.gnu.org/archive/html/info-gnu/2025-01/msg00014.html

• dlopen and dlmopen no longer make the stack executable if a shared
  library requires it, either implicitly because of a missing GNU_STACK
  ELF header (and default ABI permission having the executable bit set)
  or explicitly because of the executable bit in GNU_STACK, and the
  stack is not already executable. Instead, loading such objects will
  fail.

[gbaraldi]

JuliaMath/openlibm#307 need to make sure 1.10 and 1.11 also have an openLibm with this PR

[maleadt]

This effectively breaks Julia on Arch Linux. Workaround: use execstack to clear the openlibm executable stack flag:

❯ julia
ERROR: Unable to load dependent library /home/tim/Julia/depot/juliaup/julia-1.11.3+0.x64.linux.gnu/bin/../lib/julia/libopenlibm.so
Message:/home/tim/Julia/depot/juliaup/julia-1.11.3+0.x64.linux.gnu/bin/../lib/julia/libopenlibm.so: cannot enable executable stack as shared object requires: Invalid argument

❯ execstack -c /home/tim/Julia/depot/juliaup/julia-1.11.3+0.x64.linux.gnu/bin/../lib/julia/libopenlibm.so

❯ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.11.3 (2025-01-21)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia>

[lgoettgens]

Thank you so, so much @maleadt for providing this workaround! That made it really easy to go back to work instead of trying to fix my system for half a day.

[nsajko]

The Archlinux-packaged julia starts successfully, presumably because of the newer openlibm.

[nsajko]

Should v1.9 also get a fix for this, even though it's not supported any more? To prevent breaking CI.

[gbaraldi]

Probably not. 1.9 is not supported anymore so no reason to bother.

[kpobrien]

It looks like Ubuntu 25.04 Plucky Puffin plans to include glibc-2.41 with a scheduled release near the end of April 2025, so this issue is going to impact distros beyond Arch and other rolling releases relatively soon. I agree Julia has no obligation to backport this fix further back than 1.10, but, in my opinion, backporting further would significantly help those community members trying to maintain backwards compatibility further back than supported by the language. For example, some packages like StaticArrays.jl still test on 1.6. Of course, it's the Julia maintainers' call since you're doing the work to make these changes.

[giordano]

New builds of openlibm 0.8.1 (JuliaPackaging/Yggdrasil#10447) which include the fix (JuliaMath/openlibm#307) have been added to #57182 (julia v1.10.9) and #57183 (julia v1.11.4)

[jrharbin-york]

Same issue on Gentoo, after glibc upgraded as part of regular system updates.

execstack is not in the standard repo, but patchelf is, and the following command fixed it:

patchelf --clear-execstack /home/jharbin/.julia/juliaup/julia-1.11.3+0.x64.linux.gnu/bin/../lib/julia/libopenlibm.so

Just thought this might be helpful to anyone else who doesn't have the execstack utility.

[tecosaur]

I do wonder if there might be a path towards no longer dynamically linking to glibc in the future, to never have to deal with issues like this again.

[giordano]

I do wonder if there might be a path towards no longer dynamically linking to glibc in the future, to never have to deal with issues like this again.

You mean like statically linking a libc in julia and all shared libraries that interact with it? That'd be quite a lot of embedding.

[vchuravy]

I don't think that's feasible since this change was IIUC in ld.so so the dynamic loader. (Which Elliot and I have been wanting to rewrite and embed in Julia, but that is different than statically linking libc).

[nsajko]

no longer dynamically linking to glibc

I don't think that's viable. Statically linking glibc is strongly discouraged, because some functionality depends on dynamic linking. Furthermore, it's not in general possible to completely avoid libc, because some systems, unlike Linux, don't have a stable syscall interface. EDIT: I think the Golang folks had similar ambitions in the beginning, but had to reverse track.

[tecosaur]

You mean like statically linking a libc in julia and all shared libraries that interact with it? That'd be quite a lot of embedding.

I was thinking of the viability of a statically compiled julia executable, like Zig can do. Interesting to hear the golang folks may have tried this and run into issues.

I don't think that's feasible since this change was IIUC in ld.so so the dynamic loader. (Which Elliot and I have been wanting to rewrite and embed in Julia, but that is different than statically linking libc).

Thanks for the context 👍

[divital-coder]

Hello, I was testing out a shared object lib I built to run tests. and i got something like the following :

Info Given HepMC3 was explicitly requested, output will be shown live 
ERROR: LoadError: could not load library "/home/hurtbadly/Desktop/julia_stuff/high_energy_physics/HepMC3-dev.jl/gen/build/lib/libHepMC3Wrap.so"
libjulia.so.1: cannot enable executable stack as shared object requires: Invalid argument

Am i right with atleast partial certainty, that it makes sense to post it here?

[giordano]

Am i right with atleast partial certainty, that it makes sense to post it here?

No, it's a problem with the specific library, nothing to do with Julia itself. The only reason why the openlibm issue is somewhat related to Julia is because Julia ships openlibm by default, but it was entirely an upstream issue, again, nothing specific to Julia itself strictly speaking.

[dgsob]

The same Julia libopenlibm Invalid argument issue happens on openSUSE Tumbleweed snapshot 20250303.
The execstack workaround does work, execstack can be installed with zypper.

[My-Kingdom-for-a-Cat]

Same issue here in debian

> ldd --version
ldd (Debian GLIBC 2.41-3) 2.41
Copyright (C) 2024 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

When starting julia, I get the error

> julia
ERROR: Unable to load dependent library /snap/julia/129/bin/../lib/julia/libopenlibm.so
Message:/snap/julia/129/bin/../lib/julia/libopenlibm.so: cannot enable executable stack as shared object requires: Invalid argument

Julia version in snap is:

julia 1.11.3 129 latest/stable julialang✓ classic

The execstack workaround doesn't seem to work, nor the patchelf one on the snap version of julia (because snap directories are readonly).

Using the official release, Version 1.11.3 (2025-01-21), a similar error arises when lauching julia.
In Debian, the workaround

patchelf --clear-execstack path-to-your-julia/libopenlibm.so
does work.

[kpobrien]

Confirming that the crash with glibc-2.41 is fixed in Julia 1.10.9 and 1.11.4. The crash still exists on 1.9.4 and earlier, but my understanding is the decision was to not backport further than the LTS. Thanks all for figuring out and fixing this!

[mkitti]

https://www.phoronix.com/news/Glibc-WA-Steam-Exec-Stack

The release of the GNU C Library 2.41 at the end of January ended up inadvertently breaking some Steam games, Discord, Julia, MATLAB, and other select user-space software. A workaround was merged today for Glibc to workaround the problem.

[giordano]

For the benefit of future readers, another way to start Julia (or any other application which requires an executable stack) without changing any file on disk is to set the GLIBC_TUNABLES environment variable to control the option glibc.rtld.execstack:

% julia +1.9 -E '"Hello world"'                                      
ERROR: Unable to load dependent library /home/mose/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/bin/../lib/julia/libopenlibm.so
Message:/home/mose/.julia/juliaup/julia-1.9.4+0.x64.linux.gnu/bin/../lib/julia/libopenlibm.so: cannot enable executable stack as shared object requires: Invalid argument
% GLIBC_TUNABLES=glibc.rtld.execstack=2 julia +1.9 -E '"Hello world"'
"Hello world"

Use this option with caution.