Gem contains hardcoded path to /home/flavorjones

[paulmenzel]

Using OpenProject with Nokogiri 1.14.2, we noticed, the automounter trying to mount /home/flavorjones:

2024-02-20T11:58:08+01:00 platsch  automount[96539]: attempting to mount entry /home/flavorjones
2024-02-20T11:58:08+01:00 platsch  automount[96539]: key "flavorjones" not found in map source(s).
2024-02-20T11:58:08+01:00 platsch  automount[96539]: failed to mount /home/flavorjones

This is „caused“ by bin/rake jobs:workoff.

$ fd nokogiri
ruby-3.2.1/lib/ruby/gems/3.2.0/cache/nokogiri-1.14.2-x86_64-linux.gem
ruby-3.2.1/bin/nokogiri
ruby-3.2.1/lib/ruby/gems/3.2.0/specifications/nokogiri-1.14.2-x86_64-linux.gemspec
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/ruby-saml-1.15.0/gemfiles/nokogiri-1.5.gemfile
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/lib/xsd/xmlparser/nokogiri.rb
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/lib/nokogiri/
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/lib/nokogiri/jruby/nokogiri_jars.rb
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/lib/nokogiri/3.2/nokogiri.so
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/lib/nokogiri/3.1/nokogiri.so
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/lib/nokogiri/3.0/nokogiri.so
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/lib/nokogiri/2.7/nokogiri.so
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/lib/nokogiri.rb
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/ext/nokogiri/
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/ext/nokogiri/nokogiri.h
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/ext/nokogiri/nokogiri.c
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/bin/nokogiri
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/activesupport-7.0.4.2/lib/active_support/xml_mini/nokogirisax.rb
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/activesupport-7.0.4.2/lib/active_support/xml_mini/nokogiri.rb
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/aws-sdk-core-3.170.0/lib/aws-sdk-core/xml/parser/engines/nokogiri.rb
ruby-3.2.1/lib/ruby/gems/3.2.0/gems/tilt-2.1.0/lib/tilt/nokogiri.rb

[paulmenzel]

It tries to access the files below:

 $ sudo ./opensnoop | grep flavorjones
 […]
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib/tls/x86_64/x86_64/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib/tls/x86_64/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib/tls/x86_64/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib/tls/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib/x86_64/x86_64/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib/x86_64/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib/x86_64/libdl.so.2
 WARNING: CPU:0 [LOST 143 EVENTS]
 WARNING: CPU:0 [LOST 322 EVENTS]
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/libxml2/2.10.3/lib/glibc-hwcaps/x86-64-v3/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/libxml2/2.10.3/lib/glibc-hwcaps/x86-64-v2/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/libxml2/2.10.3/lib/tls/x86_64/x86_64/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/libxml2/2.10.3/lib/tls/x86_64/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/libxml2/2.10.3/lib/tls/x86_64/libdl.so.2
 ruby             70231    -1 /home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/libxml2/2.10.3/lib/tls/libdl.so.2

[flavorjones]

Hi! Thanks for opening this issue, I'll try to help.

I don't know what OpenProject is, or what its "automounter" is, or what opensnoop is, or why what you're seeing is a problem for you. Can you explain a bit more to help me understand?

I suspect what you're seeing is an artifact from how the precompiled libraries are linked (specifically the -rpath option), but I would like to better understand what you're seeing before speculating any further than that.

[paulmenzel]

Hi! Thanks for opening this issue, I'll try to help.

Thank you very much for your prompt reply.

I don't know what OpenProject is,

Sorry, OpenProject is a Ruby on Rails application.

or what its "automounter" is, or what opensnoop is, or why what you're seeing is a problem for you. Can you explain a bit more to help me understand?

From automount(8):

The automount program is used to manage mount points for autofs, the inlined Linux automounter. automount works by reading the auto.master(5) map and sets up mount points for each entry in the master map allowing them to be automatically mounted when accessed. The file systems are then automatically umounted after a period of inactivity.

In big environments with network shares (NFS), to avoid having all network shares mounted the whole time, autofs/automount is used to mount the directory just in time (and later unmount them). In our network environment project directories and home directories (/home/x) are managed that way.

opensnoop traces open() syscalls by using Linux’ bpftrace/eBPF, and was used to figure out, what process actually tried to open what file, as that wasn’t visible from the automount logs.

If such a directory does not exist, automount logs an error. This log message is in our logs over 30000 times and increases their size, and looking at the logs other things are harder to spot.

I suspect what you're seeing is an artifact from how the precompiled libraries are linked (specifically the -rpath option), but I would like to better understand what you're seeing before speculating any further than that.

Yes, that is also my theory. As systems won’t have the path /home/flavorjones, unneeded accesses are done on these systems too, but nobody notices, because no error is logged.

[flavorjones]

OK, thanks for giving that additional context. I'll take a look later today to try to correlate the linker paths to what you're seeing, and explore how to fix that in the precompiled builds.

In the meantime, you may want to look into upgrading to the latest version (v1.16.x) of Nokogiri, since any potential fix won't be backported to older minor branches.

[flavorjones]

OK, I'm fairly certain that the RPATH property of the nokogiri.so ELF files shipped with the precompiled gem are to blame.

Here's what I'm doing both to approximate what you're seeing, and testing this hypothesis:

# where is the precompiled nokogiri.so on my system?
$ sofile=$(ruby -r nokogiri -e 'puts $LOADED_FEATURES.grep(/nokogiri\.so/)')
$ echo $sofile
/home/flavorjones/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/nokogiri-1.16.2-x86_64-linux/lib/nokogiri/3.2/nokogiri.so

# what's in the RPATH?
$ readelf -d $sofile | fgrep RPATH
0x000000000000000f (RPATH)              Library rpath: [/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.2.0/lib:/home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.3/lib:/home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/libxml2/2.12.5/lib:/home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/libxslt/1.1.39/lib:/home/flavorjones/code/oss/nokogiri/tmp/x86_64-linux/nokogiri/3.2.0/ports/x86_64-linux/libgumbo/1.0.0-nokogiri/lib]

# let's take the first dir in the RPATH and look for a system call that tries to open that directory
$ rpathdir=/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.2.0/lib
$ strace ruby -rnokogiri -e 'exit 0' 2>&1 | fgrep $rpathdir
openat(AT_FDCWD, "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.2.0/lib/glibc-hwcaps/x86-64-v3/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.2.0/lib/glibc-hwcaps/x86-64-v3/", 0x7ffc107916a0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.2.0/lib/glibc-hwcaps/x86-64-v2/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.2.0/lib/glibc-hwcaps/x86-64-v2/", 0x7ffc107916a0, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.2.0/lib/libdl.so.2", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.2.0/lib/", 0x7ffc107916a0, 0) = -1 ENOENT (No such file or directory)

# now let's remove the RPATH
$ patchelf --shrink-rpath $sofile
$ readelf -d $sofile | fgrep RPATH
# <empty output>

# and try strace again
$ strace ruby -rnokogiri -e 'exit 0' 2>&1 | fgrep $rpathdir
# <empty output>

So at this point I'll dig into modifying how the .so files are generated and try to prevent the RPATH from being set inappropriately.

[paulmenzel]

Wow, thank you very much for your research. I can confirm the RPATH being set in the shared library:

readelf -d ruby-3.2.1/lib/ruby/gems/3.2.0/gems/nokogiri-1.14.2-x86_64-linux/lib/nokogiri/3.2/nokogiri.so | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/usr/local/rake-compiler/ruby/x86_64-redhat-linux/ruby-3.2.0/lib:/home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/zlib/1.2.13/lib:/home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/libxml2/2.10.3/lib:/home/flavorjones/code/oss/nokogiri/ports/x86_64-linux/libxslt/1.1.37/lib:/home/flavorjones/code/oss/nokogiri/tmp/x86_64-linux/nokogiri/3.2.0/ports/x86_64-linux/libgumbo/1.0.0-nokogiri/lib]

[flavorjones]

See some exploration of solutions at #3137