Metro Exodus (412020) extremely bad framerates (Arch + nvidia 465); works fine on steam-native

[dllu]

Your system information

• Steam Runtime Version: steam-runtime_0.20210317.0
• Distribution: Arch Linux
• Link to your full system information: https://gist.github.com/dllu/665971b00979b9b29db834a34f50a433
• Have you checked for system updates?: Yes
• Are you using the Steam Linux Runtime compatibility tool?: No

Please describe your issue in as much detail as possible:

Metro Exodus runs at extremely poor framerates (2 to 5 fps) even in the main menu with RTX 3090 whereas other games work fine. The slowness happens with or without RTX on.

Running perf seems to suggest that some threading stuff is causing the slowness. Meanwhile nvidia-smi suggests that the GPU is being used but barely.

NVIDIA drivers are all up to date.

Metro Exodus was able to run at good framerates at extreme settings (60 fps 3840 x 2160 with RTX On) when running steam-native from the steam native runtime Arch repo. This suggests that steam-runtime was causing the problem.

From https://steamcommunity.com/app/412020/discussions/0/3103518890055200590/?ctp=24 it seems a similar issue is affecting other players as well.

Steps for reproducing this issue:

1. pacman -S steam nvidia on Arch Linux with RTX 3090 graphics card
2. Launch Metro Exodus. Framerates are really bad.

However if you install pacman -S steam-native-runtime and then launch steam with steam-native, the performance is very good.

[smcv]

If this is a native Linux game, then the vast majority of the libraries it's loading should be coming from your host system anyway, assuming the same SONAME for the same word size is available there.

If you look in /proc/$pid/maps, where $pid is the process ID of Metro Exodus's main process, which of its libraries are coming from the Steam Runtime?

Is Metro Exodus a 32-bit or 64-bit game? (You'll be able to tell this by looking at /proc/$pid/maps too.)

For example, when I do the same with Floating Point, a 32-bit game, on a test machine, we can see that among other things, it's using libogg.so.0 from the Steam Runtime, but libxcb-shm.so.0 from my Debian system:

$ pgrep "Floating Point"
67007
$ less /proc/67007/maps
...
... r-xp ... /home/me/.local/share/Steam/ubuntu12_32/steam-runtime/usr/lib/i386-linux-gnu/libogg.so.0.8.2
...
... r-xp ... /usr/lib/i386-linux-gnu/libxcb-shm.so.0.0.0
...

On Arch, 32-bit libraries that come from your host system would show up as coming from /usr/lib32 or /lib32 instead of my /usr/lib/i386-linux-gnu or /lib/i386-linux-gnu. Similarly, if Metro Exodus is 64-bit, your Arch libraries would be in /usr/lib or /lib and my Debian libraries would be /usr/lib/x86_64-linux-gnu or /lib/x86_64-linux-gnu.

[smcv]

We specifically don't support the "native runtime" from Arch. However, it's sometimes a useful way to find and fix issues in the official Steam Runtime.

If you install all the libraries that steam-native depends on (such as for example lib32-glib2), but then run Metro Exodus in the official Steam Runtime (not the "native" runtime), do you still get the bad performance? You will probably find that more libraries from your host system and fewer libraries from the Steam Runtime are used in this situation.

If you still get the bad performance, then we know that the problem must be one of the few libraries from the Steam Runtime that you are still using, so that should narrow it down a lot.

If you get the better performance by just installing steam-native and its dependencies, then the next thing to try would be to uninstall steam-native, gradually uninstall the packages that it depends on, and see which one triggers the performance drop.

[dllu]

If this is a native Linux game, then the vast majority of the libraries it's loading should be coming from your host system anyway, assuming the same SONAME for the same word size is available there.

Yes this is a native Linux game. It was released natively on Linux several weeks ago. This is, I think, the first and currently only AAA game that works natively on Linux and has advanced graphics (RTX etc) so I think it is worthwhile to ensure that it works as well as possible.

If you install all the libraries that steam-native depends on (such as for example lib32-glib2), but then run Metro Exodus in the official Steam Runtime (not the "native" runtime), do you still get the bad performance?

Yes, I do get bad performance with the official steam runtime in this scenario.

If you get the better performance by just installing steam-native and its dependencies, then the next thing to try would be to uninstall steam-native, gradually uninstall the packages that it depends on, and see which one triggers the performance drop.

I have both steam and steam-native installed and the performance with the official steam runtime is still terrible.

more debugging

I have disabled Steam Overlay in the settings but it has not improved matters at all.

/proc/$pid/maps

Here's the cat /proc/$pid/maps > maps.log. Using steam (default steam launch script which is aliased to steam-runtime, not steam-native).

maps.log

and here is the same thing but using steam-native:

maps-native.log

So by diffing these two logs it seems that with steam-runtime it is somehow using the system version of libSDL2 and not the one that Metro Exodus ships with, whereas steam-native is loading the libSDL2 that comes with Metro. That is probably the culprit...

Now I wonder why that would be the case 🤔

[smcv]

So by diffing these two logs it seems that with steam-runtime it is somehow using the system version of libSDL2 and not the one that Metro Exodus ships with, whereas steam-native is loading the libSDL2 that comes with Metro

My guess would be that Metro Exodus (or its launcher scripts?) appends to LD_LIBRARY_PATH, rather than prepending? steam-native doesn't use LD_LIBRARY_PATH much, but the runtime relies on it for basically everything.

If a game wants to carry its own special version of a library and use it in preference to other versions, then it should probably be prepending to LD_LIBRARY_PATH - although we'd generally recommend the version from the OS or the Steam Runtime (whichever one is newer, and the Steam Runtime should automatically be choosing the newer one).

[dllu]

I think the libSDL2 that ships with Metro Exodus is 2.0.8 but the one in my system is 2.0.14. But as you can see here, for some reason the newer one on my system is resulting in extremely poor performance.

Metro Exodus does not use any launcher scripts; it simply launches the elf binary MetroExodus.

[smcv]

OK, so this is a 64-bit game. That makes things easier.

If you try changing the Launch Options for Metro Exodus to each of these, which ones get good performance and which ones get bad? They should have the same result for steam-runtime and steam-native.

Bundled libSDL2 (2.0.8, you think):

LD_PRELOAD="/home/dllu/.local/share/Steam/steamapps/common/Metro Exodus/libSDL2.so:$LD_PRELOAD" %command%

System libSDL2 (2.0.14):

LD_PRELOAD="/usr/lib/libSDL2-2.0.so.0.14.0:$LD_PRELOAD" %command%

Steam Runtime libSDL2 (should also be 2.0.14):

LD_PRELOAD="/home/dllu/.local/share/Steam/ubuntu12_32/steam-runtime/usr/lib/x86_64-linux-gnu/libSDL2-2.0.so.0:$LD_PRELOAD" %command%

[Termuellinator]

I'm also affected by this - steam-runtime used to yield solid 5fps (seems to have gotten better ;)s, no matter the resolution or quality setting, steam-native yields 120fps (vsync) on 1440p ultra.
I'm on manjaro testing.

I tried your suggestions with steam-runtime:

• Bundled libSDL2 (2.0.8, you think): 15fps
• System libSDL2 (2.0.14): 15fps
• Steam Runtime libSDL2 (should also be 2.0.14): 15fps
• steam-runtime without any LD_PRELOAD: 15fps
• steam-native doesn't start currently ( src/clientdll/appdatacache.cpp (2679) : Assertion Failed: !bSharedKVSymbols , seems to be Steam won't start "appdatacache.cpp (2078) : Assertion Failed: !bSharedKVSymbols" steam-for-linux#5014 ) but as said, when i tried some time ago (see https://steamcommunity.com/app/412020/discussions/0/3103518890055200590/?ctp=24#c5767400709474516396 ) it had the performance that is to be expected on a 6900XT

[smcv]

steam-native is not supported by Valve, you're on your own with that version. Installing lib32-libnm might help.

Bundled libSDL2 (2.0.8, you think): 15fps
System libSDL2 (2.0.14): 15fps
Steam Runtime libSDL2 (should also be 2.0.14): 15fps

That all seems consistent, at least. Did you adjust the LD_PRELOAD paths to point to files that exist on your system? (You're not @dllu and your home directory is probably different!)

If you were using the right paths, then this would indicate that the different versions of libSDL2 are not the culprit.

You can confirm that the expected libraries are in use by inspecting /proc/$pid/maps, where $pid is the process ID of the main Metro Exodus process. It would be useful if you could attach/share them for comparison, or do a diff between them.

If you are able to get steam-native working, and it does have better performance like you suspect, then comparing its /proc/$pid/maps with each of the LD_PRELOAD setups might provide useful information to track down what is going wrong in the Steam Runtime case.

[edited] #411 (comment) has some more information about what you're looking for in /proc/$pid/maps.

[Termuellinator]

i adjusted the LD_PRELOADS to my user and my metro install dir, yes.
I'll see when i find the time to do the proc maps, hopefully during the weekend :)

[smcv]

What is meant to be happening is that in the common case where "most" of your libraries are newer on the host system than in the Steam Runtime, you will find that "most" of the libraries used by the game come from the host system, the same as steam-native; so the only practical differences between steam-runtime and steam-native for 64-bit games should be:

• the minority of libraries that exist in the Steam Runtime but not on your host system, either because the Steam Runtime version is an older SONAME or because your host system just doesn't have them at all
• libcurl, which is special-cased as a result of some unfortunate ABI decisions in 2007
• some slight differences in the search path order that might affect what happens if the game appends or prepends to the LD_LIBRARY_PATH, or if it has a RPATH or RUNPATH

Another thing that might be useful is to use objdump -T -x MetroExodus | less and see whether there is a RUNPATH and/or RPATH listed in the Dynamic Section.

[Termuellinator]

I got native working again with lib32-libnm and it still runs perfectly smooth with vsynced 120fps.

Unfortunately, the maps are too different to just do a diff and get something useful out of it - as i don't really know what to look for, i'll just attach both in full lenght:
procmaps_metro_native.log
procmaps_metro_runtime.log

objdump for both steam-native and steam-runtime show the same RUNPATH:

Dynamic Section:
  NEEDED               libPxFoundation_x64.so
  NEEDED               libPhysX3_x64.so
  NEEDED               libPhysX3Common_x64.so
  NEEDED               libPhysX3Cooking_x64.so
  NEEDED               libSDL2.so
  NEEDED               libsteam_api.so
  NEEDED               libnxpro.so
  NEEDED               librt.so.1
  NEEDED               libdl.so.2
  NEEDED               libm.so.6
  NEEDED               libstdc++.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libpthread.so.0
  NEEDED               libc.so.6
  RUNPATH              .
[...]

[smcv]

OK, that's really useful information. It's a RUNPATH and not the older/mostly-deprecated RPATH, which means LD_LIBRARY_PATH is treated as "more important".

From the /proc/*/maps it looks as though none of the libraries from the Steam Runtime are actually being used for this particular game, even with steam-runtime; the only significant difference for this particular game seems to be that steam-runtime has the side-effect of searching your system library directory before the RUNPATH, whereas steam-native does not.

Having a NEEDED on libSDL2.so is weird: SDL's SONAME is libSDL2-2.0.so.0. I think this means you will get different results depending on whether you have the libSDL2.so -> libSDL2-2.0.so.0 symbolic link in your system library directories. On Arch, library packages normally include that symlink, but on Debian- and Red-Hat-derived systems, it's normally in a separate -dev or -devel package. I'm guessing that the developers of Metro Exodus tested it on a non-developer Debian- or Red-Hat-derived system, so there was no libSDL2.so in the system-wide library directory.

You could try setting the launch options to:

LD_LIBRARY_PATH="$(pwd)${LD_LIBRARY_PATH+":$LD_LIBRARY_PATH"}" %command%

and running with steam-runtime. If I am understanding the situation correctly, then that will result in the bundled libSDL2.so being used.

[Termuellinator]

Yep, that was it - with that launch options it runs as well in steam-runtime as in steam-native 👍
I'll write their support to inform them about that, thanks for your investigation! :)

[smcv]

It's probably best if you ask the Metro Exodus developers to talk to Valve about the best way to solve this, either on this issue report (I can help) or privately (I will not be able to help unless their Valve contact brings me into the loop).

One way they could resolve this would be to write something like this into a script MetroExodus.sh or similar, and change the launch command to be MetroExodus.sh instead of directly launching the main MetroExodus executable:

#!/bin/sh
set -eu
export LD_LIBRARY_PATH="$(pwd)${LD_LIBRARY_PATH+":$LD_LIBRARY_PATH"}"
exec "$(pwd)/MetroExodus" "$@"

Another way to resolve this would be to change the name and SONAME of their (modified?) libSDL2.so to a name that will not conflict with a system copy, such as libSDL2-MetroExodus.so, then rebuild their Linux MetroExodus executable to have that name in its DT_NEEDED ELF header - similar to the way id Software renamed their slightly modified/extended version of SDL to libSDL-1.2.id.so.0 when they included it in Quake 4.

A third possible way to resolve this would be to describe what's different about their libSDL2.so, and figure out how to make the system copy of SDL2 equally fast.

[Termuellinator]

So, support said they forwarded it to the devs, lets see what comes out of it :)

[smcv]

With recent changes to our container runtime environment, you might find that selecting the Steam Linux Runtime compatibility tool is also a way to work around this.

[smcv]

So, support said they forwarded it to the devs, lets see what comes out of it :)

@Termuellinator, did you ever hear back from them? I ask because #674 is another "their bundled SDL vs. system/runtime SDL" issue.

[Termuellinator]

@smcv Nope, nothing after that. Of course that doesn't necessarily mean that nothing came from it.