[Draft] Linux Porting

[Yangff]

This is just a draft PR. No guarantee of its availability.
I've finally gathered all the necessary parts for the Linux port, even though there's still a lot that's unfinished, needs to be added (other Unreal Engine versions), and needs testing (hook parts, AOB scanning, string compatibility, and compatibility with WIN32 code, etc.).
Bringing up some of the issues encountered might be beneficial for Linux porting. Even if this PR doesn't get merged, it could provide some insights for future attempts.

As for UE version support for linux, I don't think it'd be meaningful to support any game < 5.1 as that is the first version epic really starts to ship pre-built binaries for Linux.. Although devs can do it on their own, I'd not expect new games will use such an old engine versions..

Changes

• Made some adjustments to CMake.

• Disabled GUI, Input, and other features under Linux, using stderr for output, especially since Input has a high degree of coupling with Windows. It might be more reasonable to inject into UE's own input module.

• Used LD_PRELOAD for loading and reading ELF segment information with dl_iterate_phdr under Linux.

• Modified a lot of code that depends on MSVC features, such as member function pointers. However, there are some issues that couldn't be addressed, for example:

MemberVariableLayout_HeaderWrapper_ULocalPlayer.hpp:4:54: error: non-defining declaration of enumeration with a fixed underlying type is only permitted as a standalone declaration [-Welaborated-enum-base]

(clang wants all enum used by template get at least a definition for its size)
and the TypeAccessor being used twice in the DECLARE_VIRTUAL_TYPE_BASE macro, I'm unsure of the intention here.
Also, FUNCDNAME in GetDispatchMap(), which I think could be replaced with typeid(), but I'm unsure of the consequences.
-fms-extension is requred and to compile this and so clang is a must. (which is fine I guess, since UE also used clang for its linux build)

• Modified GeneratedSourceFile template code to deal with a clang tuple bug Rejects-valid with variadic alias template in variadic class template llvm/llvm-project#17042 .

• Modified macros to suppress Windows-specific things like dllexport, which maybe should be controlled in CMake.

• Modified a lot of string-related macros and functions, distinguishing between UE's character types and the C/C++ runtime character types using UEStringType and SystemStringType. This is mainly because wchar_t is 32-bit on Linux, and the standard library only provides compatibility for std::string (utf8)/wstring (unicode), while compatibility for u16string is missing.
  I'm not satisfied with these changes, and my suggestion is maybe to introduce fmtlib, so u16string could be used globally, except for the Lua part, to minimize the parts that need encoding conversion.

• Used DWARF information to generate memory and virtual function layouts and related headers under Linux.

• Added AOBs and necessary xref/xcall features related to Linux binaries in patternsleuth because Linux uses non-pie binaries, leading to many addresses being 32-bit, and the Linux calling convention is different from Windows'.
  Covered most resolvers, the rest I couldn't find in my binary files.
  Since Linux's ELF doesn't load section information by default, used segment as a substitute.
  Under Linux, UE's ELF doesn't contain any internal symbols, so this part of the scanning was removed.

• Polyhook2 should be able to hook under Linux, but this hasn't been tested yet.

• Fixed throw crash problem with UE.. The default std::throw used by UE has a CLANGC++ symbol and will be used by us to throw to glibc's c++ runtime.

• I put all changes on this PR, although some of them are on other submoduels.

[narknon]

This is really cool but we are working on moving over to UE's platform system for generic platform strings, etc. So, a lot of your systemstring work will be covered by that.

[Buckminsterfullerene02]

We really appreciate the work you're doing for UE4SS and for Patternsleuth.

To keep you properly in the loop, you can find the UEPlatform stuff here. Additionally, you may benefit from taking a look at an existing attempt at a linux support which you can find here. Though as above you might need to have proper discussions with localcc/narknon regarding their specific plans with UEPlatform.

[Yangff]

This is really cool but we are working on moving over to UE's platform system for generic platform strings, etc. So, a lot of your systemstring work will be covered by that.

that's actually nice to move away from std::string. I'm not happy with the current string handling either.

[narknon]

Yeah, we're trying to move the UE submodule to be as close to UE as possible so we can take advantage of their abstractions for different platforms and other optimizations more easily. And it will make porting code for additional functionality much easier.

[Yangff]

Made some progress and this porting is now working on the palworld linux server at least for NotifyOnNewObject, RegisterHook, StaticFindObject and FindFirstOf .
It is still missing some functions signature that's scanned in C++ code, but should not be a big deal to add them into it..

I'll take a look at the UEPlatform branch to see how to integrate current code into there.

[narknon]

That's awesome! UEPlatform doesn't have the generic platform string abstractions yet but I'll prioritize that. I previously just tried bringing over the string stuff so hopefully I can find those files and it's an easy port now

[Yangff]

That's awesome! UEPlatform doesn't have the generic platform string abstractions yet but I'll prioritize that. I previously just tried bringing over the string stuff so hopefully I can find those files and it's an easy port now

So, I think for most code just need to unify the char to u16 instaed of wchar and make use of String:: like Format and maybe comparing etc in the case of containes..
But, I think even in this case, there are still some early output and use of string where before the UE get located, what is the plan for those? use system string (utf8 on linux and u32 on win?)

[narknon]

My plan is to basically use the exact systems UE uses to cross compile. The UE platform branch is essentially porting all of that over for the generic platform abstractions and the windows platform. After that's fully ported, adding additional platforms will be as simple as porting over UE's headers for that platform as far as making types compatible. Of course that won't cover scanning and hooking though.

[UE4SS]

Note for reviewers in the future: At one point in time, several workflow files were removed in this PR (b03265f) so if this PR makes it to the point where it's going to be merged, please verify that they still exist and work before merging.
Also be aware that we might run into githubs limitations on actions if we start running more CI (i.e. Linux build) for every commit.

[Yangff]

Note for reviewers in the future: At one point in time, several workflow files were removed in this PR (b03265f) so if this PR makes it to the point where it's going to be merged, please verify that they still exist and work before merging. Also be aware that we might run into githubs limitations on actions if we start running more CI (i.e. Linux build) for every commit.

I think I'll use a new branch when it get merged, because for example a lot of the changes to string handling here I think will be discarded or rework.

[Yangff]

That's awesome! UEPlatform doesn't have the generic platform string abstractions yet but I'll prioritize that. I previously just tried bringing over the string stuff so hopefully I can find those files and it's an easy port now

I'd like to make sure that the plan here is to copy the source from UE FString/FStringFormatter and compile as it, instead of trying to locate the address of FString::Format at the runtime? Also things like FFileHelper?

[UE4SS]

That's awesome! UEPlatform doesn't have the generic platform string abstractions yet but I'll prioritize that. I previously just tried bringing over the string stuff so hopefully I can find those files and it's an easy port now

I'd like to make sure that the plan here is to copy the source from UE FString/FStringFormatter and compile as it, instead of trying to locate the address of FString::Format at the runtime? Also things like FFileHelper?

Trying to locate the address of any function is a last resort, or if it's trivial, e.g. if it's a virtual function and we have handy access to an instance.
I think it's pretty safe to say that we shouldn't be trying to locate the address of FString::Format.

[Yangff]

I think multiple character types are still needed in every sense, with or without FString.
This is because of the following platform limitations:

1. UE will always use u16, however on Windows this is wchar_t/wstring, on Linux it's char16_t/u16string.
2. The wide character Windows API can only handle wchar_t and does not accept u16, even though they are essentially the same;
3. Linux can accept all kinds of u8;
4. wchat_t and char16_t can't be implicitly converted by the platform's compiler on Windows;
5. It's not possible to deal with UTF-8 directly through the Windows API (i.e., it's not possible to submit the correct string to the Windows api unless it only used as a binary stream (say writfilel) )
6. All local files are essentially utf8
7. c++ std::exceptions require char
8. There are no u16-related library functions, such as isdigit, on the Linux platform, although they are not difficult to implement.
   std::format also does not support u16 formatting on Linux, so either use all u8, or use the wide character version on Windows and the u8 version on Linux (as no reason to use u32 on Linux).

Therefore, regardless of the future of FString, we need to make a distinction between the following three character types.

1. FileString -> always u8
2. UEString -> always u16, u16string on Linux, wstring on Windows, used by UE, header and etc..
3. SystemString -> Unicode encoded string supported by the platform API, wstring on Windows, using utf16, string on Linux, using utf8, used by API interface.
4. std::string is used for interaction with the c++ api and for interaction with Lua, using utf8.

And for different modules:

1. Obviously all files will use FileString, and I'll keep UEString as a compatibility backup to the original wstring interface:
2. All interactions with UE will use UEString.
3. All interactions with Lua use string.
4. All interactions with the GUI use UEString.
5. SDK Generators and JSON parser etc.. uses SystemString...
6. SDK uses UEString
7. Output uses SystemString for format parameters and interoperability with devices, and can accept any type of string as input.
8. Parser ... ok.. still using SystemString because they're too rely on std::format and etc..

In general, the string flows in either direction of File::String (utf8) <---> SystemString (u8 or u16) <---> UEString (u16).

And in the end, I can only see FString as a replacement for wstring/u16string (UEString) here, assuming it can return correct wchar_t/char16_t for both platform, even.. still it requires extra works to integrate with the c++ std stuff.

If we can have a better string manupilation eco-system, replace whole SystemString to FString should be possible but may not be very necessary..

[Sasurtio]

@Yangff

Glad to see someone interested in linux port!

I tried to do some testing of this implementation but I got stucked. In case this is trivial to answer can you help me with it?

I'm already able to inject the program into the before launching a game (for example Palworld) using LD_PRELOAD=/path/to/libUE4SS.so /path/to/server/binary. The server boots up except that the program doesn't load any mod. It finds all the folders (root, game and mods) and even detects if there is a mods.txt or not. How did you managed to make the program load mods correctly?

My asumption is that I'm missing correct UE4SS-Signatures, because when I used them (different versions of windows) the program actually logged (after PS scan) a LUA scan attempt and then crashed with signal 11 (segmentation fault), and without the signatures this never happens, therefore, no mod is loaded.

Again, in case this is trivial to answer I'll appreciate any help on it, and in case its possible I can help with documentation of how to setup and load the program when this PR is ready to be merged (or before, if you want more testers to try it)

Thx for the amazing job done already!

[Yangff]

@Yangff

Glad to see someone interested in linux port!

I tried to do some testing of this implementation but I got stucked. In case this is trivial to answer can you help me with it?

I'm already able to inject the program into the before launching a game (for example Palworld) using LD_PRELOAD=/path/to/libUE4SS.so /path/to/server/binary. The server boots up except that the program doesn't load any mod. It finds all the folders (root, game and mods) and even detects if there is a mods.txt or not. How did you managed to make the program load mods correctly?

My asumption is that I'm missing correct UE4SS-Signatures, because when I used them (different versions of windows) the program actually logged (after PS scan) a LUA scan attempt and then crashed with signal 11 (segmentation fault), and without the signatures this never happens, therefore, no mod is loaded.

Again, in case this is trivial to answer I'll appreciate any help on it, and in case its possible I can help with documentation of how to setup and load the program when this PR is ready to be merged (or before, if you want more testers to try it)

Thx for the amazing job done already!

Can you send your directory tree and UE4SS.log under the same folder as the .so?
In genreal, I'd suggset you start from this simple hello world to see if it works.

print(string.format("Hello world from Lua Mod!!!"))

One thing to notice is the scripts folder needs to be all lowercase in order to work.

[Sasurtio]

Thx for the quick response!

The issue was with the "Scripts" folder, which I've created with capital case. Renaming to "scripts" did the trick, now the mods are loading correctly. (Thank you, Windows case insensitiveness)

Now that I can test this, do you want me to write a .md file with requirements (GLIBC >= 2.35, GLIBCXX >= 3.4.32, etc), how to upgrade stdc++ lib in case is needed and how to setup a script to run a game server with UE4SS injected?

This will bring more testers, and reveal any issue that is not already considered for linux implementation

I'm asking because this will attract (I hope) A LOT more people to try it while this MR is still in draft

[UE4SS]

Regarding file & path case sensitivity problems, should we maybe implement a case-insensitive function ?

Instead of just checking if the path exists, we'd have to iterate the path instead and case-insensitively compare each directory (and file when appropriate) in the path to the path (and/or file) we're looking for, and we'd need a counter that increases with every match so that we can generate a runtime error for the user to solve because if there are multiple directories with the same name but different casings then we can't possibly tell which one should be used.

Upsides:

1. Most of the time, this will just work for everyone, no need to think about the casing, and this also solves the enabled.txt/Enabled.txt problem.

Downsides:

1. It should be slower but do we do any path or file lookups in any hotpaths, does it matter if it's a bit slower ?
2. We'd effectively be reserving any directories with the names that we look for on Linux.
   Users wouldn't be able to have both a "scripts" and a "Scripts" directory or a "mods" and a "Mods" directory or whatever other directories that happens to be in the path for any reason.

[Yangff]

@Sasurtio Regarding the libc version, I think maybe the version requirement could be lowered to 2.17, but I'm not so sure that because ue4ss itself uses a lot of c++17/20 features, a lower version of glibc might not work correctly?
I don't recommend upgrading the system's glibc, if users are using a Linux distro with rolling upgrades like archlinux they should be well aware of such issues, and conversely, upgrading glibc on a distro with a fixed glibc version such as Ubuntu can lead to even more problems.
It might make more sense (as many Linux palworld servers currently do) to use docker/podman + mount an external mods directory to load it.

@UE4SS In the recent commit I tried to detect both scripts and Scripts, I guess arbitrary case combinations don't really make sense?
A similar problem occurs with lua looking for import files, I have it looking for scripts and Scripts.
I can only hope no one has written LogicMods as logicMods or something ......

I haven't changed the logic for enabled.txt yet, but I could similarly add a check for Enabled.txt.

As for the Mods directory itself, currently it will only check the Mods directory, and checking all mods directories might result in the user having two mods.txts in their respective directories? Seems like it would require changes to the loading logic and result in mods.txt being loaded in an uncontrollable order?
Considering that the Mods dir should be distributed with UE4SS, this doesn't seem to be a serious problem?

[Sasurtio]

@Yangff I tried with default libc in Ubuntu 20.04 (glibc 2.30) and it crashed with seg fault because of it. Also the process to recompile a higher glibc version and then patch the program to use it was not possible (I tried with patchelf) because the ".interp" property is missing in the loader, meaning it is trying to use a fixed interpreter.

Regarding libstdc++ there is no issue going up, but the glibc is another story for static libc distros.

[Yangff]

recompile a higher glibc version

You can install glibc to another location and execute its ld.so directly to bypass the interp, or modify the .interp of PalServer via patchelf.
Regarding lowering the version dependency of glibc, I meant to do it by linking https://github.com/wheybags/glibc_version_header at compile time, but I'm not sure if that would cause any other problems.

[UE4SS]

As for the Mods directory itself, currently it will only check the Mods directory, and checking all mods directories might result in the user having two mods.txts in their respective directories? Seems like it would require changes to the loading logic and result in mods.txt being loaded in an uncontrollable order? Considering that the Mods dir should be distributed with UE4SS, this doesn't seem to be a serious problem?

Avoiding the described problem should be trivial, either error if both Mods and mods exist or prioritize Mods, meaning ignore mods if both exist, but regardless, it shouldn't be a problem for the reason you stated.

The reason I prefer a more standardized way to deal with directories is so that devs don't have to consider both Windows & Linux for every single file and path, and for that same reason I think we should probably change to std::filesystem::path everywhere.
We could have our own path_exists function for example, and our File system could implicitly do the proper checks as well.
For example, instead of this:

current_paths.append(std::format(";{}" LUA_DIRSEP "{}" LUA_DIRSEP "scripts" LUA_DIRSEP "?.lua", to_string(m_program.get_mods_directory()).c_str(), to_string(get_name())));
current_paths.append(std::format(";{}" LUA_DIRSEP "{}" LUA_DIRSEP "Scripts" LUA_DIRSEP "?.lua", to_string(m_program.get_mods_directory()).c_str(), to_string(get_name())));

We could have something like this:

// Assuming 'get_mods_directory()' returns std::filesystem::path.
auto scripts_path = File::get_path_if_exists(m_program.get_mods_directory() / get_name() / "Scripts");
current_paths.append(scripts_path.string().c_str());

The new File::get_path_if_exists function would check for both Scripts and scripts return the correct one based on what exists, and it would probably throw or return an empty path or something if the path doesn't exist.
If you think doing a fully case-insensitive implementation isn't worth it, the abstraction itself can still be implemented with just the first character being changed, and both dirs being found doesn't have to be an error, it could just return whichever it finds first or whatever other priority you think is best.
If you were to open a file, we could also avoid having to explicitly call get_path_if_exists, and just do it inside File::open instead.

[UE4SS]

@Yangff Have you not been using clang-format ? I doesn't seem like your code is formatted properly.

[UE4SS]

What's the status of the GUI ?

1. Why is there a need to use ImTui instead of ImGui ?
2. We're already using GLFW for Windows, is there a reason we can't use it for Linux ?
3. Could it be feasible to use GLFW for input handling instead of our own custom solution ? That would be preferable since our own solution is not cross-platform compatible.
   I'm not sure how input handling is done in GLFW, if you can make use of it even while not rendering anything or if it's intrinsically linked to the UI in some way.

[narknon]

I think multiple character types are still needed in every sense, with or without FString.

@Yangff I will defer to you on how best to handle this, as it seems like you have a much better understanding of what would be required. The one thing I'd like to be kept in mind is that we want to move the UE module to be as close to UE code as possible, so where UE has a system or abstraction or optimization for different platforms, we'd like to use their systems if it will also work for our scenario.

My end goal is to have coding plugins for UE4SS as similar as possible to coding for UE while allowing them to be pseudo-universal across as many games/engine versions as possible. And I think for maintainability's sake, using as much of the UE code base as possible will make it much easier to update alongside UE. There are a lot more features in UE's Core engine source to move over and use.

[Yangff]

Have you not been using clang-format

I've not formated the code yet.

What's the status of the GUI ?

I do not have a desktop env for linux right now. So I'm using TUI to allow access to the UI interface without the need of having a desktop or so. It'd be more accessible when using ssh compared to X11-forwarding?

1. ImTui instead of ImGui

It's still using the ImGui for rendering, but just using text +necurs as its input/output deivice. Mostly just need to have a scale factor becuause in TUI everything take space line by line instead of pixel by pixel.
Unfortunately it has to modify part of imgui so far so it's using its own imgui.

GLFW

Rendering with GLFW under a real Linux desktop environment shouldn't be too difficult, but overall, the purpose is to make it more easy to use within the server environment as that seems to be the only motivated use case right now..

And at lesat on my wslg environment, I can't even run the example opengl application..(I mean likely it's because of some of my misconfiguration but I'm not using a lot wslg because it's buggy anyway)

3. use GLFW for input handling

The current input handle used by UE4SS is to get keyboard status directly from the Windows API. It's because for the gaming you want to get the key while the UE4SS Window is not on focus?
On linux it is kinda of tricky, you can interact with X say using X Record to get something like the windows api.. but I'm not sure how avialable is that if the wayland is used for the desktop which might have different result depends on what window management they're using.. I think the same situation for whatever underlaying API used by GLFW..
But for current linux user that is fine, as there will be no gaming window for dedicated server anyway.. but on Windows that will prevent the player from accessing the shotcuts when playing I think because GLFW will only subscribe to its own message queue.

I would leave enough abstraction space for the Input/Handler so that you can use any InputSource to add event sources to it.

You could have whatever InputSource running behind the scene and either pumping events into its queue or polling when the handler requested, and the input handler takes those inputs and act accordingly.

The benefit for that is, even if we're not using the TUI, you can still receive keyboard events from ssh to allow some functionality like reload mods or interact with lua mods.

There are a lot more features in UE's Core engine source to move over and use.

yeah, and I think switching to UEPlatform means at least to clean up all that code marked as using SystemString so far.. they're marked as SystemString mostly beucase they're using stl for string manupilation... like all the codes involve in the parser and header dumping..
Once FString::Format is ready and compatible with other STL container currently in use (which might need more work, or also replace them with UE's) this should not be too hard I think.

[UE4SS]

I do not have a desktop env for linux right now. So I'm using TUI to allow access to the UI interface without the need of having a desktop or so. It'd be more accessible when using ssh compared to X11-forwarding?

1. ImTui instead of ImGui

It's still using the ImGui for rendering, but just using text +necurs as its input/output deivice. Mostly just need to have a scale factor becuause in TUI everything take space line by line instead of pixel by pixel. Unfortunately it has to modify part of imgui so far so it's using its own imgui.

I didn't consider non-desktop environments but this sounds very cool.
As long as we don't have to duplicate any UI elements or specialize our code specifically for ImTui, I think it's fine if you intend to keep it in.

I would leave enough abstraction space for the Input/Handler so that you can use any InputSource to add event sources to it.

You could have whatever InputSource running behind the scene and either pumping events into its queue or polling when the handler requested, and the input handler takes those inputs and act accordingly.

The benefit for that is, even if we're not using the TUI, you can still receive keyboard events from ssh to allow some functionality like reload mods or interact with lua mods.

This sounds fine.

[Yangff]

Btw, I'm not sure https://github.com/UE4SS-RE/RE-UE4SS/blob/main/deps/first/Input/include/Input/Handler.hpp#L38 why the m_key_sets is defined as a vector of a map of a vector..
a map (map<key,vector <key data >> for example ) should be enough for mapping keys to the array of bind events on that keys with different modifier keys requirement?
It seems that in registration function it will create a new map for each key and push it to the vector which.. I'm not seeing any usage for it?

[UE4SS]

Btw, I'm not sure https://github.com/UE4SS-RE/RE-UE4SS/blob/main/deps/first/Input/include/Input/Handler.hpp#L38 why the m_key_sets is defined as a vector of a map of a vector.. a map (map<key,vector <key data >> for example ) should be enough for mapping keys to the array of bind events on that keys with different modifier keys requirement? It seems that in registration function it will create a new map for each key and push it to the vector which.. I'm not seeing any usage for it?

The input system was written years ago when I first started learning C++ and I no longer remember the details of it so I can't answer your question unfortunately, and the quality of that code is probably subpar in many ways.
It does seem to me that an unordered_map should be sufficient, but you'd have to test this yourself to make sure.

[Yangff]

rebased to main again, I will try to create PRs and see how it works..

[Yangff]

Try to PR some less platform-affected code first to avoid and overly long history that makes it impossible to ensure the correctness of the rebase..