generate_emu_config overhaul (a lot of bugfixes, improvements, new features, new file structure, new helper tools)

[alex47exe]

• [alex47exe] major overhaul of generate_emu_config - custom configs, proper ini parsing, better logging and error handling, helper tools:

  • add -def1 ... -def5 arguments, which can be used to generate your preferred custom config
    if no -def argument is provided, -def1 will be used by default, to automatically copy from the following folders:

    • .\_DEFAULT\0 ............... essential emu files, like latest GSE dlls (steam_api.dll and steam_api64.dll)
    • .\_DEFAULT\1 ............... other GSE files and folders, including default ini files
    • .\_DEFAULT\<appid> ... other GSE files and folders, but only for the current <appid>, if the folder exists

  • (Windows only) add some useful helper tools, written in AutoIt3:

    • gse_acw_helper.exe - add the required achievements schema db files for Achievement Watcher, if .\steam_misc\extra_acw\extra_acw.zip file exists (if generated by generate_emu_config.exe -acw <appid>)
    • gse_debug_switch.exe - automatically switch between release and debug versions of the emulator, if steam_api.7z / steam_api64.7z file exists (or steamclient.7z / steamclient64.7z, if you use the steamclient version)
      paths to release and debug files inside 7z, can be customized in .\steam_misc\tools\au3\scripts\gse_debug_switch.ini
    • gse_generate_interfaces.exe - simple x64-x86 launcher for generate_interfaces.exe
      it also writes all found steam interfaces to CODEX steam_emu.ini (if generated by generate_emu_config -cdx <appid>)
      make sure to name your original dll to one of these formats, so it can automatically find its interfaces:
      • valve_api.dll / valve_api64.dll
      • steam_api.dll.bak / steam_api64.dll.bak or steam_api.dll.org / steam_api64.dll.org
      • steam_api.bak / steam_api64.bak or steam_api.org / steam_api64.org
      • steam_api_orig.dll / steam_api64_orig.dll or steam_api_legit.dll / steam_api64_legit.dll
    • gse_lobby_connect.exe - simple x64-x86 launcher for lobby_connect.exe

  • new folder structure, compatible with current and future helper tools --- default arguments are -acw -cdx -clr <appid>
    NEVER delete .\steam_misc\app_backup, .\steam_misc\app_info, .\steam_misc\tools and .\steam_settings folders
    MIGHT need .\steam_misc\extra_acw and .\steam_misc\extra_cdx for compatibility with Achievement Watcher and CODEX

  • add -scx argument to automatically download images / videos for trading cards, backgrounds, badges, emoticons and other tradable items
    unfortunately I couldn't find any direct steam api method to download the files, so I had to write a rudimentary web scrapper to extract the download links from a third-party website, hence the scx_gen.py script might need updating in the future if the website design changes

  • download screenshots and videos:

    • download thumbnails for both screenshots and videos, and compress them to .zip files
    • screenshots and videos are now numbered from first to last published, as in the Steam store page
    • add -vids_low / -vids_max arguments to download all videos, in low and / or high quality

  • create / update .\top_owners_ids.txt when .\top_owners_ids.html is present

  • generate controller action sets txt files for all found controller vdf configs, and zip them inside .\steam_misc\app_backup\app_backup.zip
    by default, the emu supports only xboxone and xbox360 controller configs, though if the're are any issues with the default supported controller action sets inside .\steam_settings\controller folder, you could try to unpack and overwrite action sets for other unsupported controller configs

  • (Windows only) add AdvancedRun launchers (cmd console + silent) for .bat files and .py scripts

• [alex47exe] major overhaul of migrate_gse - uses the same .\_DEFAULT\0 and .\_DEFAULT\1 folder structure for default configs
  it can convert old .txt format to .ini format, minus branches.json, which would require using top_owners_ids.txt and some login code from generate_emu_config, which should actually be used to properly generate the config files, instead of converting from the old .txt format

• [alex47exe] generate_interfaces.exe - find all Steam Interfaces instead of only old ones
  the emu will ignore the ones it doesn't require, while we'll have the complete list to write it to CODEX steam_emu.ini

• [alex47exe] lobby_connect.exe - improve cmd console text alignment

• [alex47exe] mods_img instead of mod_images (better folder consistency), better example for mods_img, minor tweaks to .ini and .md files

[alex47exe]

Not sure why it takes so long to build the experimental debug api here. Mine was done in less than 3 minutes:
https://github.com/alex47exe/gse_fork/actions/runs/10154145974/job/28078796689
Maybe you could try to re-run checks again. Or try a merge to a different branch, and run 'Prepare release' action manually.

[Detanup01]

Seems like github issue (I could merge without actually waiting to finish all builds btw) will reopen and probably request some changes (mainly in the ini what should be default) not today but tomorrow.

[alex47exe]

Sure, any suggestions are welcome! I just copy pasted the custom ini files included in default configuration for generate_emu_config, but forgot to restore the original values for the example ini files.
I guess the whole update should've been split in more commits, but it has been cooking offline for the past 3 weeks or so, without using git to properly commit and document all changes and iterations.

[Detanup01]

:D no worries

[Detanup01]

tools/generate_emu_config/generate_emu_config.py_ini.exe
tools/generate_emu_config/generate_emu_config.*.exe
why?
Also the default plus zips are probably not needed we can let users place steamapi dlls there.
also 7z exists inside the third-party folder

[alex47exe]

___ The exe files are basically just stripped down (no interface) AdvancedRun executables plus their required cfg (commented / documented by me), basically a better method to run various scripts and apps with parameters (including python), with or without interface if required (useful when you need to run an app or script silently -no cmd window flash even-, e.g. when we might need to integrate generate_emu_config into any other automated tool). To be honest, I'm surprised Microsoft doesn't include AdvancedRun by default in an app bundle (and properly integrated with context menu), like they do with Sysinternals Suite or PowerToys, 'cause I really find it the best method to easily run apps and scripts with parameters. Anyway, about those exes:

• *py_ini.exe - only required to run once, installs required python packages to allow running python generate_emu_config.py directly, useful during development
• *py_anon.exe - runs generate_emu_config.py script (needs python installed) in anonymous mode, doesn't fetch achievements, stats and maybe some other things too
• *py_user.exe - runs generate_emu_config.py script (needs python installed) with your own user account and password, to fetch the complete configuration for the game
• *exe_anon.exe (in post_build folder) - runs compiled generate_emu_config in anonymous mode, no need to have an *exe_ini.exe, as all required packages are included in the compiled exe
• *exe_user.exe (in post_build folder) - runs compiled generate_emu_config with your own user account and password, no need to have an *exe_ini.exe, as all required packages are included in the compiled exe

___ If you are referring to steam_api.7z and steam_api64.7z, they include both release and debug versions of the emulator, needed for gse_debug_switch.exe - I think one double-click to run it is a much superior method of changing between release and debug versions when needed, instead of having to manually backup and copy paste them

___ 7za.exe is currently required by gse_debug_switch.exe and gse_acw_helper.exe, and might be required by future tools I currently have some ideas for. I could've used an AutoIt3 UDF script (user defined functions) for zip compression / decompression, but I prefer 7z. Though I used zip compression in generate_emu_config.py for the app_backup folder, screenshots and trailers thumbnails, and parsed download links for SCX scrapper. What I could do is intergrate 7za.exe in a3x script, and unpack it when the script runs, but that would mean every other future a3x script would have to integrate it as well, in the end taking more hdd space than the current method.

[Detanup01]

For 7z:
if you pull eveything from here and even the submodules, you find out the
third-party/deps/linux
third-party/deps/windows
already contains your 7zip.

For the steam apis 7z:
You cannot make push to update because it will always be behind it.

Also probably could make a github action for building the tools rather than having the exe inside there (which I dont like it.)

[alex47exe]

Regarding steam apis 7z, the way I handled them was to manually build the emu on windows, then manually create the steam apis 7z containing both release and debug versions, then push the whole github folder using GitExtensions. Sure, I would love to automate this, but unfortunately I am not experienced with github actions. I guess it would basically mean that the generate_emu_config build would have to be delayed after the emu build, so that the compiled steam apis can be copied and archived to steam apis 7z first.
Same for the already included 7za.exe (though I am not sure why it's an issue) and compiled a3x scripts, both could be achieved automatically through some github actions and / or bat files. So we would need a github action (or modifications to one) to automatically install latest 7zip and AutoIt3, then add 7za.exe to the correct folder, and also build the au3 scripts with Aut2Exe.exe, as explained here: https://www.autoitscript.com/autoit3/docs/intro/compiler.htm
Finally, after all these actions complete successfully, we can build the generate_emu_config executable.

[rcyggdra]

Seems like github issue (I could merge without actually waiting to finish all builds btw) will reopen and probably request some changes (mainly in the ini what should be default) not today but tomorrow.

For familiar people, this PR appears bloated.
If you want to merge this PR, I hope it will be on the new branch instead of replacing the existing one.

[alex47exe]

For familiar people, this PR appears bloated. If you want to merge this PR, I hope it will be on the new branch instead of replacing the existing one.

What do you mean 'bloated'? Sure, maybe it should have been split in more commits, as I said above, but other than that, it only improves generate_config_emu and migrate_gse (though currently it can't generate branches.json, so I don't recommend using it - just regenerate the entire config instead), adds much needed helper tools for automation of various tasks, and corrects spelling mistakes / adds minor changes to wording and punctuation to some readme files. That's literally all it does. It's not 'bloated', it's improved - you just haven't tested it thoroughly to understand it.

[alex47exe]

Just reopened it to be able to fix conflicts with upstream fork, so I can correctly merge the latest changes into my own

[God-damnit-all]

The changes make -rel (formerly -reldir) not work correctly, at least not with the compiled exe.

Traceback (most recent call last):
  File "generate_emu_config.py", line 1348, in <module>
  File "generate_emu_config.py", line 835, in main
  File "shutil.py", line 598, in copytree
FileNotFoundError: [WinError 3] The system cannot find the path specified: '_DEFAULT\\0'
``

[alex47exe]

Just tested with a few app ids and -rel doesn't error out for me, though it also doesn't make a difference in the output files. To be honest, I haven't really used that (or -reldir, how it was named before), as the generator uses a totally different folder structure in order to support all improved features and helper tools. I was mostly focused on improving and fixing the generator as far as I could...
Not sure what it was actually supposed to do, as it's not so well explained...

• generate the entire config in same folder as the generator?
• expect input files... what input files? the generator doesn't need any file or folder parameter.
• all the files generated (aka the complete game configuration for steam_api dll folder) are already relative to the generator's working folder

Since it seems to be useless for the current version, I'll probably remove it, unless someone can say what it's needed for.

[God-damnit-all]

Just tested with a few app ids and -rel doesn't error out for me, though it also doesn't make a difference in the output files. To be honest, I haven't really used that (or -reldir, how it was named before), as the generator uses a totally different folder structure in order to support all improved features and helper tools. I was mostly focused on improving and fixing the generator as far as I could... Not sure what it was actually supposed to do, as it's not so well explained...

• generate the entire config in same folder as the generator?
• expect input files... what input files? the generator doesn't need any file or folder parameter.
• all the files generated (aka the complete game configuration for steam_api dll folder) are already relative to the generator's working folder

Since it seems to be useless for the current version, I'll probably remove it, unless someone can say what it's needed for.

Please don't do that, I use it. I'm the one who requested it to be added in the first place in the previous iteration of gbe_fork. It makes it so the working directory isn't based on the binary's directory. So try to run it from one directory up, e.g. ./generate_emu_config/generate_emu_config

[alex47exe]

Please don't do that, I use it. I'm the one who requested it to be added in the first place in the previous iteration of gbe_fork. It makes it so the working directory isn't based on the binary's directory. So try to run it from one directory up, e.g. ./generate_emu_config/generate_emu_config

I see now... let me know if I understand what you're trying to do:

• run the generator through a bat or external tool from another folder, and output the complete config to it instead of next to generate_emu_config.exe
• do you want to still output it in a _OUTPUT<appid> folder or directly next to the new bat or external tool calling the generator?
• do you want to still copy the _DEFAULT\0 and _DEFAULT\1 (or 2-5) template configs from the generator folder or from the new one, where your bat or external tool is located, possibly overriding the previous _DEFAULT template configs?

The script will need a few modifications to properly use -rel argument

[God-damnit-all]

Please don't do that, I use it. I'm the one who requested it to be added in the first place in the previous iteration of gbe_fork. It makes it so the working directory isn't based on the binary's directory. So try to run it from one directory up, e.g. ./generate_emu_config/generate_emu_config

I see now... let me know if I understand what you're trying to do:

• run the generator through a bat or external tool from another folder, and output the complete config to it instead of next to generate_emu_config.exe
• do you want to still output it in a _OUTPUT folder or directly next to the new bat or external tool calling the generator?
• do you want to still copy the _DEFAULT\0 and _DEFAULT\1 (or 2-5) template configs from the generator folder or from the new one, where your bat or external tool is located, possibly overriding the previous _DEFAULT template configs?

The script will need a few modifications to properly use -rel argument

Basically I have everything that the generate_emu_confg archive comes with independently existing in another folder for ease of upgrading, while my configs and outputs are elsewhere because I'm calling all that from another folder.

So yes to the first two questions. Indifferent to the last one.

[alex47exe]

@God-damnit-all, take a look at the latest release and let me know if it works correctly with the new -rel_out and -rel_raw arguments
https://github.com/alex47exe/gse_fork/releases/tag/2024_10_21

[God-damnit-all]

@alex47exe They work, but I'd also used -reldir to make it use the working directory's my_login.txt file.

[alex47exe]

https://github.com/alex47exe/gse_fork/releases/tag/2024_11_12

• add latest changes from Detanup01's gbe_fork up to this commit
  • add 'appid_finder' mode for 'generate_emu_config':
  • runs with only '-find' and '-rel_raw' parameters, automatically finds appid by searching game folder name on google
  • then it starts a simple external interface (written in AutoIt3) to check if it found the correct appid
  • if it hasn't found the correct appid you can browse left or right to see other possible matches, or write the game name under the game image, then press 'NO, RETRY' - you can do this as many thimes as you like...
  • once you found the correct appid, just press 'YES, CONTINUE' and generate_emu_config will start without asking you for appid
  • for now you can test 'appid_finder' byt running 'appid_finder.bat' - it will later be properly integrated to be able to do it all in one go, though it's already possible if you copy the contents of 'appid_finder' folder to your game folder, then edit the two bat files with the correct generate_emu_config location

Building it on github will fail, unless you also modify the github workflows as those on my fork, to only start generate_emu_config build after emulator build completes.