RBDOOM-3-BFG 1.5.0

29 April 2023 - RBDOOM-3-BFG 1.5.0 adds DX12, Vulkan and glTF 2 Support

So over 600 commits later and probably over 1200 hours provided by 4 programmers RBDOOM-3-BFG 1.5.0 is final after one year of development.

This update is huge and replaces OpenGL with DX12 and Vulkan using Nvidia's NVRHI API and adds support for glTF 2 models and animations.

DX12 & Vulkan through NVRHI

OpenGL is dead and won't be further developed. OpenGL was holding back the development of this source port for quite some time and all the HLSL -> GLSL translation code with different outputs for Windows, Mesa and Vulkan GLSL was a time eater.

I was excited when Vulkan was announced but it wasn't fun at all to work with because it is way too verbose and difficult to work with even after 17 years of OpenGL experience. It was even annoying to just render anything offscreen like shadow maps. Right now, Vulkan offers many extensions to make those tasks simpler but if you are using modern Vulkan then it is already so different that people demand to make a new Vulkan 2 API.
Vulkan became immediatly the extension hell OpenGL was and all I wanted was a nice and simple API like OpenGL ES 3 but with proper error feedback and more speed.

I was looking for several years for a good abstraction layer of modern graphics APIs like DX12 and Vulkan. I considered The-Forge, Dilligent Engine, Tiny Renderers (Google) or BGFX for reducing the maintencance costs of the RBDOOM-3-BFG renderer.

Then I stumbled over an excellent article by Alexey Panteleev about Writing Portable Rendering Code with NVRHI.

NVRHI (NVIDIA Rendering Hardware Interface) is a library that implements a common abstraction layer over multiple graphics APIs (GAPIs): Direct3D 11, Direct3D 12, and Vulkan 1.2. It works on Windows (x64 and ARM64) and Linux (x64 and ARM64) and macOS through MoltenVK. It is not tied to Nvidia GPUs but also works with AMD and Intel drivers.

I suggested it in Discord and that we should go into that direction if we move the renderer to an abstraction library. Stephen Pridham took the initiative and wrote the initial port to NVRHI. Porting a game like Doom 3 BFG to another render API is not trivial.
It also changed my plans for 2022 and ruined many weekends by fixing renderer related bugs.
However I want to continue to work on this project in my spare time for the next 10 years so the effort was worth it. In the end it turned out that NVRHI is as easy to use as OpenGL with the Donut samples provided.
There is definitely some lack of documentation but now you have a former AAA game rendered with NVRHI as an example.
RBDOOM-3-BFG has a similar renderer complexity to older Call of Duty titles or the Source engine.
The implementation is not optimal in terms of performance and very close to the original rendering architecture of Doom 3 BFG.
However it is nice and simple that way.

Stephen Pridham did the major porting effort and was a big help when addressing certain difficult issues.
Stephen Saunders helped out with many Vulkan related rendering bugs. He also got it working with MoltenVk on macOS and improved support for AMD Vulkan users by integrating AMD's Vulkan Memory Allocator (VMA). He also contributed many other smaller bugfixes and improvements. Big thanks for both guys for all the help!
Also thanks to Alexey Panteleev at Nvidia who helped us through Discord and by pointing us into the right directions.

NVRHI will be also the gate opener for cross platform hardware accelerated Raytracing in the future so it is overall a big win for this source port.

There are also some immediate advantages for RBDOOM-3-BFG besides the smaller renderer backend.

DX12 is faster than OpenGL

This is actually one of the most demananding scenes in Doom 3 with many shadow casting point lights. DX12 is now almost twice as fast as OpenGL. Vulkan is still lacking behind but DX12 is the primary API for Windows users which are the vast majority.
Stephen Saunders also provided triple buffering with NVRHI wich leads to a higher parallelization between the CPU and graphics card.

Temporal Anti Aliasing

MSAA/SMAA have been replaced with a Temporal Anti Aliasing solution by Nvidia. This not only fixes geometric aliasing at the edges but also shader based aliasing within the geometries like extreme specular highlights caused by the new PBR shaders.

New SSAO

The SSAO implementation has been updated with the latest code from Nvidia's Donut samples which outperforms the old implementation and also adds better stability on different view angles.
The old SSAO in 1.4.0 also lacked filtering and looked very noisy. The TAA would help but now it's really fixed and better.
It's hard to tell in these screenshots but the new SSAO gives most scenes a lot more depth.

glTF 2 Support

Harrie van Ginneken contributed a new glTF 2 loader written from scratch using the awesome id Tech 4 idlib framework.
Also a big thanks here because this is huge and took a lot of time to develop.

The importer is not designed to work with all glTF 2 models you can find on the internet but it serves as a solid import facility from Blender. It can read ASCII .gltf and binary .glb models. It is recommend to use .glb for faster load times.
Models imported as .glb will be cached as .bglb models in the generated folder like all other model formats.
Cool about the current implementation is that animations are converted straight to the .bMD5Anim binary format of the Doom 3 BFG edition. This makes it a very tight integration into the existing architecture.

The basic idea about the glTF 2 loader is to make it easier getting models into the engine without any third party plugins in Blender. Another big disadvantage of .md5mesh exporters is that the format simply lacks support for normals.
So all normalmaps baked in Blender or Substance Painter using the Mikktspace algorithm would look wrong.
glTF 2 fixes this problem and it not only has support for normals but also tangents and those have to be calculated the Mikkelsen way so that's a perfect fit.

You can also use .glb files for static models without a skeleton just like with the original 3ds Max .ase and Lightwave .lwo files.
On top of that you can also use glTF 2 for new camera animations but that needs to be covered in an upcoming tutorial.

I also want to go back and keep things simple so I don't want to import and configure a model like in Unity or Unreal. I just want to load models and use them and the maximum should be to specify which frame plays a foot sound or which bone needs to be used to attach an item to it.

However it might be the case that someone exports a model from Blender and in a team environment someone else needs an option to load it differently. E.g. the front in Blender is -Y and forward in Doom is +X. So we need to rotate a model by 90 degrees around Z axis. All these things did the Maya .mb -> .md5mesh converter plugin for Doom 3 and we allow similar options in model declarations.

model character_mcneil
{
  // scale from meters to Doom units (inches), add an "origin" bone from rig origin to pelvis and 
  // build automatically a root motion from the pelvis bone
	options -scale 39.37 -addorigin -transfermotion "pelvis"
	
	mesh models/characters/mcneil/mcneil.glb
	
	channel legs ( origin *pelvis *foot_l *foot_r)
	
	anim walk models/characters/mcneil/mcneil.Walk.glb
	anim idle models/characters/mcneil/mcneil.Idle.glb
}

Supported options are:

• -armature "armatureName" if a glTF 2 file has more than one skeleton

• -reorient pitch, yaw, roll

• -rotate yaw

• -transferMotion "boneName" to enable root animations like in Unity

• -addorigin so we don't need put that extra "origin" bone in every Armature and can use common rigs like Mixamo or Rigify

• -scale factor

• -nomikktspace, specify if you want to use old normalmaps from an original Doom 3 model

NOTE: Material names in Blender need to be 1:1 the material names defined in the corresponding materials/*.mtr files.

Models are expected to be exported by Blender with the following settings:

Format: glTF Binary (.glb)
Transform: +Y Up On
Materials: Export
Images: None
Compression: Off
Data: Custom Properties On
Animation: Animation On, Shape Keys Off, Skinning On

glTF 2 dmap Support and Blender Mapping Workflow

I rewrote dmap 8 years ago to compile a BSP from a polygon based map format instead of brushes.
Now, Harrie van Ginneken has extended it so we can compile glTF 2 files.
The trick is to use custom properties in Blender and the engine treats the .glb file like a .map file.

There is too much to explain here and this needs to be covered in a tutorial series along with a new Blender addon for editing the entities. But this is quite cool. You can now make maps without TrenchBroom or a Radiant based editor. Theoretically it is possible to use any 3D modelling package as a level editor now.

Just save your map as .glb file and compile it like:

dmap mymap.glb

Harrie managed to port entire asset packs with animations and hundreds of meshes by using a few clever Python scripts within Blender.

Here is a small demo of his WIP game which shows GPU skinning using Mixamo animations and a working BSP with typical Doom 3 style portal culling:

Here is an another example map by Dmitri Engman that was done entirely in Blender. Kudos to him for pulling this off without documentation.

He also wrote a tutorial here: https://modwiki.dhewm3.org/RBDoom3BFG-Blender-Mapping#2-BLENDER-SET-UP

Imgui Articulated Figure Editor

Stephen Pridham added a new Imgui based Articulated Figure editor.

Imgui Light Editor

The ImGui light editor, which you can spawn with editLights, has been slightly improved.
It shows the bounding boxes of the lights and you can move the lights with the arrow keys similar to TrenchBroom.

Support for OGG Vorbis Soundtracks

Like in modern Quake sourceports users can now just throw their favorite soundtrack into base/music/ like base/music/aqm/*.ogg or into a modfolder/music/ path.
The game will play automatically it in the background as a looping track until the end of the map.
Each map will pick a different track but mappers can also define a "music" track in the worldspawn entity.
You can also fine tune the volume of each track if you write a sound shader for your files.

There is an example in mod_alternativequakemusic/sound/_rbdoom_aqm_tracks.sndshd for the Alternative Quake Music sound track.

Optick Profiler

The super lightweight C++ profiler Optick has been integrated. You need to recompile the engine with the CMake option -DOPTICK=ON but then you can run the profiler along with RBDOOM-3-BFG and make detailed captures about what is going on.
It's also very useful if you want to understand the multithreaded architecture of the BFG edition.

Borderless Fullscreen

Stephen Saunders contributed support for borderlesss fullscreen mode which can also be set by commandline with r_fullscreen -2. It's enabled automatically if you switch from windowed mode to fullscreen by using Alt + Enter. You can’t tell the difference visually. However, stick to borderless mode over fullscreen mode if you use multiple monitors.
You can access your displays and launch various apps while your game or work stays on the main display.

This has the advantage that you can run RBDOOM-3-BFG simultanously along with TrenchBroomBFG, Blender, Krita or any other gamedev weapon of choice and it makes it easier to to switch between the applications.

But, there is a downside to borderless mode, depending on your device.
Most operating systems like Windows, for example, will continue dedicating resources to all background processes.
This means you might get less performance and experience input lag.
If you need to squeeze some extra FPS, go back to real fullscreen mode to see if it makes a difference.

TrenchBroomBFG

The custom TrenchBroom build has been renamed to TrenchBroomBFG. This avoids conflicts with the original program like config files that are saved in the user directory. It also ignores all the other unsupported game profiles.
The new version has been updated to TrenchBroom 2023.1. It also supports the glTF 2 formats through the Assimp library but in this version the material loader has been tweaked to work directly with Doom 3 .mtr materials.
The icon size option also affects the entity inspector now which helps with high resolutions.

The engine command exportFGD was changed so it generates more variations of FGD files:

DOOM-3-all-and-models.fgd
DOOM-3-all.fgd
DOOM-3-multiplayer.fgd
DOOM-3-slim-and-models.fgd
DOOM-3-slim.fgd

If you switch to one of those with the additional autogenerated model entities then you can drag n drop meshes into your map without the models zoomap.

Summarized Changelog

[RENDERING]

• Renderer uses DX12 & Vulkan instead of OpenGL

• Shaders are not compiled at runtime anymore. They are compiled in advance by CMake using the DXC shader compiler and distributed in binary form under base/renderprogs2/dxil/*.bin

• All shaders have been rewritten to proper HLSL

• MSAA/SMAA have been replaced with a temporal anti aliasing (TAA) solution by Nvidia. This not only fixes geometric aliasing but also shader based aliasing like extreme specular highlights by the PBR shaders.

• Tone mapping has been replaced using new shaders by Nvidia

• Removed stencil shadow volumes with additional fixes by icecoldduke

• Integrate the AMD's VMA allocator for images and buffer objects

• Shadow mapping uses a fat shadowmap atlas instead of switching between shadowmap buffers and the HDR render target for each light. This improves the speed of the offscreen shadowmap generation.

• Blend shadowmaps smoothly in for small distant lights

• Skip shadowmaps for small distant lights

• Added menu option to choose between DX12 and Vulkan

• Detect displays that only support 144, 165 and 240 Hz

• Show picked GPU with com_showFPS > 1

• Went back to 15 bits for VERTCACHE_FRAME_MASK to avoid weird model distortions in a single frame every 68 seconds

• Reverted com_smp back to CVAR_BOOL

• Removed obsolete cvar r_useHDR

• Removed obsolete cvar r_useSeamlessCubeMap

• Removed unused cvar r_useHalfLambert

• Show shadow atlas stats with com_showFPS 3

• Enable DX12 / Vulkan debug layers with r_useValidationLayers 2

[DMAP]

• Dmap: always write a .cm file, especially when overwriting from a mod dir

• Support the Valve 220 texture projection in MapPolygonMesh::ConvertFromBrush()

• Automatically remove map collision .cm, .bcm files before running dmap

• Changed dmap to support compiling maps from glTF 2 .glb models instead of .map files

• Fixed AAS compiler to work with maps compiled from glTF 2 .glb files

• Tweaked dmap -glview option to dump an .obj next to the .proc file with similar content and to print an ASCII art BSP tree in the proc file

[MISCELLANEOUS]

• Don't generate collision models for every rendermodel in advance. Restored vanilla Doom 3 behaviour.

• Crashfix: Don't refesh the screen using prints during engine shutdown

• Print engine version when starting to write a qconsole.log

• Improved Quake .map converter to get Makkon's samplemaps working

• Bumped the required C++ standard to C++14

• Add move semantics to idList and idStr (thanks Admer)

  • idListArrayResize uses std::move
  • idStr implements move constructor
  • and move operator
  • mpMap_t also implements a move operator

• Fix snprintf() buffer length issues for Doom Classic on Linux with GCC 12 compiler

• Fix ImGui include path handling to be consistent with other external libs

• Applied MD3 crashfix by Daniel Gibson

• Restore support for FFMPEG legacy channel_layout & channels fields, remove self-assignment in idSoundDecoder_Vorbis::Open()

• Update FFMPEG deprecated calls to use supported ch_layout structure

• R_TestVideo_f command fixes and Cinematics startup fix for NVRHI (BinkDec green frame)

• Add s_playCinematicAudio cvar to enable/disable cinematic audio playback

• Support mp4/webm/mkv cinematics for mods, fix duration calculation, fix testVideo shutdown on exit

• Tweaked exportFGD to output more FGDs with autogenerated model definitions

• Fixed missing shadows in the kitchen of Mars City 1

• Fixed fake parallel light in Mars City 1 opening scene

• Integrated Optick profiler. Needs CMake option -DOPTICK=ON

• LightEditor: Draw volume of selected light and move it with arrow keys

• Support Borderless Fullscreen and Borderless Window modes for Win32 and SDL platforms

• Fixed crash when executing extract_resources.cfg on Linux

[TRENCHBROOMBFG]

• Renamed custom TrenchBroom fork to TrenchBroomBFG

• Change TrenchBroomBFG to only load Doom 3 related game profiles as it breaks compatibility to other Quake based games

• TrenchBroomBFG code was updated with the stable TrenchBroom 2023.1 release which contains tons of bugfixes and improvements

• If you turn off certain brushes like visportals in the View Options you won't accidently select by clicking to brushes behind them

• TrenchBroomBFG uses the Assimp library for glTF2 support

• func_door FGD fix

• Use Icon size preferences not only in the textures browser but also in the entity browser

• FGD proxymodel evaluation fix #4052

• Use fixed size (-8 -8 0, 8 8 16) bounds for all model based entities

• Fixed missing support for the basecolormap keyword in .mtr files