steamcompmgr hacks unnecessarily shortcut window decoration drawing code in "emulate a virtual desktop" mode

[ticky]

Proton Wine includes some hacks specific to the steamcompmgr window manager, for turning off window decorations. This is obviously reasonable and useful for several reasons which aren't pertinent to this issue.

One thing that this gets in the way of is Wine's "emulate a virtual desktop" mode, wherein Wine will create a single window into which it draws all running applications' windows. It's most important for older Windows 3/95/98 era games (and, of course, productivity software), where sometimes you had a separate window for a fancy new 3-D view and other controls ran in another window with basic GDI/winforms.

Wine's support for this is available graphically through winecfg or through winetricks/protontricks' vd=640x480 verbs. Both result in the same registry change which affects Wine's start-up behaviour.

Proton still supports this mode, and its Wine will gladly draw you a (delightfully windows 2000-coloured) desktop and window decorations, but when run under steamcompmgr/gaming mode, Wine's window decorations are missing within Wine's desktop. Run under desktop mode, Wine's window decorations are drawn:

However within gaming mode, the window decorations are missing:

Both of these screenshots are of the same program running under the same current Proton Experimental build launched under stable Steam, the only difference is launching in desktop vs gaming mode.

This is due to these checks (collapsed for brevity)

wine/dlls/user32/nonclient.c

Lines 64 to 65 in bdb21c7

	if (__wine_get_window_manager() == WINE_WM_X11_STEAMCOMPMGR && !((style & WS_POPUP) && (exStyle & WS_EX_TOOLWINDOW)))
	return;

wine/dlls/user32/nonclient.c

Lines 364 to 365 in bdb21c7

	if (__wine_get_window_manager() == WINE_WM_X11_STEAMCOMPMGR && !((style & WS_POPUP) && (exStyle & WS_EX_TOOLWINDOW)))
	return 0;

wine/dlls/user32/winpos.c

Lines 1672 to 1679 in df982e6

	/* HACK: This code changes the window's size to fit the display. However,
	* some games (Bayonetta, Dragon's Dogma) will then have the incorrect
	* render size. So just let windows be too big to fit the display. */
	if (__wine_get_window_manager() != WINE_WM_X11_STEAMCOMPMGR)
	{
	winpos->cx = min( winpos->cx, info.ptMaxTrackSize.x );
	winpos->cy = min( winpos->cy, info.ptMaxTrackSize.y );
	}

wine/dlls/user32/win.c

Lines 1743 to 1750 in bdb21c7

	/* HACK: This code changes the window's size to fit the display. However,
	* some games (Bayonetta, Dragon's Dogma) will then have the incorrect
	* render size. So just let windows be too big to fit the display. */
	if (__wine_get_window_manager() != WINE_WM_X11_STEAMCOMPMGR)
	{
	cx = min( cx, info.ptMaxTrackSize.x );
	cy = min( cy, info.ptMaxTrackSize.y );
	}

In a build with the above shortcuts removed, the window decorations are available under gaming mode.

I would much prefer for this to be the default behaviour when in "emulate a virtual desktop" mode.

To that end, I investigated shortcutting the code which sets the Proton-specific window manager information in dlls/winex11.drv/window.c to skip setting the window manager hint if under desktop mode, but my approach of checking if the root window was the default root window (based on a similar check in desktop.c) was not successful:

diff --git a/dlls/winex11.drv/window.c b/dlls/winex11.drv/window.c
index 5d46391f2ce..3b7f862c14b 100644
--- a/dlls/winex11.drv/window.c
+++ b/dlls/winex11.drv/window.c
@@ -129,7 +129,7 @@ static int detect_wm(Display *dpy)
 
     static int cached = -1;
 
-    if(cached < 0){
+    if(cached < 0 && root_window == root){
 
         if (XGetWindowProperty( display, root, x11drv_atom(_NET_SUPPORTING_WM_CHECK), 0,
                                  sizeof(*wm_check)/sizeof(CARD32), False, x11drv_atom(WINDOW),

I believe this window code may be run once per window instead of once per desktop, defeating this approach.

My knowledge of the wine codebase is not deep, so this is as far as I've been able to get. I would welcome any input on how I might accomplish this while keeping the Proton-specific code in place for the intended effect in non-virtual desktop mode. Thank you!

[kakra]

I think this rather due to gamescope running the main wine window stretched/resized without decorations to scale games to fullscreen in gaming mode. It's probably completely expected due to how gamescope works, and that doesn't expect running a virtual wine desktop.

[ticky]

It's nothing to do with how gamescope itself treats the window; I've already identified the code paths where this is done, and it's all done on the Wine side when it sees it's running under gamescope.