OpenBoard on Wayland

At some point in time, X11 will no longer be maintained and will vanish from the Linux distributions. OpenBoard must be prepared for this.

There are however several issues people have created which use OpenBoard on Wayland. Problems range from crashes to the inability to create snapshots. Here I want to collect some problematic topics related to the way how Wayland operates, mainly for screenshots, screen mirroring and screen recording.

Screenshots

To put the content of a screen area in a QPixmap Qt provides two completely different ways:

• If the screen area is part of a widget of the application, then this widget can be rendered in a QPixmap by using a QPainter.
• For other cases, it is however also possible to retrieve the QScreen object representing a screen and call the grab() function on the screen.

The first method is independent of the windowing system and works in all cases. The second method however does not work with Wayland, because it disallows grabbing arbitrary screen areas.

The Wayland alternative is using the XDG Desktop Portal. Here a screenshot can be requested via DBus. The system implementation of the portal may notify the user and ask for permission before taking the snapshot. The result is then delivered as a file and the application is notified. This sounds practicable, but has several problems in practice:

• The implementation of the screenshot portal function is different for each desktop environment. So Gnome, KDE and all others have their own implementation, which might differ in user behavior. Whether and how the user is notified is the choice of the DE.
• The screenshot takes much longer than screen grabbing. On my system with KDE a single screenshot takes about 350ms.
• Taking a screenshot always involves recording the content of all connected monitors. The relevant part must then be cut out by the application. This makes taking screenshots not faster.
• At least on KDE the screenshot file is created in the Images folder and must be deleted afterwards. So each screenshot involves a disk operation.

So in the end this function might be useful for screenshots in Web and Desktop modes. But it is not applicable for screen mirroring in Desktop mode, as it is way too slow. For the same reason it is also impossible to use it for Podcast recording.

ScreenCast

But there is an alternative function in the XDG Desktop Portal: ScreenCast. This function is intended for screen sharing applications. It grabs a screen content as a video stream. Such a stream could be recorded in a file or replayed in a widget on another screen.

But my first tests revealed other problems:

• The application cannot tell the portal which screen to record. Instead a dialog is presented to the user, where they have to select the proper monitor or application for sharing. This dialog is not intuitive for an OpenBoard user and it is likely that the user selects the wrong option.
• At least in my first tests this dialog was somehow hidden behind the mask when I was using it in Desktop mode, so there was no way to interact with the dialog. Further tests revealed that we have to hide the desktop annotation widget before calling Start on the ScreenCast portal and show it again afterwards.
  Edit Problem solved as described.
• Sometimes we get a response from the SelectSources call, sometimes we don't. FlameShot already has a comment on that (Hahaha, ist true und geht nicht). Have to work on that - or try some workarounds.
  Edit Problem was that the responses arrived faster than we could connect to the DBus to listen for them. Solution is to connect to the DBus and listen for the response before we make the call. This is possible as we can anticipate the path of the response.

Portal implementations

As already stated above, the XDG Desktop Portal is just a thin layer which relies on implementations provided by the individual desktop environments. When looking at the bug trackers of both Gnome and KDE it seems that there are sill many unresolved problems.

The different implementations also increase the testing effort necessary for OpenBoard. We have to test at least with Gnome and KDE, but probably also with others. I can also imagine that we get bug reports from users using another desktop environment and experience different behavior. This makes life for an application developer not easier.

Consequences

In general we have to state that using the XDG Desktop Portal may involve user interactions which cannot be influenced or inhibited by the application. These interactions even depend on the desktop environment. So what can we do?

• We can try to explain to the user what they have to do when such a dialog pops up. Still, these dialogs are problematic for the user experience.
• We can remove functions from OpenBoard which require these things. This would especially be screen mirroring in Desktop mode and the Podcast function.
• We should further investigate what "persistence" means for such a selection. The documentation is not clear about that. At least it can be for one session, so the user only has to do that once after OpenBoard was started.
  Edit In a screencast session we get a token which can be used in the next session to skip the dialog providing the screen configuration has not changed. I think this provides a good enough user experience.
• We should shift from ffmpeg to GStreamer for the podcast feature. We already have to use GStreamer for Desktop mode screen mirroring, so it is logical to use that for podcast, too. And in fact, a UBGstVideoEncoder is much less code than the current UBFFmpegVideoEncoder, as GStreamer provides a slightly higher level API, but still with full control over the encoding process.