HTML5 client image quality degrades during image segmentation within 3D Slicer

[jeffmax]

Versions

Xpra Version 6.2.3 on Ubuntu (also tested on Mint)
HTML 5 Client: v17 (latest) on Chrome (Version 132.0.6834.160 (Official Build) (x86_64) on OS X) (also tested in Mint in XFCE).

---

I am trying to annotate an image in 3D Slicer (open source medical imaging software). The application has an image mask annotation tool that puts a circle on the user's mouse to represent the mask drawing tool (like a Photoshop drawing tool). Within the HTML5 client, worse if the app is maximized to the screen, the circle lags behind the mouse movement significantly and the image that the user is annotating degrades in quality (almost as if it is being forced to redraw the whole image as you draw a mask over a small portion of it and running into bandwidth issues). The Python client does not have the same issue. Does anyone have an idea of what might be causing this issue only within the HTML5 client or suggestions for debugging further?

[totaam]

You can run with -d compress on the server side to see what type of screen update is being sent.
Perhaps it is switching to a video encoding too aggressively, this would match the "worse if the app is maximized".

[jeffmax]

Hi,

This is the -d compress output while the performance is degrading. I've set a content-type of "picture" for the application in 50_class.con but it hasn't seemed to matter. The performance is poor at very large screen sizes, but it is still laggier than usual at smaller display sizes. Thanks!

025-02-13 16:35:59,975 client display size is 3440x1440                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               │
│ 2025-02-13 16:35:59,975   Google Chrome 132 (910x381 mm - DPI: 96x96)                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  │
│ 2025-02-13 16:35:59,978 DPI set to 96 x 96                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             │
│ 2025-02-13 16:35:59,979 sent updated screen size to 1 clients: 3440x1440 (max 8192x4096)                                                                                                                                                                                                                                                                                                                                                                                                                                                               │
│ 2025-02-13 16:36:00,173 compress: 134.3ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   3.8%  (18934KB to   728KB), sequence     8, client_options={'rgb_format': 'BGRA', 'quality': 100, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 100, 'speed': 80, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229826.699753284, 'process-damage-time': 9229826.754898995, 'depth': 32})                                                   │
│ 2025-02-13 16:36:00,924 compress: 309.4ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   3.9%  (18934KB to   731KB), sequence     9, client_options={'rgb_format': 'BGRA', 'quality': 100, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 100, 'speed': 80, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229827.30083522, 'process-damage-time': 9229827.330558604, 'depth': 32})                                                    │
│ 2025-02-13 16:36:01,293 compress: 368.4ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   3.9%  (18934KB to   731KB), sequence    10, client_options={'rgb_format': 'BGRA', 'quality': 100, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 100, 'speed': 80, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229827.406027876, 'process-damage-time': 9229827.51666546, 'depth': 32})                                                    │
│ 2025-02-13 16:36:01,609 compress: 315.6ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   3.9%  (18934KB to   731KB), sequence    11, client_options={'rgb_format': 'BGRA', 'quality': 100, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 100, 'speed': 80, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229827.51864174, 'process-damage-time': 9229827.727829404, 'depth': 32})                                                    │
│ 2025-02-13 16:36:01,746 compress: 136.5ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   3.9%  (18934KB to   731KB), sequence    12, client_options={'rgb_format': 'BGRA', 'quality': 100, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 100, 'speed': 80, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229827.728201108, 'process-damage-time': 9229828.104618, 'depth': 32})                                                      │
│ 2025-02-13 16:36:01,925 compress: 162.2ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   0.6%  (18934KB to   108KB), sequence    13, client_options={'rgb_format': 'BGRA', 'subsampling': 'YUV420P', 'quality': 32, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 32, 'speed': 68, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229828.10497044, 'process-damage-time': 9229828.478835613, 'depth': 32})                            │
│ 2025-02-13 16:36:02,196 compress: 201.1ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   0.6%  (18934KB to   110KB), sequence    14, client_options={'rgb_format': 'BGRA', 'subsampling': 'YUV420P', 'quality': 33, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 33, 'speed': 68, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229828.479254443, 'process-damage-time': 9229828.710375872, 'depth': 32})                           │
│ 2025-02-13 16:36:02,398 compress: 186.9ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   0.6%  (18934KB to   108KB), sequence    15, client_options={'rgb_format': 'BGRA', 'subsampling': 'YUV420P', 'quality': 32, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 32, 'speed': 68, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229828.805737035, 'process-damage-time': 9229828.926884472, 'depth': 32})                           │
│ 2025-02-13 16:36:02,716 compress: 317.1ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   2.3%  (18934KB to   433KB), sequence    16, client_options={'rgb_format': 'BGRA', 'quality': 98, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 98, 'speed': 89, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229828.927325096, 'process-damage-time': 9229829.113837363, 'depth': 32})                                                     │
│ 2025-02-13 16:36:03,101 compress: 303.0ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   1.9%  (18934KB to   368KB), sequence    17, client_options={'rgb_format': 'BGRA', 'quality': 95, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 95, 'speed': 89, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229829.114803284, 'process-damage-time': 9229829.51357976, 'depth': 32})                                                      │
│ 2025-02-13 16:36:03,416 compress: 301.3ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   1.9%  (18934KB to   359KB), sequence    18, client_options={'rgb_format': 'BGRA', 'quality': 94, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 94, 'speed': 89, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229829.61457476, 'process-damage-time': 9229829.830422487, 'depth': 32})                                                      │
│ 2025-02-13 16:36:03,790 compress: 374.1ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   3.9%  (18934KB to   732KB), sequence    19, client_options={'rgb_format': 'BGRA', 'quality': 100, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 100, 'speed': 80, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229829.840121824, 'process-damage-time': 9229830.115753513, 'depth': 32})                                                   │
│ 2025-02-13 16:36:04,200 compress: 369.6ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   2.3%  (18934KB to   430KB), sequence    20, client_options={'rgb_format': 'BGRA', 'quality': 97, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 97, 'speed': 80, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229830.131157354, 'process-damage-time': 9229830.545986088, 'depth': 32})                                                     │
│ 2025-02-13 16:36:04,423 compress: 128.8ms for 3440x1409 pixels at    0,0    for wid=3     using      webp with ratio   2.4%  (18934KB to   459KB), sequence    21, client_options={'rgb_format': 'BGRA', 'quality': 99, 'has_alpha': True, 'encoder': 'webp', 'flush': 0}, options=typedict({'quality': 99, 'speed': 80, 'rgb_formats': ('RGBX', 'RGBA', 'RGB'), 'lz4': True, 'content-type': 'picture', 'damage-time': 9229830.613038728, 'process-damage-time': 9229831.009950098, 'depth': 32

[totaam]

Wow!
~300ms for a screen update is unbearable!
If most of the screen does not update, the server should be sending the updated regions, not the whole window every time.
That's probably what happens with the python client.

Please include more details: server command line, xpra info, etc

[jeffmax]

Hi @totaam, this is the start command:

 "/usr/bin/xpra clean && /usr/bin/xpra start :13531 --env='XPRA_CLIPBOARD_TEXT_TARGETS=text/plain,UTF8_STRING,TEXT,STRING' --open-url=no --printing=no --audio=no --pulseaudio=no --html=on --mdns=no --bind-tcp=0.0.0.0:13531 --notifications=no --speaker=off --av-sync=no --file-transfer=no --idle-timeout=1200 --use-display=no --daemon=no --tray=no --mousewheel=on --dbus-control=no --dbus-launch=no --systemd-run=no --system-tray=no --webcam=no --start-after-connect='/opt/slicer/Slicer"

I've tried changing the encoding and setting different --speed and --quality and nothing seems to change much.

This is what I think is relevant from xpra info but please let me know if something else would be helpful or any other info would help.
Is that 300ms the time its taking for the server to compress all the data it needs to send?

video.csc.BGRX_to_YUV420P=('cython',)
video.csc.BGRX_to_YUV444P=('cython',)
video.csc.BGR_to_YUV420P=('cython',)
video.csc.BGR_to_YUV444P=('cython',)
video.csc.GBRP10_to_r210=('cython',)
video.csc.GBRP_to_BGRX=('cython',)
video.csc.GBRP_to_RGBX=('cython',)
video.csc.RGBX_to_YUV420P=('cython',)
video.csc.RGBX_to_YUV444P=('cython',)
video.csc.RGB_to_YUV420P=('cython',)
video.csc.RGB_to_YUV444P=('cython',)
video.csc.YUV420P_to_BGR=('cython',)
video.csc.YUV420P_to_BGRX=('cython',)
video.csc.YUV420P_to_RGB=('cython',)
video.csc.YUV420P_to_RGBX=('cython',)
video.csc.YUV444P10_to_r210=('cython',)
video.csc.YUV444P_to_BGRX=('cython',)
video.csc.YUV444P_to_RGBX=('cython',)
video.csc.r210_to_BGR48=('cython',)
video.csc.r210_to_YUV420P=('cython',)
video.csc.r210_to_YUV444P10=('cython',)
video.encoding.BGR48_to_h264=('x264',)
video.encoding.BGRA_to_jpega=('jpeg',)
video.encoding.BGRA_to_webp=('webp',)
video.encoding.BGRX_to_h264=('x264',)
video.encoding.BGRX_to_jpeg=('jpeg',)
video.encoding.BGRX_to_webp=('webp',)
video.encoding.BGR_to_jpeg=('jpeg',)
video.encoding.BGR_to_webp=('webp',)
video.encoding.RGBA_to_jpega=('jpeg',)
video.encoding.RGBA_to_webp=('webp',)
video.encoding.RGBX_to_jpeg=('jpeg',)
video.encoding.RGBX_to_webp=('webp',)
video.encoding.RGB_to_jpeg=('jpeg',)
video.encoding.RGB_to_webp=('webp',)
video.encoding.XBGR_to_jpeg=('jpeg',)
video.encoding.XRGB_to_jpeg=('jpeg',)
video.encoding.YUV420P_to_av1=('gstreamer-av1enc',)
video.encoding.YUV420P_to_h264=('x264', 'openh264', 'gstreamer-x264enc', 'gstreamer-openh264enc')
video.encoding.YUV420P_to_jpeg=('jpeg',)
video.encoding.YUV420P_to_vp8=('vpx', 'gstreamer-vp8enc')
video.encoding.YUV420P_to_vp9=('vpx',)
video.encoding.YUV422P_to_h264=('x264',)
video.encoding.YUV422P_to_jpeg=('jpeg',)
video.encoding.YUV444P10_to_vp9=('vpx',)
video.encoding.YUV444P_to_av1=('gstreamer-av1enc',)
video.encoding.YUV444P_to_h264=('x264', 'gstreamer-x264enc')
video.encoding.YUV444P_to_jpeg=('jpeg',)
video.encoding.YUV444P_to_vp9=('vpx', 'gstreamer-vp9enc')
video.encoding.csc-module.cython=active
video.encoding.csc-module.libyuv=not found
video.encoding.video-encoder.gstreamer=active
video.encoding.video-encoder.jpeg=active
video.encoding.video-encoder.nvenc=not found
video.encoding.video-encoder.nvjpeg=not found
video.encoding.video-encoder.openh264=active
video.encoding.video-encoder.vpx=active
video.encoding.video-encoder.webp=active
video.encoding.video-encoder.x264=active
video.gpu.csc=()
video.gpu.decodings=()
video.gpu.encodings=()
windows.3.XShm=True
windows.3.above=False
windows.3.allowed-actions=('_NET_WM_ACTION_CLOSE', '_NET_WM_ACTION_MOVE', '_NET_WM_ACTION_RESIZE', '_NET_WM_ACTION_FULLSCREEN', '_NET_WM_ACTION_MINIMIZE', '_NET_WM_ACTION_SHADE', '_NET_WM_ACTION_STICK', '_NET_WM_ACTION_MAXIMIZE_HORZ', '_NET_WM_ACTION_MAXIMIZE_VERT', '_NET_WM_ACTION_CHANGE_DESKTOP', '_NET_WM_ACTION_ABOVE', '_NET_WM_ACTION_BELOW')
windows.3.below=False
windows.3.bypass-compositor=0
windows.3.children=()
windows.3.class-instance=('SlicerApp-real', 'Slicer')
windows.3.client-geometry=(0, 31, 1993, 1101)
windows.3.client-machine=testtesttest
windows.3.command=/opt/slicer/bin/SlicerApp-real
windows.3.content-type=picture
windows.3.decorations=126
windows.3.depth=32
windows.3.focused=True
windows.3.fullscreen=False
windows.3.geometry=(0, 31, 1993, 1101)
windows.3.grabbed=False
windows.3.group-leader-xid=0x100000c
windows.3.has-alpha=True
windows.3.icon-title=None
windows.3.iconic=False
windows.3.maximized=True
windows.3.modal=False
windows.3.opacity=-1
windows.3.opaque-region=()
windows.3.override-redirect=False
windows.3.protocols=('WM_DELETE_WINDOW', 'WM_TAKE_FOCUS', '_NET_WM_PING', '_NET_WM_SYNC_REQUEST')
windows.3.requested-position=(0, 31)
windows.3.set-initial-position=True
windows.3.shaded=False
windows.3.shown=True
windows.3.size=(1993, 1101)
windows.3.size-constraints.gravity=10
windows.3.size-constraints.minimum-size=(856, 340)
windows.3.skip-pager=False
windows.3.skip-taskbar=False
windows.3.state=('_NET_WM_STATE_MAXIMIZED_HORZ', '_NET_WM_STATE_FOCUSED', '_NET_WM_STATE_MAXIMIZED_VERT')
windows.3.sticky=False
windows.3.title=3D Slicer 5.8.0
windows.3.tray=False
windows.3.window-type=('NORMAL',)

[totaam]

It's missing a lot of the encoding information for the current client: the current encoding ("auto"? "stream?"), the corresponding "selection" function, etc.

[jeffmax]

Ah, sorry. Does the attached file have the relevant info?

xprainfo.txt

[totaam]

TILs:

child.2.command=Xvfb +extension GLX +extension Composite +extension  RANDR +extension RENDER -screen 0 8192x4096x24+32  -nolisten tcp -noreset -auth $XAUTHORITY -dpi 96x96
display.opengl.renderer=llvmpipe (LLVM 15.0.7, 256 bits)

client.connection.client.ping_latency.50p=24
client.connection.client.ping_latency.80p=63
client.connection.client.ping_latency.90p=116
client.connection.client.ping_latency.avg=71
client.connection.client.ping_latency.cur=21
client.connection.client.ping_latency.max=1357
client.connection.client.ping_latency.min=18

client.damage.client-latency=471
client.damage.frame-total-latency=471

client.encoding.quality.avg=69
client.encoding.regions_per_second=2
client.encoding.speed.avg=86

client.encodings=('rgb24', 'rgb32', 'png', 'webp', 'jpeg')

client.screen.0.display=Chromium 133

client.window.3.batch.actual_delays.90p=238
client.window.3.batch.actual_delays.avg=110
client.window.3.pixel-format=BGRA
client.window.3.property.quality.cur=92
client.window.3.property.speed.cur=92
client.window.3.rgb_formats=('RGBX', 'RGBA', 'RGB')

client.window.3.video-max-size=(1024, 768)

windows.3.class-instance=('SlicerApp-real', 'Slicer')
windows.3.client-geometry=(0, 31, 2560, 1409)
windows.3.client-machine=testtesttest
windows.3.command=/opt/slicer/bin/SlicerApp-real
windows.3.geometry=(0, 31, 2560, 1409)
windows.3.has-alpha=True
windows.3.title=3D Slicer 5.8.0

So you're running a 3D application without a GPU? (llvmpipe)

webp is the "best" encoding available for compressing window data with transparency as the html5 client does not yet support jpega (jpeg with alpha).
You could workaround that using XPRA_ALPHA=0 when starting the server, or removing encodings that support transparency (--encodings=rgb,jpeg,h264).
Or better yet, tell the application not to use RGBA visuals unless it really does need to use the alpha channel.
It's unlikely that the webp encoder uses the wrong settings, you can run with -d webp to verify.

The python client can use h264 and jpega which are much faster.
Adding jpega to the html5 client would improve things quite a bit, but I don't know how to combine RGB and A into a single ImageBitmap. Maybe someone else can help with that?

[jeffmax]

Thanks for looking at this @totaam. One question- I have tested this with a GPU (using VirtualGL and EGL) and confirmed with nvidia-smi that it was being used. It did not seem to change the performance for this function (it does make a huge difference in other aspects of the application). Would you expect the GPU to help with this issue if it is related to the alpha channel/transparency?

[totaam]

It did not seem to change the performance for this function

It wouldn't without changing the settings.
GPU encoders are seriously fast for compressing h264 and jpeg, it would make a massive difference with those settings.

Would you expect the GPU to help with this issue if it is related to the alpha channel/transparency?

No.

[jeffmax]

Question about setting the '--encodings=rgb,jpeg,h264' flag and/or XPRA_ALPHA=0. As I've tested both of those out, neither seems to prevent webp from being used (I still see ' 30.5ms for 2560x1188 pixels at 0,0 for wid=3 using webp ' showing in the -d compress log). Is that expected?

[jeffmax]

Adding a bit more details to this: When I use the Python client, while it is frustratingly inconsistent, it is more likely to perfom much smaller screen updates (say 1738x7) as I move the mouse around the segmentation screen (using webp for the most part, but mixing in rgb24). If I disconnect the same session and then reconnect with the HTML5 client, immediately the compress debug shows basically fully screen updates (1728x885 using webp) as I move the mouse. I imagine that overall these issues are due to a poorly behaving application, but I'd like to dig into this to understand what might be going on. If you have any ideas or directions to explore that would be helpful. Thank you!