Video / Audio playback disruption

[DougShuffield]

I am using OMXplayer on a Pi 3B+ running Rasbian stretch to display a PC HDMI output broadcast via UDP multicast from an HDMI extender in my system. I am having an issue where the video / audio playback gets disrupted when I start a UE4 based game on the PC. It seems that the dts / pts gets reset during the transition when the UE4 game starts which then causes OMXplayer to not be able to play the audio and sometimes not be able to play the video. Here is my command:

omxplayer --dbus_name "org.mpris.MediaPlayer2.omxplayer.gameplay" --with-info --live --threshold 0.5 --fps 30 --crop 320,0,1600,1080 --win 640,0,1920,1080 --layer 10 $gpIP

Here is the log when the issue occurs somewhere between 21:28:20 and 21:28:23:

21:28:20 T:18446744072662452240 DEBUG: Normal M:189392402 (A:188754000 V:188610000) P:0 A:-0.64 V:-0.78/T:0.50 (1,1,0,0) A:0% V:0% (-0.61,6.34)
21:28:20 T:18446744072662452343 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:20 T:18446744072662452373 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:20 T:18446744072662452642 DEBUG: Live: 0.66 (-0.64) S:1.001 T:0.50
21:28:20 T:18446744072662453001 INFO: CDVDPlayerAudio::Decode dts:188806000 pts:188806000 size:627
21:28:20 T:18446744072662453454 INFO: CDVDPlayerAudio::Decode dts:188832000 pts:188832000 size:627
21:28:20 T:18446744072662465283 INFO: COMXAudio::ApplyVolume - Volume=1.00 (* 1.00 * 1.00)
21:28:20 T:18446744072662495931 INFO: CDVDPlayerAudio::Decode dts:188858000 pts:188858000 size:627
21:28:20 T:18446744072662496227 DEBUG: Normal M:189436515 (A:188832000 V:188610000) P:0 A:-0.60 V:-0.83/T:0.50 (1,1,0,0) A:0% V:0% (-0.66,6.34)
21:28:20 T:18446744072662496323 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:20 T:18446744072662496350 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:20 T:18446744072662496863 DEBUG: Live: 0.65 (-0.60) S:1.001 T:0.50
21:28:21 T:18446744072662931688 INFO: CDVDPlayerAudio::Decode dts:188676000 pts:188676000 size:626
21:28:21 T:18446744072662932096 ERROR: COMXAudioCodecOMX::GetData Unexpected change of size (9216->4608)
21:28:21 T:18446744072662935281 INFO: COMXAudio::ApplyVolume - Volume=1.00 (* 1.00 * 1.00)
21:28:21 T:18446744072662935571 DEBUG: Normal M:189874233 (A:188676000 V:188610000) P:0 A:-1.20 V:-1.26/T:0.50 (1,1,0,0) A:0% V:0% (-1.02,6.34)
21:28:21 T:18446744072662935716 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:21 T:18446744072662935777 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:21 T:18446744072662936236 DEBUG: Live: 0.63 (-1.20) S:1.001 T:0.50
21:28:21 T:18446744072662936573 INFO: CDVDPlayerAudio::Decode dts:188702000 pts:188702000 size:627
21:28:21 T:18446744072662937599 INFO: CDVDPlayerAudio::Decode dts:188728000 pts:188728000 size:627
21:28:21 T:18446744072662938028 INFO: CDVDPlayerAudio::Decode dts:188754000 pts:188754000 size:627
21:28:21 T:18446744072662959971 INFO: CDVDPlayerAudio::Decode dts:188780000 pts:188780000 size:627
21:28:21 T:18446744072662960429 DEBUG: Normal M:189900978 (A:188754000 V:188610000) P:0 A:-1.15 V:-1.29/T:0.50 (1,1,0,0) A:0% V:0% (-1.04,6.34)
21:28:21 T:18446744072662960563 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:21 T:18446744072662960622 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:21 T:18446744072662960971 DEBUG: Live: 0.61 (-1.15) S:1.001 T:0.50
21:28:21 T:18446744072662961218 INFO: CDVDPlayerAudio::Decode dts:188806000 pts:188806000 size:627
21:28:21 T:18446744072662961673 INFO: CDVDPlayerAudio::Decode dts:188832000 pts:188832000 size:627
21:28:21 T:18446744072663051943 INFO: CDVDPlayerAudio::Decode dts:188858000 pts:188858000 size:627
21:28:21 T:18446744072663052406 DEBUG: Normal M:189993083 (A:188832000 V:188610000) P:0 A:-1.16 V:-1.38/T:0.50 (1,1,0,0) A:0% V:0% (-1.14,6.34)
21:28:21 T:18446744072663052542 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:21 T:18446744072663052605 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:21 T:18446744072663052954 DEBUG: Live: 0.59 (-1.16) S:1.001 T:0.50
21:28:23 T:18446744072664819247 INFO: CDVDPlayerAudio::Decode dts:330000 pts:330000 size:627
21:28:23 T:18446744072664819580 DEBUG: Normal M:191762084 (A:188858000 V:188610000) P:0 A:-2.90 V:-3.15/T:0.50 (1,1,0,0) A:0% V:0% (-2.90,6.34)
21:28:23 T:18446744072664819716 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:0
21:28:23 T:18446744072664819777 DEBUG: OMXClock::OMXSetSpeed(1.00) pause_resume:1
21:28:23 T:18446744072664820231 DEBUG: Live: 0.56 (-2.90) S:1.001 T:0.50
21:28:23 T:18446744072664820367 ERROR: COMXAudioCodecOMX::GetData Unexpected change of size (4608->9216)
21:28:23 T:18446744072664822839 INFO: COMXAudio::ApplyVolume - Volume=1.00 (* 1.00 * 1.00)
21:28:23 T:18446744072664822988 INFO: CDVDPlayerAudio::Decode dts:356000 pts:356000 size:627
21:28:23 T:18446744072664824004 INFO: CDVDPlayerAudio::Decode dts:382000 pts:382000 size:627
21:28:23 T:18446744072664824759 INFO: CDVDPlayerAudio::Decode dts:408000 pts:408000 size:627
21:28:23 T:18446744072664848176 INFO: CDVDPlayerAudio::Decode dts:435000 pts:435000 size:627
21:28:23 T:18446744072664848467 DEBUG: Normal M:191791151 (A:435000 V:188610000) P:0 A:-191.36 V:-3.18/T:0.50 (1,1,0,0) A:0% V:0% (-2.93,6.34)
21:28:23 T:18446744072664848583 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:0
21:28:23 T:18446744072664848643 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:1
21:28:23 T:18446744072664849201 DEBUG: Live: -1.36 (-191.36) S:0.990 T:0.50
21:28:23 T:18446744072664921256 INFO: CDVDPlayerAudio::Decode dts:461000 pts:461000 size:627
21:28:23 T:18446744072664922742 DEBUG: Normal M:191864631 (A:461000 V:188610000) P:0 A:-191.40 V:-3.25/T:0.50 (1,1,0,0) A:0% V:0% (-3.01,6.34)
21:28:23 T:18446744072664922953 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:0
21:28:23 T:18446744072664923017 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:1
21:28:23 T:18446744072664923485 DEBUG: Live: -3.26 (-191.40) S:0.990 T:0.50
21:28:23 T:18446744072664923787 INFO: CDVDPlayerAudio::Decode dts:487000 pts:487000 size:627
21:28:23 T:18446744072664950956 INFO: CDVDPlayerAudio::Decode dts:513000 pts:513000 size:627
21:28:23 T:18446744072664954353 INFO: COMXAudio::ApplyVolume - Volume=1.00 (* 1.00 * 1.00)
21:28:23 T:18446744072664954579 DEBUG: Normal M:191892783 (A:513000 V:188610000) P:0 A:-191.38 V:-3.28/T:0.50 (1,1,0,0) A:0% V:0% (-3.04,6.34)
21:28:23 T:18446744072664954658 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:0
21:28:23 T:18446744072664954684 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:1
21:28:23 T:18446744072664954924 DEBUG: Live: -5.14 (-191.38) S:0.990 T:0.50
21:28:23 T:18446744072664955291 INFO: CDVDPlayerAudio::Decode dts:539000 pts:539000 size:627
21:28:23 T:18446744072665010146 INFO: CDVDPlayerAudio::Decode dts:565000 pts:565000 size:627
21:28:23 T:18446744072665010768 DEBUG: Normal M:191951381 (A:565000 V:188610000) P:0 A:-191.39 V:-3.34/T:0.50 (1,1,0,0) A:0% V:0% (-3.09,6.34)
21:28:23 T:18446744072665010866 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:0
21:28:23 T:18446744072665010895 DEBUG: OMXClock::OMXSetSpeed(0.99) pause_resume:1
21:28:23 T:18446744072665011197 DEBUG: Live: -7.01 (-191.39) S:0.990 T:0.50

Any help resolving this would be much appreciated. Let me know if there is any thing else I can provide to help troubleshoot.

Doug
omxplayer.log

[popcornmix]

omxplayer is being deprecated in favour of vlc. Are you able to try with vlc?

[DougShuffield]

I saw that in another thread and am trying it out as we speak.

Currently looking for the command line options to specify dispmanx layer, window size, and window position with cvlc...

In my particular instance I am using two instances of omxplayer for a split screen camera / gameply and using kivy for an overlay. I am a little concerned that vlc may be too resource heavy to do it on the 3B+, but I will try it out and report back. My only other thought was to compile omxplayer with the latest ffmpeg to see if that might have more robust handling of the stream transition.

[DougShuffield]

Any help in converting these calls to omxplayer into their equivalent calls to cvlc would be much appreciated. The info on realtime crop, window position, and dispmanx layer is lacking for cvlc. I continue to look, but not very clear.

Camera left hand split

omxplayer -n -1 --threshold 0.01 --fps 30 --crop 547,0,1094,922 --win 0,0,640,1080 --layer 20 $camrxIP &

Gameplay right hand split

omxplayer --live --threshold 0.5 --fps 30 --crop 320,0,1600,1080 --win 640,0,1920,1080 --layer 10 $gpIP &

[popcornmix]

@jc-kynesim can you advise?
I believe we can use window_size and window_x/window_y but google isn't too helpful.
Did you add display_layer and fullscreen?

[jc-kynesim]

Current vlc certainly allows positioning & sizing of the video playback window in "fullscreen". The --mmal-layer and -f options should do the trick for layer & fullscreen, crop may be unimplemented unless it turns up automagically (I'm sure I haven't tested it). A possible problem as it stands is that I think that the video layer disables all layers beneath it and there is no option for stopping that.

[popcornmix]

vlc certainly allows positioning & sizing of the video playback window in "fullscreen"

Do you know what command line options are for this (running cvlc)?
--mmallayer and -f are accepted. --window_size and --window_x/--window_y don't seem to be.

[KenT2]

The existence of 'fullscreen commands is really great.

A possible problem as it stands is that I think that the video layer disables all layers beneath it and there is no option for stopping that.

Is it a case of adding an option or is this disabling a permanent drawback. I would like to move Pi Presents from omxplayer to cvlc. In PP I overlay omx videos over x windows text and images using Tkinter but this would stop all of that.

Are all the 'fullscreen' options documented anywhere. Is the code viewable?

[popcornmix]

You should be able to get the code from apt source.
I think this is it:
https://github.com/RPi-Distro/vlc/tree/buster-rpt/debian/patches
(upstream vlc in top level directory plus patches in this directory)

Adding an option to not disable the layers beneath the video should be straightforward in a future update.

[jc-kynesim]

If you are brave then vlc -H will list everything along with at least some description. Beware of the dusty corners of the --mmal-xxx options - some of those were invented for debug purposes only and may have rotted.

[popcornmix]

@jc-kynesim I've tried:

cvlc -f --video-x 64 --video-y 64 --width 256 --height 320 <video>

but it has no obvious effect. Do you know how to alter size and positioning of the area video is rendered to?

[jc-kynesim]

I'll check tomorrow - I've run out of time right now. I know I did manage to move stuff around when I was testing but I've forgotten - sorry.

[DougShuffield]

Thanks for the efforts guys. I will continue to look into the cvlc path as well.
I am also going to try out the omxplayer pull request: Added option --force-fps #678 to see if that can possibly help with the reset of the dts / pts.

Doug

[jc-kynesim]

OK - looks like I am misremembering doing positioning in fullscreen. It clearly doesn't work currently. I'll try and make sure it gets into the next patch.

[DougShuffield]

No dice with the force-fps branch as it seems the fake pts is only generated for video, not audio. When the audio pts/dts disruption occurs, still craps out the audio output and all I hear is static.

@popcornmix : Is there a fairly simple way to modify the code to detect the condition where the audio pts / dts resets like this (large negative delta between last pts and current pts) and then flush the buffer and start again with the new dts/pts. Maybe adding a call to OMXPlayerAudio::Flush() in OMXPlayerAudio::Decode of OMXPlayerAudio.cpp when the condition is met? I am investigating the code, but still not certain on the appropriate place to add, or if it will work at all...

Thanks again for all your help!

[DougShuffield]

@popcornmix Any thoughts?

What I would like to do:

1. Detect reset of PTS (current pts < last pts)
2. Disregard previous values of PTS in calculation of m_latency or reset calculation.
3. Reset or flush audio buffers (maybe not necessary?)
4. Start audio decoding and calculation of latency for --live fresh starting with new PTS

I've tried a few things, but there are many moving parts, so any insight here would be appreciated.

[jc-kynesim]

I'm now looking at the VLC fullscreen positioning thing. It seems I have a couple of choices - I can try and reuse the existing height, width, x, y options but that is confusing if we are playing from X where we have both fullscreen & windowed coords & sizes which will normally want to be different, or I just invent a --mmal_vout_window w,h,x,y option which is less "obvious" / documented but has the advantage of being clear in what it refers to. I'm currently favouring the latter. Any opinions?

[KenT2]

I suspect most users of the --fullscreen option with other than fullscreen use will be people embedding the videos so using the --mmal_vout_window option will be OK and will match the other mmal options.

Hopefully you will also be able to implement an option to get rid of the black background layer.

[popcornmix]

I'm happy with the mmal_ options.

[KenT2]

Further to my post above I have been experimenting with controlling cvlc via dbus using a Pi4 with two monitors. I have found that, in addition to the black 'mmal' background on the monitor showing the video there is also a x/framebuffer related black background on the other display when a video is playing (suspect it covers the whole desktop). I wonder if it possible to remove this when --fullscreen is selected.

Have just given up dbus as I cannot get volume control to work and am trying pyhon-vlc

[DougShuffield]

+1 for mmal_ options.

[jc-kynesim]

This is the wrong repo to be discussing VLC on but never mind. I've got fullscreen window positioning working and I have a transparency option. If you start from X then I cannot easily remove the parent X window beneath it so there is still a black layer. I've a few more issues to deal with so it'll be a short while before release.

[DougShuffield]

Agreed on the repo. Feel free to add a new issue in vlc repo with the specifics of what you are working on to get cvlc to have omxplayer functionality of --live --layer --crop and --win.

I won't be using X at all in my implementation, so should not have an issue with the black layer.

I will leave this issue open until either:

1. I can use 2 instances of cvlc to do what I was doing with omxplayer (two streams split screen with equivalent functionality of --live, --layer, --crop, --win)
2. I can modify omxplayer code to fix and/or workaround the issue with a DTS / PTS reset during a stream using --live option.

Thanks again for all the help!

[jc-kynesim]

I've hardly ever used omxplayer - what functionality are you expecting from --crop & and --live (and in the case of crop how do you expect it to interact with scaling & rotation (both of source & screen))? Oh! and where do you hope subtitles turn up on a cropped image (this may be outside my control but I'd like a guess as to what you are expecting)?

[popcornmix]

--live enables subtle playback speed changes (through resampling) in order to keep the amount of buffered video/audio within a desired window.
Without that if your server's clock is a little slower than yours you'll continually underrun and if a little faster you'll end up with continually expanding buffered data.
I suspect vlc has some comparable logic but it may be automatically enabled when playing from suitable URLS (rtsp/rtp/rtmp)

[DougShuffield]

@jc-kynesim: It doesn't need to work exactly the same, just can produce an equivalent output. Here are the requirements in my specific case:

1. --crop x1,y1,x2,y2 Set crop area for input video. In my case:
   a. Camera: I crop a camera input stream (from a RPI + camera v2 over TCP) to take the center third of the full frame (1640x922) with --crop 547,0,1094,922
   b. Gameplay: I crop a PC display input stream from an HDMI extender via UDP multicast to take the center two thirds of the full frame (1920x1080) with --crop 320,0,1600,1080

2. --win x1,y1,x2,y2 Set position of video window
   a. Camera: I then position the cropped camera stream in the full screen output frame (1920x1080) at the left most third of the screen with --win 0,0,640,1080. By using this window size, the source stream is scaled up from the cropped 547 x 922 to fit in 640 x 1080. Same (or close to the same) aspect ratio.
   b. Gameplay: I position the cropped gameplay in the full screen output frame in the right most 2/3 of the screen with --win 640,0,1920,1080

3. --layer n Set video render layer number (higher numbers are on top)
   a. Camera: I then place the cropped and positioned camera stream to be displayed on dispmanx layer 20
   b. Gameplay: I place the cropped and positioned gameplay stream to be displayed on dispmanx layer 10

4. --live Set for live tv or vod type stream. @popcornmix can provide more details on what he is doing internally to make this happen.
   a. Camera: this option is not used for the camera stream as it is TCP and the latency is very small when streaming from raspivid on the camera pi (~0.5 seconds).
   b. Gameplay: This option reduces latency in the display of the stream from 2-3 seconds down to about 0.5 seconds. This is important so that the player movements caught on the camera correspond to the gameplay.

5. In the final step I add a graphic overlay that can hold player name ("None"), and two logos (blank on this one) using Kivy on dispmanx layer 40 with KIVY_BCM_DISPMANX_LAYER=40 python /home/pi/Scripts/videooverlay.py &

6. No subtitles are required.

A picture is worth a thousand words, so here is the resulting video. Hope this helps.

[KenT2]

Hi
I have created two new issues in the RPi VLC repository for the VLC related issues.

Thanks for your work @jc-kynesim, looking forward to a vlc with --win

[jc-kynesim]

I've got windowing & transparency working. I'm working on crop. If "live" doesn't just work with the base vlc then I'm unlikely to find a fix for it in this release.

[DougShuffield]

Any status?

[DougShuffield]

I am sufficiently satisfied that the features discussed are operational in vlc. Moving to vlc. Thanks for all the great work on omxplayer!

[DougShuffield]

One final comment for posterity. I was unable to get VLC working with sufficient stability for my configuration. The original design included omxplayer for the camera and mpv for gameplay (compiled using https://www.raspberrypi.org/forums/viewtopic.php?f=38&t=199775 and ffmpeg 4.1). After failing with vlc, I revisited mpv and found the --video-pan-x=0.16667 option which allowed me to shift the underlying gameplay video to the right as desired. I am unable to crop the gameplay video, but since the camera is on top of the gameplay in my dispmanx stack up, it doesn't matter. This also allowed me to bypass the hours long update from stretch to buster to use the new features of vlc.