-
Notifications
You must be signed in to change notification settings - Fork 131
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Proposal] Improve FREEZING
mechanisms and add Representation
deprecation
#1523
Open
peaBerberian
wants to merge
7
commits into
dev
Choose a base branch
from
misc/lg-detect
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
peaBerberian
force-pushed
the
misc/lg-detect
branch
from
September 2, 2024 13:54
e4f6ede
to
2c9eb7d
Compare
peaBerberian
added
proposal
This Pull Request or Issue is only a proposal for a change with the expectation of a debate on it
Priority: 2 (Medium)
This issue or PR has a medium priority.
labels
Sep 3, 2024
peaBerberian
force-pushed
the
misc/lg-detect
branch
2 times, most recently
from
September 5, 2024 09:32
d853087
to
173db61
Compare
…recation Situation ========= We have lately seen on some LG and Philips TVs what we call "infinite `FREEZING`" occurences: the playback position (the `HTMLMediaElement`'s `currentTime` property) was not advancing and most of the time video media was not playing either (though in some occurences audio was) despite having largely enough media data in media buffers. Sadly our usual trick to restart playback, seeking close to the current position, didn't seem to have an effect. Most of those freezing cases were happening as playback switched from a video quality to another (sometimes both had to be specific, sometimes only the destination one had an impact). It is probably better to detect which situations causes which problems on which devices to work-around it either in the RxPlayer (when for example we KNOW that a device has issues with a given codec in general) or inside the application (when we only reproduced it on a specific application and there's many unknowns left). Yet we found out after encountering issues again and again, especially on some smart TVs, that it may not be realistic to catch in advance (before production) all potential breakages that could exist. So instead, we thought about the possibility to define an heuristic detecting that we're probably encountering this type of issue, to then work-around it automatically. Rough idea of the solution ========================== So, the idea is to detect when a `FREEZING` case occurs right when switching from a given video/audio quality to another. When that's the case: 1. We first try to "un-freeze" playback, by e.g. performing a small seek. This is what we already do today in `FREEZING` cases. 2. If that doesn't work (we're still freezing), I added here a second mechanism: we "deprecate" (the name can change) the corresponding new quality played, so it isn't played anymore in most cases. So basically once we see that switching to another quality lead to `FREEZING`, we avoid playing that quality for the rest of playback. Unlike other scenarios where we prevent playback of qualities (in the RxPlayer called `Representation`), here the deprecation is only considered if there's other, non-deprecated, qualities we can play on the current track: meaning that if all available qualities are deprecated, we will play them again. This is to protect against errors in cases where all qualities would be deprecated: because our heuristic is not 100% sure science, we would here prefer there to keep playback potentially happening (the alternative would have been stopping playback on an error). As of now, the deprecation has no impact on the API: corresponding `Representation` are still returned as if they are playable through our `getVideoTrack` API for example, and the user can still choose to call `lockVideoRepresentation` on deprecated `Representation` without knowing. As of now, the "main thread" part of the code isn't even aware of deprecation happening, for it we're just curiously never seem to be playing some `Representation`. This also means that our `DEBUG_ELEMENT` feature won't even notify for now if there's deprecation happening. This may change in the future, but for now, this is because I did not want to complexify this experiment too much. PS: I like this "deprecation" concept, as it seems to me to be portable for much more future cases where we would prefer not playing qualities because they seem to present issues (e.g. if `MEDIA_ERR_DECODE` or `BUFFER_APPEND_ERROR` errors seem to only happen with some `Representation`). Bonus: also reloading in other cases ==================================== There are several other posibilities for infinite `FREEZING`: Period changes gone wrong, random decoding issues, etc. To also provide a better solution for those cases, I choose to "reload" (in the RxPlayer, "reloading" means removing and re-creating all the media buffers, leading to a temporary black screen) if the initial "un-freezing" attempt (the seek-based one) didn't have any effect. Unlike when the freezing seems to be linked to a quality change though, there's no deprecation happening in that case. Implementation ============== FreezeResolver -------------- I renamed the very-specific `DecipherabilityFreezeDetector` class - which only handled freezes linked to DRM - into a more general `FreezeResolver` class, which will now perform all un-freezing attempts (even the seekinf one, which previously happened in the `RebufferingController` in main thread whereas it's now in `core` - thus optionally running in our `Worker`). The `FreezeResolver`'s `onObservation` method has to be called even when playback goes well, at each produced media observation, because it needs to construct its history of played segments (see next chapter). Quality switch detection ------------------------ Knowing whether we switched from a given media quality to another is not straightforward and we can never really know for sure as there's edge cases where it might be device-specific (e.g. segment replacements, segment pushing still pending etc.). However, we have a rough idea by inspecting the `HTMLMediaElement`'s `currentTime` property (which is roughly the current media position being played), and the content of our `SegmentInventory` class, which stores metadata about all media segments available in the buffer. From there, I maintain a short-term history in the `FreezeResolver` of what seemed to be the current video AND audio quality played. If a `FREEZING`, which we do not seem to be able to fix by seeking, seems to coincide with a quality switch, we propose to deprecate the `Representation`. For now this also lead to a `RELOADING` operation, though it may not even be required in some cases (e.g. replacing segments and seeking in place might do the trick). Yet reloading should have more chance of fixing it (even though it leads to a temporary black screen, where a still video frame would have been less disruptive for users). Avoiding deprecated `Representation` ------------------------------------ Then, an `AdaptationStream` is able to filter out deprecated `Representation` (unless it has no choice), when asking the `AdaptiveRepresentationSelector` (managing our Adaptive BitRate logic) which `Representation` should be played (though I'm still not sure whether the `AdaptiveRepresentationSelector` should actually be the one doing that?). Where we go from there ====================== This heuristic seems somewhat risky. as we're essentially blacklisting qualities from ever be playing again on the current content. So what I thought was to put the deprecation mechanism behind an experimental `experimental.enableRepresentationDeprecation` `loadVideo` option, that applications would have to enable (making it also possible to disable at any time easily through some config). However the reload mechanism if un-freezing fails seems OK to me for enabling it by default. If we do notice clear improvements, we might think of enabling the deprecation mechanism by default, removing the `experimental.enableRepresentationDeprecation` option. Thoughts?
peaBerberian
force-pushed
the
misc/lg-detect
branch
from
September 5, 2024 09:38
173db61
to
6287c96
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Priority: 2 (Medium)
This issue or PR has a medium priority.
proposal
This Pull Request or Issue is only a proposal for a change with the expectation of a debate on it
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Situation
We have lately seen on some LG and Philips TVs what we call "infinite
FREEZING
" occurences: the playback position (theHTMLMediaElement
'scurrentTime
property) was not advancing and most of the time video media was not playing either (though in some occurences audio was) despite having largely enough media data in media buffers. Sadly our usual trick to restart playback, seeking close to the current position, didn't seem to have an effect.Most of those freezing cases were happening as playback switched from a video quality to another (sometimes both had to be specific, sometimes only the destination one had an impact).
It is probably better to detect which situations causes which problems on which devices to work-around it either in the RxPlayer (when for example we KNOW that a device has issues with a given codec in general) or inside the application (when we only reproduced it on a specific application and there's many unknowns left). Yet we found out after encountering issues again and again, especially on some smart TVs, that it may not be realistic to catch in advance (before production) all potential breakages that could exist.
So instead, we thought about the possibility to define an heuristic detecting that we're probably encountering this type of issue, to then work-around it automatically.
Rough idea of the solution
So, the idea is to detect when a
FREEZING
case occurs right when switching from a given video/audio quality to another. When that's the case:We first try to "un-freeze" playback, by e.g. performing a small seek. This is what we already do today in
FREEZING
cases.If that doesn't work (we're still freezing), I added here a second mechanism: we "deprecate" (the name can change) the corresponding new quality played, so it isn't played anymore in most cases.
So basically once we see that switching to another quality lead to
FREEZING
, we avoid playing that quality for the rest of playback.Unlike other scenarios where we prevent playback of qualities (in the RxPlayer called
Representation
), here the deprecation is only considered if there's other, non-deprecated, qualities we can play on the current track: meaning that if all available qualities are deprecated, we will play them again. This is to protect against errors in cases where all qualities would be deprecated: because our heuristic is not 100% sure science, we would here prefer there to keep playback potentially happening (the alternative would have been stopping playback on an error).As of now, the deprecation has no impact on the API: corresponding
Representation
are still returned as if they are playable through ourgetVideoTrack
API for example, and the user can still choose to calllockVideoRepresentation
on deprecatedRepresentation
without knowing. As of now, the "main thread" part of the code isn't even aware of deprecation happening, for it we're just curiously never seem to be playing someRepresentation
. This also means that ourDEBUG_ELEMENT
feature won't even notify for now if there's deprecation happening.This may change in the future, but for now, this is because I did not want to complexify this experiment too much.
PS: I like this "deprecation" concept, as it seems to me to be portable for much more future cases where we would prefer not playing qualities because they seem to present issues (e.g. if
MEDIA_ERR_DECODE
orBUFFER_APPEND_ERROR
errors seem to only happen with someRepresentation
).Bonus: also reloading in other cases
There are several other posibilities for infinite
FREEZING
: Period changes gone wrong, random decoding issues, etc.To also provide a better solution for those cases, I choose to "reload" (in the RxPlayer, "reloading" means removing and re-creating all the media buffers, leading to a temporary black screen) if the initial "un-freezing" attempt (the seek-based one) didn't have any effect.
Unlike when the freezing seems to be linked to a quality change though, there's no deprecation happening in that case.
Implementation
FreezeResolver
I renamed the very-specific
DecipherabilityFreezeDetector
class - which only handled freezes linked to DRM - into a more generalFreezeResolver
class, which will now perform all un-freezing attempts (even the seekinf one, which previously happened in theRebufferingController
in main thread whereas it's now incore
- thus optionally running in ourWorker
).The
FreezeResolver
'sonObservation
method has to be called even when playback goes well, at each produced media observation, because it needs to construct its history of played segments (see next chapter).Quality switch detection
Knowing whether we switched from a given media quality to another is not straightforward and we can never really know for sure as there's edge cases where it might be device-specific (e.g. segment replacements, segment pushing still pending etc.).
However, we have a rough idea by inspecting the
HTMLMediaElement
'scurrentTime
property (which is roughly the current media position being played), and the content of ourSegmentInventory
class, which stores metadata about all media segments available in the buffer.From there, I maintain a short-term history in the
FreezeResolver
of what seemed to be the current video AND audio quality played. If aFREEZING
, which we do not seem to be able to fix by seeking, seems to coincide with a quality switch, we propose to deprecate theRepresentation
.For now this also lead to a
RELOADING
operation, though it may not even be required in some cases (e.g. replacing segments and seeking in place might do the trick). Yet reloading should have more chance of fixing it (even though it leads to a temporary black screen, where a still video frame would have been less disruptive for users).Avoiding deprecated
Representation
Then, an
AdaptationStream
is able to filter out deprecatedRepresentation
(unless it has no choice), when asking theAdaptiveRepresentationSelector
(managing our Adaptive BitRate logic) whichRepresentation
should be played (though I'm still not sure whether theAdaptiveRepresentationSelector
should actually be the one doing that?).Where we go from there
This heuristic seems somewhat risky. as we're essentially blacklisting qualities from ever be playing again on the current content.
So what I thought was to put the deprecation mechanism behind an experimental
experimental.enableRepresentationDeprecation
loadVideo
option, that applications would have to enable (making it also possible to disable at any time easily through some config).However the reload mechanism if un-freezing fails seems OK to me for enabling it by default.
If we do notice clear improvements, we might think of enabling the deprecation mechanism by default, removing the
experimental.enableRepresentationDeprecation
option.Thoughts?