Shaka release 4.9.2-caf2 isn't playing our HLS streams anymore

[ChristiaanScheermeijer]

Have you read the FAQ and checked for duplicate open issues?
Yes

If the problem is related to FairPlay, have you read the tutorial?
Not related

What version of Shaka Player are you using?
4.9.2-caf2

Can you reproduce the issue with our latest release version?
No

Can you reproduce the issue with the latest code from main?
No

Are you using the demo app or your own custom app?
Custom receiver

If custom app, can you reproduce the issue using our demo app?
No

What browser and OS are you using?
MacOS 15.0 - Google Chrome 130 - Google Chromecast 4K

For embedded devices (smart TVs, etc.), what model and firmware version are you using?
Google Chromecast 4K - Android TV 12

What are the manifest and license server URIs?
https://livestreaming.b67.tweedekamer.nl/2024-11-13/thorbeckezaal/index.m3u8?hd=1&sourcetimestamps=1&subtitles=vod&keyframes=1&start=2024-11-13T10%3A00%3A12%2B0100&end=2024-11-13T13%3A11%3A47%2B0100

What configuration are you using? What is the output of player.getNonDefaultConfiguration()?

// using default shaka settings set by custom receiver except for this:
playbackConfig.shakaConfig = {
  streaming: {
    lowLatencyMode: false
  }
};

What did you do?

• Playback stream on website (non shaka)
• Start casting the stream

What did you expect to happen?

• The stream starts playing on the Chromecast device

What actually happened?

• The stream keeps loading for a while and shows a black screen
• After some time the Chromecast session is automatically stopped

Are you planning send a PR to fix it?
Uncertain at this moment

Additional information
We're receiving multiple reports of our users that casting is not working anymore. We're able to trace it down to the Shaka player upgrade done recently because it was working fine earlier this week. When we manually downgrade the Shaka version to 4.3.4 (the previous version used by the receiver) the issue is gone.

[joeyparrish]

Can you please try the latest in each branch? If we've fixed it in a more recent release, it would be great to be able to roll you forward instead of backward.

For now, you can continue to use 4.3.4 in production, but if you could please test each of these in a staging or QA environment, that could help us figure out what went wrong:

• shakaVersion: "4.9.31" (latest in 4.9 branch)
• shakaVersion: "4.10.26" (latest in 4.10 branch)
• shakaVersion: "4.11.12" (latest in 4.11 branch)
• shakaVersion: "4.12.0" (released this week)
• shakaUrl: "https://nightly-dot-shaka-player-demo.appspot.com/dist/shaka-player.compiled.js" (nightly build)

If none of these work, we would ask you to help us find the first branch that broke it (v4.4.0 - v4.9.0). Ultimately, we would ask you to work with us on a git bisect to identify the specific change that is to blame (for breaking it if it's not fixed anywhere, or for fixing it if it is already fixed in a newer branch).

Thanks!

[ChristiaanScheermeijer]

Hi @joeyparrish, thanks! I have tested all versions using our stream and here are the results:

• 4.9.31 - same issue
• 4.10.26 - same issue
• 4.11.12 - same issue
• 4.12.0 - same issue

So, I went back in versions:

Versions 4.4.0 <> 4.5.0 also don't play our stream, but different. It stops playback immediately and shows this error in the console:

cast_receiver_framework.js:79  [  1.687s] [cast.framework.media.ShakaPlayer] category: 1
code: 1007
stack: Error: Shaka Error 1007
    at new N (https://ajax.googleapis.com/ajax/libs/shaka-player/4.4.3/shaka-player.compiled.js:80:670)
    at https://ajax.googleapis.com/ajax/libs/shaka-player/4.4.3/shaka-player.compiled.js:172:338
    at Qe (https://ajax.googleapis.com/ajax/libs/shaka-player/4.4.3/shaka-player.compiled.js:160:30)
    at https://ajax.googleapis.com/ajax/libs/shaka-player/4.4.3/shaka-player.compiled.js:159:107
severity: CRITICAL
data: [{}]

[  1.710s] [cast.framework.PlayerManager] Load failed:  
kf @ cast_receiver_framework.js:79

Uncaught (in promise) 
customData: undefined
detailedErrorCode: undefined
itemId: 1
reason: undefined
requestId: 245
type: "LOAD_FAILED"

From version 4.6.18 and up the behaviour is the same as for the newer versions. I found that the issue started in 4.3.6. So it most likely is a change between 4.3.5 and 4.3.6.

v4.3.5...v4.3.6

[joeyparrish]

I will attempt to bisect through these versions in a custom app with your content. Thanks for the info!

[avelad]

On the main branch the stream works perfectly

[ChristiaanScheermeijer]

@avelad have you tested the main branch on a Chromecast receiver? I was only able to reproduce this issue on a Cast receiver.

[joeyparrish]

Sorry, the holidays were crazy. I promised to bisect this, and I'm doing so now. Each step of bisecting Shaka, I'm uploading that build to cloud storage with a custom receiver to test with your content. It definitely plays at 4.3.5 and not at 4.3.6. I should have it pinned down later today.

[joeyparrish]

This is the first bad commit: https://github.com/shaka-project/shaka-player/pull/5180/files

[joeyparrish]

This made changes to the same file: https://github.com/shaka-project/shaka-player/pull/6653/files

With both reverted, I can play your content on Chromecast up to v4.10.0. Still exploring how far those changes will take us. Then I plan to look more closely into how these changes can explain the failure with your content.

[joeyparrish]

After analyzing the changes in #5180, it seems that we changed an output for HEAD requests. In the older Chromium (as present on Chromecast), the arrayBuffer for a HEAD request was an empty ArrayBuffer. After our change, we avoid calling for it, and we leave the local arrayBuffer variable undefined. This then becomes part of the shaka.extern.Response object.

It's not clear who would be accessing that, or why, but it should be easy enough to restore the old behavior and always have a defined ArrayBuffer (even an empty one). If it only fails on Chromecast, it's probably failing in the Cast SDK itself for some reason. Perhaps a filter inserted into the network stack of Shaka Player.

With ArrayBuffer defined (a 1-line change), this is now working all the way up to today's main branch.

I'll put that change up for review shortly, but I'll also continue to dig into the Cast SDK to see why that seems to trigger this. We should patch the SDK as well, if possible.

[joeyparrish]

The HLS parser makes a HEAD request with request type SEGMENT, to determine the MIME type of a segment when it can't otherwise guess from the URL. Your segment URLs don't have file extensions (which is fine, generally).

The Cast SDK has response filters that assume SEGMENT responses always have data. (The SDK does not assume this about MANIFEST or KEY responses.) So I'll fix the SDK, too.

[ChristiaanScheermeijer]

Many thanks @joeyparrish! 💪 Kudos for the detailed explanation of the problem, I've seen the SEGMENT requests before and didn't know what it was for. I do now 😄