Subtitle (Track - VTT)

[talaysa]

Hi,
Is there any subtitle support on chromecast? We have external track with VTT format, how we can send them with chromecast?

Thanks,
Salih

[warren-bank]

Support for playing subtitles on Chromecast is a feature that I would like, as well. I spent some time yesterday exploring this issue, and will share my findings.

Ideally, I'd like this to trigger some discussion because there are a few moving parts and proper support for this feature would require touching both clappr-core and hlsjs-playback.

For my next step, I believe that I can write a small plugin to extend (ie: monkey-patch) this clappr-chromecast-plugin to add support for settings.playback.externalTracks. Even if successful, it still won't be able to support in-stream text tracks.

---

before digging into code, lets start with some Clappr settings to test text tracks:

// -----------------------------------------------
// references:
//   https://github.com/clappr/clappr/issues/1477
// -----------------------------------------------
var initialize_text_tracks = function() {
  var video = document.querySelector('video')
  if (!video) return

  var textTracks = video.textTracks
  if (!textTracks || !textTracks.length) return

  for (var i=0; i < textTracks.length; i++) {
    if (textTracks[i].mode === 'showing') return
  }

  // turn on the first subtitles text track (which is always "Disabled") to display the "CC" icon/menu in the media-control panel
  textTracks[0].mode = 'showing'
}

// -----------------------------------------------
// references:
//   http://demo.theoplayer.com/closed-captions-subtitles
//   https://github.com/videojs/video.js/tree/v7.10.2/docs/examples/elephantsdream
// -----------------------------------------------
var player = new Clappr.Player({
  source: 'https://cdn.theoplayer.com/video/elephants-dream/playlist-single-audio.m3u8',  // in-stream WebVTT: Chinese, French
  poster: 'https://demo.theoplayer.com/hubfs/Demo_zone/elephants-dream.jpg',
  height: 360,
  width: 640,
  plugins: [ChromecastPlugin],
  chromecast: {
    appId: '9DFB77C0',
    media: {
      type: ChromecastPlugin.Movie,
      title: 'Elephants Dream',
      subtitle: '2006 Dutch computer animated science fiction fantasy experimental short film produced by Blender Foundation'
    }
  },
  playback: {
    crossOrigin: 'anonymous',
    externalTracks: [{
      lang:  'en-US',
      label: 'English',
      kind:  'subtitles',
      src:   'https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.en.vtt'
    },{
      lang:  'sv',
      label: 'Swedish',
      kind:  'subtitles',
      src:   'https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.sv.vtt'
    },{
      lang:  'ru',
      label: 'Russian',
      kind:  'subtitles',
      src:   'https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.ru.vtt'
    },{
      lang:  'ja',
      label: 'Japanese',
      kind:  'subtitles',
      src:   'https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.ja.vtt'
    },{
      lang:  'ar',
      label: 'Arabic',
      kind:  'subtitles',
      src:   'https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.ar.vtt'
    }]
  },
  events: {
    onPlay:  initialize_text_tracks
  }
});

---

observations without any changes to Clappr:

• on desktop Chrome browser:
  • all text tracks load in Clappr and function correctly
• on Chromecast (gen 1)
  • no text tracks load
    • as expected, because this feature isn't implemented
  • the HLS video doesn't play
    • interesting, because I've never seen Chromecast refuse an HLS
    • this is obviously an issue with Chromecast and its support for HLS with in-stream text tracks

necessary changes to settings to continue with testing:

var player = new Clappr.Player({
  source: 'https://d2zihajmogu5jn.cloudfront.net/elephantsdream/ed_hd.mp4',
  ...
});

---

• the "Add Advanced Features" section in the Chromecast docs for building a Chrome sender app outlines how to manage (external) text tracks
• as a first step, I confirmed that the above methodology will work with the Clappr Chromecast receiver app by making the following changes to chromecast.js and testing them while running the webpack dev server:

    get activeTrackIds() {
      let trackId = this.container.closedCaptionsTrackId
      return (trackId >= 0)
        ? [trackId]
        : []
    }
  
    loadMedia() {
      this.container.pause()
      let src = this.container.options.src
      Log.debug(this.name, 'loading... ' + src)
      let mediaInfo = this.createMediaInfo(src)
      let request = new chrome.cast.media.LoadRequest(mediaInfo)
      request.autoplay = true
      request.activeTrackIds = this.activeTrackIds
      if (this.currentTime) {
        request.currentTime = this.currentTime
      }
      this.session.loadMedia(request, (mediaSession) => this.loadMediaSuccess('loadMedia', mediaSession), (e) => this.loadMediaError(e))
    }
  
    createMediaInfo(src) {
      let mimeType = ChromecastPlugin.mimeTypeFor(src)
      let mediaInfo = new chrome.cast.media.MediaInfo(src)
      mediaInfo.contentType = this.options.contentType || mimeType
      mediaInfo.customData = this.options.customData
      let metadata = this.createMediaMetadata()
      mediaInfo.metadata = metadata
      let tracks = this.createMediaTracks()
      mediaInfo.tracks = tracks
      return mediaInfo
    }
  
    // ---------------------------------------------
    // references:
    //   https://developers.google.com/cast/docs/chrome_sender/advanced
    // ---------------------------------------------
    createMediaTracks() {
      const tracks = []
      const langs  = [['en','English'],['sv','Swedish'],['ru','Russian'],['ja','Japanese'],['ar','Arabic']]
  
      for (let i=0; i < langs.length; i++) {
        const [language, name] = langs[i]
        const url = `https://raw.githubusercontent.com/videojs/video.js/v7.10.2/docs/examples/elephantsdream/captions.${language}.vtt`
  
        const track            = new chrome.cast.media.Track(i, chrome.cast.media.TrackType.TEXT)
        track.trackContentId   = url
        track.trackContentType = 'text/vtt'
        track.subtype          = chrome.cast.media.TextTrackType.SUBTITLES
        track.language         = language
        track.name             = name
        track.customData       = null
  
        tracks.push(track)
      }
  
      return tracks
    }

---

• having confirmed this will work, the next step was to reimplement createMediaTracks:

    createMediaTracks() {
      const textTracks = this.container.closedCaptionsTracks  // [{id, name, track}]
  
      return textTracks.map(textTrack => {
        const track            = new chrome.cast.media.Track(textTrack.id, chrome.cast.media.TrackType.TEXT)
      //track.trackContentId   = undefined
      //track.trackContentType = undefined
        track.subtype          = chrome.cast.media.TextTrackType.SUBTITLES
        track.language         = textTrack.track.language
        track.name             = textTrack.name
        track.customData       = null
  
        return track
      })
    }

• this quickly hit a roadblock
  • textTracks is obtained from the getter: closedCaptionsTracks
    • textTracks[0].track === document.querySelector('video').textTracks[0]
  • API:
    • TextTrackList
    • TextTrack
  • TextTrack API doesn't provide any way to obtain the URL from which it was loaded
    • an in-stream HLS TextTrack doesn't include the URL to its m3u8 manifest
    • an external TextTrack doesn't include the URL from its underlying <track> element

---

• thoughts..

  • external text tracks are low-hanging fruit
    • their URLs can be obtained from both settings.playback.externalTracks and DOM <track> elements
  • in-stream HLS text tracks are more complicated

• external text tracks:

  • problems:
    • mapping from this.container.closedCaptionsTrackId to the correct URL
      • this.container.closedCaptionsTrackId refers to the complete list of this.container.closedCaptionsTracks
        • if there are in-stream text tracks, then they will be included and make the mapping more complicated
    • TextTrack includes an id attribute that can be used to query the underlying <track> element, if any
      • when clappr-core creates DOM <track> element, it doesn't add any (unique) id attribute
  • observations:
    • by testing the aforementioned Clappr settings
      • settings.playback.externalTracks are always loaded BEFORE in-stream text tracks
        • if this can be guaranteed, then things become much easier

• in-stream HLS text tracks:

  • problems:
    • hlsjs-playback depends upon hls.js
      • the hls.js API fires several interesting events
        • SUBTITLE_TRACK_LOADING
          • provides the text track manifest's URL
          • hlsjs-playback doesn't add any listener
        • SUBTITLE_TRACK_LOADED
          • hlsjs-playback adds a listener, but doesn't inspect the data for its underlying URLs

---

I think that's a fairly thorough summary of where I'm at right now.

As I said, I'm going after the low-hanging fruit.. and will report back with any progress.

[warren-bank]

Bam!

imho, it works great; feedback (and PRs) welcome 😃

PS, the live demo doesn't set crossOrigin, because that would break video playback. The result is that the text tracks won't play in the web browser, but they'll play fine on Chromecast.

PS, regarding clappr#1477, the initialize_text_tracks workaround included in the example Clappr settings (above) is no-longer needed when using this enhanced plugin; it performs this task automagically (and more cleanly).