Commits on Oct 8, 2024

1. [Proposal] Implement DASH Thumbnail tracks …

   Overview
   ========
   
   This is a feature proposal to add support for DASH thumbnail tracks as
   specified in the DASH-IF IOP 4.3 6.2.6.
   
   Those thumbnail tracks generally allow to provide previews when seeking,
   and it has been linked as such in our demo page.
   
   In a DASH MPD
   =============
   
   In a DASH MPD (its manifest file), such tracks are set as regular
   `AdaptationSet`, with an `contentType` attribute set to `"image"` and a
   specific `EssentialProperty` element.
   
   To support multiple thumbnail qualities (e.g. bigger or smaller
   thumbnails depending on the UI, the usage etc.), multiple
   `Representation` are also possible.
   
   A curiosity is that unlike for "trickmode" tracks (which also fill
   the role of optionally providing thumbnail previews in the RxPlayer,
   through our experimental `VideoThumbnailLoader` tool), thumbnail tracks
   are not linked to any video `AdaptationSet`.
   
   So if there's multiple video tracks with different content in it, I'm
   not sure of how we may be able to choose the right thumbnail track, nor
   how to communicate it through the API.
   I guess it could be communicated through a `Subset` element, as defined
   in the DASH specification to force usage of specific AdaptationSets
   together, but I never actually encountered this element in the wild and
   it doesn't seem to be supported by any player.
   
   The API
   =======
   
   Simple solution from other players
   ----------------------------------
   
   For the API, I saw that most other players do very few things. They
   generally just synchronously return the metadata on a thumbnail
   corresponding to a specified timestamp.
   
   That metadata includes the thumbnail's URL (e.g. to a jpeg), height and
   width, but also x and y coordinates as thumbnails are often in image
   sprites (images including multiple images). It is then the role of the
   application/UI to load and crop this correctly.
   
   This seems acceptable to me, after all UI developers are generally
   experienced working with images and browsers are also very efficient
   with it (e.g. doing an `<img>.src = url` vs fetching the jpeg through a
   fetch request + linking the content to the DOM), but I did want to
   explore another way for multiple reasons:
   
     1. As the core of the RxPlayer may run in another thread (in
        what we call "multithreading mode"), and as for now precize
        manifest information is only available in the WebWorker, we would
        either have to make such kind of API asynchronous (which makes it
        much less easy to handle for an application), or to send back the
        corresponding metadata to main thread (with thus supplementary
        synchronization complexities).
   
     2. As the thumbnail track is just another AdaptationSet/Representation
        in the MPD, it may be impacted in the same way by other MPD
        elements and attributes, like multiple CDNs, content steering...
   
        Though thumbnail tracks are much less critical (and they also seem
        explicitely more limited by the DASH-IF IOP than other media types),
        I have less confidence on being able to provide a stable API in
        which the RxPlayer would provide all necessary metadata to the
        application so it can load and render thumbnails, than just do the
        loading and thumbnail rendering ourselves.
   
   Solution I propose
   ------------------
   
   So I propose here two APIs:
   
   ```ts
   /**
    * Get synchronously thumbnail information for the specified time, or
    * `null` if there's no thumbnail information for that time.
    *
    * The returned metadata does not allow an application to load and
    * render thumbnails, it is mainly meant for an application to check if
    * thumbnails are available at a particular time and which qualities if
    * there's multiple ones.
    */
   getThumbnailMetadata({ time }: { time: number }): IThumbnailMetadata[] | null;
   
   /** Information returned by the `getThumbnailMetadata` method. */
   export interface IThumbnailMetadata {
     /** Identifier identifying a particular thumbnail track. */
     id: string;
     /**
      * Width in pixels of the individual thumbnails available in that
      * thumbnail track.
      */
     width: number | undefined;
     /**
      * Height in pixels of the individual thumbnails available in that
      * thumbnail track.
      */
     height: number | undefined;
     /**
      * Expected mime-type of the images in that thumbnail track (e.g.
      * `image/jpeg` or `image/png`.
      */
     mimeType: string | undefined;
   }
   ```
   Though with that API, it means that an application continuously has to
   check if there's thumbnail at each timestamp by calling again and again
   `getThumbnailMetadata` e.g. as a user moves its mouse on top of the
   seeking bar. So I'm still unsure with that part, we could also
   communicate like audio and video tracks per Period and only once.
   
   And more importantly the loading and rendering API:
   ```ts
   /**
    * Render inside the given `container` the thumbnail corresponding to the
    * given time.
    *
    * If no thumbnail is available at that time or if the RxPlayer does not succeed
    * to load or render it, reject the corresponding Promise and remove the
    * potential previous thumbnail from the container.
    *
    * If a new `renderThumbnail` call is made with the same `container` before it
    * had time to finish, the Promise is also rejected but the previous thumbnail
    * potentially found in the container is untouched.
    */
   public async renderThumbnail(options: IThumbnailRenderingOptions): Promise<void>;
   
   export interface IThumbnailRenderingOptions {
     /**
      * HTMLElement inside which the thumbnail should be displayed.
      *
      * The resulting thumbnail will fill that container if the thumbnail loading
      * and rendering operations succeeds.
      *
      * If there was already a thumbnail rendering request on that container, the
      * previous operation is cancelled.
      */
     container: HTMLElement;
     /** Position, in seconds, for which you want to provide an image thumbnail. */
     time: number;
     /**
      * If set to `true`, we'll keep the potential previous thumbnail found inside
      * the container if the current `renderThumbnail` call fail on an error.
      * We'll still replace it if the new `renderThumbnail` call succeeds (with the
      * new thumbnail).
      *
      * If set to `false`, to `undefined`, or not set, the previous thumbnail
      * potentially found inside the container will also be removed if the new
      * new `renderThumbnail` call fails.
      *
      * The default behavior (equivalent to `false`) is generally more expected, as
      * you usually don't want to provide an unrelated preview thumbnail for a
      * completely different time and prefer to display no thumbnail at all.
      */
     keepPreviousThumbnailOnError?: boolean | undefined;
     /**
      * If set, specify from which thumbnail track you want to display the
      * thumbnail from. That identifier can be obtained from the
      * `getThumbnailMetadata` call (the `id` property).
      *
      * This is mainly useful when encountering multiple thumbnail track qualities.
      */
     thumbnailTrackId?: string | undefined;
   }
   ```
   
   Basically this method checks which thumbnail to load, load it and render
   it inside the given element.
   
   For now this is done by going through a Canvas element for easy
   cropping/resizing. I could also go through an image tag and CSS but I was
   unsure of how my CSS would interact with outside CSS I do not control, so
   I chose for now the maybe-less efficient canvas way.
   
   As you can see in the method description and in its implementation,
   there's a lot of added complexities from the fact that we do not control
   the container element (the application is) and that we're doing the
   loading ourselves instead of just e.g. the browser through an image tag:
   
     - Multiple `renderThumbnail` calls may be performed in a row, in which
       case we have to cancel the previous requests to avoid rendering
       thumbnails in the wrong order.
   
     - If a new thumbnail request fails, we also have to remove the older
       thumbnail to avoid having stale data.
   
     - Because there's a lot of operations which may take some (still minor)
       time and as often thumbnails are just present in the same image
       sprite than the one before, there is a tiny cache implementation
       which handles just that case: if the previous image sprite already
       contains the right data, we do not go through the RxPlayer's core
       code (which may be in another thread) and back.
   
   Still, I find the corresponding usage by an application relatively
   simple and elegant:
   
   ```js
   rxPlayer.renderThumbnail({ time, container })
     .then(() => console.log("Thumbnail rendered!"))
     .catch((err) => {
       if (err,code !== "ABORTED") {
         console.warn("Error while loading thumbnails:", err);
       }
     );
   ```

   

   peaBerberian committed Oct 8, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for e122553
   • Browse repository at this point

   Copy the full SHA

   e122553 View commit details

   Browse the repository at this point in the history

2. Add Thumbnail tracks update

   

   peaBerberian committed Oct 8, 2024
   Configuration menu

   • View commit details
   • Copy full SHA for e811d07
   • Browse repository at this point

   Copy the full SHA

   e811d07 View commit details

   Browse the repository at this point in the history