Skip to content

Commit

Permalink
[Proposal] Implement DASH Thumbnail tracks
Browse files Browse the repository at this point in the history
Overview
========

This is a feature proposal to add support for DASH thumbnail tracks as
specified in the DASH-IF IOP 4.3 6.2.6.

Those thumbnail tracks generally allow to provide previews when seeking,
and it has been linked as such in our demo page.

In a DASH MPD
=============

In a DASH MPD (its manifest file), such tracks are set as regular
`AdaptationSet`, with an `contentType` attribute set to `"image"` and a
specific `EssentialProperty` element.

To support multiple thumbnail qualities (e.g. bigger or smaller
thumbnails depending on the UI, the usage etc.), multiple
`Representation` are also possible.

A curiosity is that unlike for "trickmode" tracks (which also fill
the role of optionally providing thumbnail previews in the RxPlayer,
through our experimental `VideoThumbnailLoader` tool), thumbnail tracks
are not linked to any video `AdaptationSet`.

So if there's multiple video tracks with different content in it, I'm
not sure of how we may be able to choose the right thumbnail track, nor
how to communicate it through the API.
I guess it could be communicated through a `Subset` element, as defined
in the DASH specification to force usage of specific AdaptationSets
together, but I never actually encountered this element in the wild and
it doesn't seem to be supported by any player.

The API
=======

Simple solution from other players
----------------------------------

For the API, I saw that most other players do very few things. They
generally just synchronously return the metadata on a thumbnail
corresponding to a specified timestamp.

That metadata includes the thumbnail's URL (e.g. to a jpeg), height and
width, but also x and y coordinates as thumbnails are often in image
sprites (images including multiple images). It is then the role of the
application/UI to load and crop this correctly.

This seems acceptable to me, after all UI developers are generally
experienced working with images and browsers are also very efficient
with it (e.g. doing an `<img>.src = url` vs fetching the jpeg through a
fetch request + linking the content to the DOM), but I did want to
explore another way for multiple reasons:

  1. As the core of the RxPlayer may run in another thread (in
     what we call "multithreading mode"), and as for now precize
     manifest information is only available in the WebWorker, we would
     either have to make such kind of API asynchronous (which makes it
     much less easy to handle for an application), or to send back the
     corresponding metadata to main thread (with thus supplementary
     synchronization complexities).

  2. As the thumbnail track is just another AdaptationSet/Representation
     in the MPD, it may be impacted in the same way by other MPD
     elements and attributes, like multiple CDNs, content steering...

     Though thumbnail tracks are much less critical (and they also seem
     explicitely more limited by the DASH-IF IOP than other media types),
     I have less confidence on being able to provide a stable API in
     which the RxPlayer would provide all necessary metadata to the
     application so it can load and render thumbnails, than just do the
     loading and thumbnail rendering ourselves.

Solution I propose
------------------

So I propose here two APIs:

```ts
/**
 * Get synchronously thumbnail information for the specified time, or
 * `null` if there's no thumbnail information for that time.
 *
 * The returned metadata does not allow an application to load and
 * render thumbnails, it is mainly meant for an application to check if
 * thumbnails are available at a particular time and which qualities if
 * there's multiple ones.
 */
getThumbnailMetadata({ time }: { time: number }): IThumbnailMetadata[] | null;

/** Information returned by the `getThumbnailMetadata` method. */
export interface IThumbnailMetadata {
  /** Identifier identifying a particular thumbnail track. */
  id: string;
  /**
   * Width in pixels of the individual thumbnails available in that
   * thumbnail track.
   */
  width: number | undefined;
  /**
   * Height in pixels of the individual thumbnails available in that
   * thumbnail track.
   */
  height: number | undefined;
  /**
   * Expected mime-type of the images in that thumbnail track (e.g.
   * `image/jpeg` or `image/png`.
   */
  mimeType: string | undefined;
}
```
Though with that API, it means that an application continuously has to
check if there's thumbnail at each timestamp by calling again and again
`getThumbnailMetadata` e.g. as a user moves its mouse on top of the
seeking bar. So I'm still unsure with that part, we could also
communicate like audio and video tracks per Period and only once.

And more importantly the loading and rendering API:
```ts
/**
 * Render inside the given `container` the thumbnail corresponding to the
 * given time.
 *
 * If no thumbnail is available at that time or if the RxPlayer does not succeed
 * to load or render it, reject the corresponding Promise and remove the
 * potential previous thumbnail from the container.
 *
 * If a new `renderThumbnail` call is made with the same `container` before it
 * had time to finish, the Promise is also rejected but the previous thumbnail
 * potentially found in the container is untouched.
 */
public async renderThumbnail(options: IThumbnailRenderingOptions): Promise<void>;

export interface IThumbnailRenderingOptions {
  /**
   * HTMLElement inside which the thumbnail should be displayed.
   *
   * The resulting thumbnail will fill that container if the thumbnail loading
   * and rendering operations succeeds.
   *
   * If there was already a thumbnail rendering request on that container, the
   * previous operation is cancelled.
   */
  container: HTMLElement;
  /** Position, in seconds, for which you want to provide an image thumbnail. */
  time: number;
  /**
   * If set to `true`, we'll keep the potential previous thumbnail found inside
   * the container if the current `renderThumbnail` call fail on an error.
   * We'll still replace it if the new `renderThumbnail` call succeeds (with the
   * new thumbnail).
   *
   * If set to `false`, to `undefined`, or not set, the previous thumbnail
   * potentially found inside the container will also be removed if the new
   * new `renderThumbnail` call fails.
   *
   * The default behavior (equivalent to `false`) is generally more expected, as
   * you usually don't want to provide an unrelated preview thumbnail for a
   * completely different time and prefer to display no thumbnail at all.
   */
  keepPreviousThumbnailOnError?: boolean | undefined;
  /**
   * If set, specify from which thumbnail track you want to display the
   * thumbnail from. That identifier can be obtained from the
   * `getThumbnailMetadata` call (the `id` property).
   *
   * This is mainly useful when encountering multiple thumbnail track qualities.
   */
  thumbnailTrackId?: string | undefined;
}
```

Basically this method checks which thumbnail to load, load it and render
it inside the given element.

For now this is done by going through a Canvas element for easy
cropping/resizing. I could also go through an image tag and CSS but I was
unsure of how my CSS would interact with outside CSS I do not control, so
I chose for now the maybe-less efficient canvas way.

As you can see in the method description and in its implementation,
there's a lot of added complexities from the fact that we do not control
the container element (the application is) and that we're doing the
loading ourselves instead of just e.g. the browser through an image tag:

  - Multiple `renderThumbnail` calls may be performed in a row, in which
    case we have to cancel the previous requests to avoid rendering
    thumbnails in the wrong order.

  - If a new thumbnail request fails, we also have to remove the older
    thumbnail to avoid having stale data.

  - Because there's a lot of operations which may take some (still minor)
    time and as often thumbnails are just present in the same image
    sprite than the one before, there is a tiny cache implementation
    which handles just that case: if the previous image sprite already
    contains the right data, we do not go through the RxPlayer's core
    code (which may be in another thread) and back.

Still, I find the corresponding usage by an application relatively
simple and elegant:

```js
rxPlayer.renderThumbnail({ time, container })
  .then(() => console.log("Thumbnail rendered!"))
  .catch((err) => {
    if (err,code !== "ABORTED") {
      console.warn("Error while loading thumbnails:", err);
    }
  );
```
  • Loading branch information
peaBerberian committed Sep 4, 2024
1 parent 19c66e6 commit fe207f0
Show file tree
Hide file tree
Showing 48 changed files with 1,851 additions and 292 deletions.
212 changes: 212 additions & 0 deletions demo/scripts/components/ThumbnailPreview.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
import * as React from "react";
import useModuleState from "../lib/useModuleState";
import { IPlayerModule } from "../modules/player";
import { IThumbnailMetadata } from "../../../src/public_types";

const DIV_SPINNER_STYLE = {
backgroundColor: "gray",
position: "absolute",
width: "100%",
height: "100%",
opacity: "50%",
display: "flex",
justifyContent: "center",
alignItems: "center",
} as const;

const IMG_SPINNER_STYLE = {
width: "50%",
margin: "auto",
} as const;

export default function ThumbnailPreview({
xPosition,
time,
player,
showVideoThumbnail,
}: {
player: IPlayerModule;
xPosition: number | null;
time: number;
showVideoThumbnail: boolean;
}): JSX.Element {
const videoThumbnailLoader = useModuleState(player, "videoThumbnailLoader");
const videoElement = useModuleState(player, "videoThumbnailsElement");
const imageThumbnailElement = useModuleState(player, "imageThumbnailContainerElement");
const parentElementRef = React.useRef<HTMLDivElement>(null);
const [shouldDisplaySpinner, setShouldDisplaySpinner] = React.useState(true);
const ceiledTime = Math.ceil(time);

// Insert the div element containing the image thumbnail
React.useEffect(() => {
if (showVideoThumbnail) {
return;
}

if (parentElementRef.current !== null) {
parentElementRef.current.appendChild(imageThumbnailElement);
}
return () => {
if (
parentElementRef.current !== null &&
parentElementRef.current.contains(imageThumbnailElement)
) {
parentElementRef.current.removeChild(imageThumbnailElement);
}
};
}, [showVideoThumbnail]);

// OR insert the video element containing the thumbnail
React.useEffect(() => {
if (!showVideoThumbnail) {
return;
}
if (videoElement !== null && parentElementRef.current !== null) {
parentElementRef.current.appendChild(videoElement);
}
return () => {
if (
videoElement !== null &&
parentElementRef.current !== null &&
parentElementRef.current.contains(videoElement)
) {
parentElementRef.current.removeChild(videoElement);
}
};
}, [videoElement, showVideoThumbnail]);

React.useEffect(() => {
if (!showVideoThumbnail) {
return;
}
player.actions.attachVideoThumbnailLoader();
return () => {
player.actions.dettachVideoThumbnailLoader();
};
}, [showVideoThumbnail]);

// Change the thumbnail when a new time is wanted
React.useEffect(() => {
let spinnerTimeout: number | null = null;
let loadThumbnailTimeout: number | null = null;

startSpinnerTimeoutIfNotAlreadyStarted();

// load thumbnail after a 40ms timer to avoid doing too many requests
// when the user quickly moves its pointer or whatever is calling this
loadThumbnailTimeout = window.setTimeout(() => {
loadThumbnailTimeout = null;
if (showVideoThumbnail) {
if (videoThumbnailLoader === null) {
return;
}
videoThumbnailLoader
.setTime(ceiledTime)
.then(hideSpinner)
.catch((err) => {
if (
typeof err === "object" &&
err !== null &&
(err as Partial<Record<string, unknown>>).code === "ABORTED"
) {
return;
} else {
hideSpinner();

// eslint-disable-next-line no-console
console.error("Error while loading thumbnails:", err);
}
});
} else {
const metadata = player.actions.getThumbnailMetadata(ceiledTime);
const thumbnailTrack = metadata.reduce((acc: IThumbnailMetadata | null, t) => {
if (acc === null || acc.height === undefined) {
return t;
}
if (t.height === undefined) {
return acc;
}
if (acc.height > t.height) {
return t.height > 100 ? t : acc;
} else {
return acc.height > 100 ? acc : t;
}
}, null);
if (thumbnailTrack === null) {
hideSpinner();
return;
}
player.actions
.renderThumbnail(ceiledTime, thumbnailTrack.id)
.then(hideSpinner)
.catch((err) => {
if (
typeof err === "object" &&
err !== null &&
(err as Partial<Record<string, unknown>>).code === "ABORTED"
) {
return;
} else {
hideSpinner();
// eslint-disable-next-line no-console
console.warn("Error while loading thumbnails:", err);
}
});
}
}, 30);

return () => {
if (loadThumbnailTimeout !== null) {
clearTimeout(loadThumbnailTimeout);
}
hideSpinner();
};

/**
* Display a spinner after some delay if `stopSpinnerTimeout` hasn't been
* called since.
* This function allows to schedule a spinner if the request to display a
* thumbnail takes too much time.
*/
function startSpinnerTimeoutIfNotAlreadyStarted() {
if (spinnerTimeout !== null) {
return;
}

// Wait a little before displaying spinner, to
// be sure loading takes time
spinnerTimeout = window.setTimeout(() => {
spinnerTimeout = null;
setShouldDisplaySpinner(true);
}, 150);
}

/**
* Hide the spinner if one is active and stop the last started spinner
* timeout.
* Allow to avoid showing a spinner when the thumbnail we were waiting for
* was succesfully loaded.
*/
function hideSpinner() {
if (spinnerTimeout !== null) {
clearTimeout(spinnerTimeout);
spinnerTimeout = null;
}
setShouldDisplaySpinner(false);
}
}, [ceiledTime, videoThumbnailLoader, parentElementRef]);

return (
<div
className="thumbnail-wrapper"
style={xPosition !== null ? { transform: `translate(${xPosition}px, -136px)` } : {}}
ref={parentElementRef}
>
{shouldDisplaySpinner ? (
<div style={DIV_SPINNER_STYLE}>
<img src="./assets/spinner.gif" style={IMG_SPINNER_STYLE} />
</div>
) : null}
</div>
);
}
152 changes: 0 additions & 152 deletions demo/scripts/components/VideoThumbnail.tsx

This file was deleted.

12 changes: 12 additions & 0 deletions demo/scripts/contents.ts
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,12 @@ const DEFAULT_CONTENTS: IDefaultContent[] = [
transport: "dash",
live: false,
},
{
name: "Live with thumbnail track",
url: "https://livesim2.dashif.org/livesim2/testpic_2s/Manifest_thumbs.mpd",
transport: "dash",
live: true,
},
{
name: "Axinom CMAF multiple Audio and Text tracks Tears of steel",
url: "https://media.axprod.net/TestVectors/Cmaf/clear_1080p_h264/manifest.mpd",
Expand Down Expand Up @@ -64,6 +70,12 @@ const DEFAULT_CONTENTS: IDefaultContent[] = [
transport: "dash",
live: true,
},
{
name: "VOD with thumbnail track",
url: "https://dash.akamaized.net/akamai/bbb_30fps/bbb_with_tiled_thumbnails.mpd",
transport: "dash",
live: false,
},
{
name: "Super SpeedWay",
url: "https://playready.directtaps.net/smoothstreaming/SSWSS720H264/SuperSpeedway_720.ism/Manifest",
Expand Down
Loading

0 comments on commit fe207f0

Please sign in to comment.