Improved video handling

[rofe]

Problem

Today, authors can upload videos and animations as MP4 using the Helix Slack Bot and reference them as links in the source document. The block code is responsible for turning the link into a video and source element:

<a href="/media_0123456789.mp4">/media_0123456789.mp4</a>

turns into

<video autoplay loop>
  <source src="./media_0123456789.mp4" type="video/mp4">
</video>

If they want to provide multiple variants, e.g. for streaming to different devices, lower quality for slower bandwidths etc, they have to create all the variants themselves using desktop or cloud-based tools and upload each variant separately, keep track of the resulting URLs and then provide a bunch of links in the source document. The complexity of the resulting video element is on the block developer. Gathering all "video-like" links in a block/context and turning them into source elements in a single video element is tedious and error-prone:

<video>
  <source src="/media_9876543210.webm" type="video/webm">
  <source src="/media_5678901234.m3u8" type="application/vnd.apple.mpegURL">
  <source src="/media_0123456789.mp4" type="video/mp4">
</video>

There's also no easy way to specify different quality images based on media queries without making the block content in the source really complex and technical.

Proposed solution

Helix should absorb a lot of the complexity that comes with videos:

1. The author uploads a high-res video via Helix Slack Bot
2. Helix auto-generates a reasonable selection of variants with different compression levels as well as streaming containers (WebM, HLS) using AWS MediaConvert
3. Helix Slack Bot returns a single media URLs as identifier for the entire collection, which the author can add to the source document
4. The block developer can rely on a documented variety of selectors and extensions to craft the video element they need, e.g.:
   • /media_0123456789.mp4 (default, possibly capped at a maximum bandwidth)
   • /media_0123456789.webm (WebM for streaming to Android)
   • /media_0123456789.m3u8 (HLS for streaming to iOS, containing references to single .ts chunks under the same media ID)
   • all of the above could also be made available as lower (band)width variants when called with a .low and/or .medium selector, so developers could taylor source elements with srcset attributes combined with media queries to their specific use cases (short autoplaying animations, long streaming tutorials, etc).

Next up: Helix Pipeline

As a future enhancement, once we've gathered sufficient experience and confidence, similar to what we do for images rendered as pictures, we could have the Helix Pipeline render a reasonable video element which the developer could further decorate to their liking, e.g.

<video>
  <source src="./media_0123456789.webm" type="video/webm">
  <source src="./media_0123456789.m3u8" type="application/vnd.apple.mpegURL">
  <source src="./media_0123456789.mp4" type="video/mp4">
  <source src="./media_0123456789.low.webm" type="video/webm" media="(max-width: 400px)">
  <source src="./media_0123456789.low.m3u8" type="application/vnd.apple.mpegURL" media="(max-width: 400px)">
  <source src="./media_0123456789.low.mp4" type="video/mp4"media="(max-width: 400px)">
</video>

References

• https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Video_and_audio_content

[bdelacretaz]

Helix auto-generates a reasonable selection of variants...

Note that Asset Compute, which is very (hopefully completely) decoupled from AEM, does exactly that. Alex Klimetschek would have all the required information.

[trieloff]

I'd second that. For the https://github.com/adobe/helix-assets-ui prototype we need to connect Asset Compute/NUI anyway so that we can retire https://github.com/adobe/helix-asset-ingestor

[tripodsan]

but how easy is it for non AEM customers to use asset compute? AFAIK, NUI doesn't support HLS for example
/cc @cgoudie @alexkli ?

how easy would it be to control Asset Compute/NUI from an AWS service and read/write the HLS chunks back to s3, using our content encoded format? (i.e. without using Adobe I/O Runtime :-)

[alexkli]

Sorry, missed the github notification here.

Asset Compute is available on IO Developer Console, using technical accounts (per client) as credential to invoke the service. It depends on how we'd want to integrate that into a Helix flow how to best setup this.

Yes, I believe HLS is not supported yet. But could be added I assume.

Without using Adobe I/O Runtime

That's not possible, especially video encoding needs big containers that run for longer (currently up to 1 hour), which isn't possible on AWS Lambda.

[tripodsan]

4. The block developer can rely on a documented variety of selectors and extensions to craft the video element they need, e.g.:
   /media_0123456789.mp4 (default, possibly capped at a maximum bandwidth)
   /media_0123456789.webm (WebM for streaming to Android)
   /media_0123456789.m3u8 (HLS for streaming to iOS, containing references to single .ts chunks under the same media ID)

note, that each file will have a different ID. the extension is not part of the blob key, and the blob key is content encoded. so I would not use the same ID for the example here, otherwise it's confusing.

[rofe]

the extension is not part of the blob key

Can we change this? My goal here was to relieve the author of the burden of juggling multiple video URLs, and also avoid making the source document look weird by having to paste more than 1 URL per video...

[tripodsan]

hmm. we could argue, that all others files are derived from the original file, hence when the original changes, the derived ones change too... so yes. we could change this. but we would need a completely new addressing scheme, as currently the edge is computing the key based on the url (ignoring the extension).

something like:

./media_0123456789/default.webm
./media_0123456789/default.m3u8
./media_0123456789/default.mp4
./media_0123456789/low.webm
./media_0123456789/low.m3u8
./media_0123456789/low.mp3

the HLS parts could still follow the original content addressed storage

[trieloff]

See also https://github.com/adobe/helix-asset-ingestor/issues/8

[rofe]

After some more web research, I think HLS is ultimately way to go. It appears to provide everything for streaming to the greatest variety of modern browsers and devices, and supports a master playlist with different dimensions so the browser can decide to up- or downgrade based on bandwidth.

Sources:

• https://www.dacast.com/blog/hls-streaming-protocol/
• https://www.heartinternet.uk/blog/best-practices-for-optimising-video-streams-on-the-web/