RFC: Media Use Cases in Amplify

[Amplifiyer]

This is a Request For Comments (RFC). RFCs are intended to elicit feedback regarding a proposed change to the Amplify Framework. Please feel free to post comments or questions here.

This RFC is to gather customer feedback for a new proposed functionality in Amplify related to media use cases. In the next section, we have listed some use cases and we would like to hear from Amplify customers which one of these you would want to see in Amplify and if there are any use cases that are not covered below.

Please comment with your feedback such as

• How and which particular use case below will be helpful to you?
• What are you current paint points when working with online audio/video platforms and real time communications?
• Are there any use cases that are missing here that Amplify should consider?

Motivation

Developers require deep audio/video streaming expertise to compose streaming services on AWS. The deceptively simple task of hosting a audio/video for viewers introduces immediate friction for developers as they need to understand media players, processing, and content management. To make video experiences intelligent and engaging, developers also require a deep understanding of analytics engines, machine learning, and interactive APIs.

Similarly developing applications for real time communication require low level understanding of webRTC, signaling and media relay and forwarding services.

1. Media Use Cases

1.1 Live Video Streaming

“1 to many” audio/video streaming (Conferences/Broadcast). It need not be real time (> 2-3 seconds) but high throughput (in thousands or millions of viewers) Few examples for this use case are

1. Live streaming of events such as sports, game, concerts, weddings
2. Live Podcasts, talk shows
3. Instructor led workouts

Question: Do you want capabilities to create new live streams with in the app, or as an output of a playback/resource creation workflow that can then be hard-coded in your apps for playback?
Think of it as S3 bucket. Some developers would like a programmatic way to create buckets while others would create one and use it for their entire app (which is what Amplify supports as of now)

• B2B customers might want to support creating streams at runtime.
• B2C customers will probably want one stream and reusing it (avoids using API/DataStore category or similar to communicate the playback URL)

1.2 Video On Demand

A VOD streaming link that is available after uploading a video (e.g. to an S3 bucket) and supports codecs or formats: A, B and C (e.g. HLS) that can be viewed on variety of devices and network connections.
Highly requested use case by developers who are or support content creators. Content creators would prefer to host and serve their content (Live video included) by themselves to avoid paying fees to other big platforms and monetize their content themselves. e.g.

1. Streaming and transactional based VODs
2. Ad based VODs
3. Recorded workouts, educational and training material.
4. Video assets for web or apps such as house videos in real estate, product videos in eCommerce, trailers/demo for paid video content.
5. Lifecycle policy on these video assets (like S3)

1.3 Multi party audio/video calls

“1 to 1” or “Many to many” audio/video calls (Group conferences, group chats) where at least N participants can share videos simultaneously and communicate with each other in real time. Some examples for this use case are

1. Multi party audio/video meetings/calls like chime.
2. Secure 1:1 conversations → Tele-health (between clinician and patient), private discussions between client and financial/tax advisors etc.
3. Virtual teaching/classroom → e.g. Teacher (Host) see all children(Viewer) and all children see Teacher but not each other.
4. eCommerce → Video chat with a customer service agent to resolve issues, mechanic help me fix my car, give live demos to customers before they buy products.
5. Streaming from security cameras with 2-way audio conversation, smart home use cases.

1.4 Chat

Peer to peer or group chat including text and media (Images/video).

1. Group or 1-1 social media chat.
2. Customer/Product service → Chat with a customer service agent to resolve issues.

1.5 Security

1. Permissions or access control per stream/video asset.
2. DRM support for VOD and live streaming (all the way to player SDKs)
3. Permissions or access control for RTC channels (group video chats/calls)
4. E2E encryption for chats and audio/video calls.

2. Complementary Use Cases

2.1 CMS for media assets (videos, streams and meetings)

1. Manage (read/search and update) streams/videos or meetings that have been created/ingested or are live.

2.2 Attach metadata to media assets

1. Live comments for live streaming
2. Comments for VODs
3. Reactions (likes / up-votes / favorites etc)

2.3 Record live video streams

Recording the live streams or meetings and storing in S3 such that they are accessible as VOD later on or for archival purposes.

2.4 Record Analytics

Record analytics information such as

1. number of views (Play/pause/stop)
2. Resolution streamed
3. Buffering
4. clicks/loads
5. length watched
6. Reactions

2.5 AI/Predictions

1. Generate cc (closed captions) automatically using AWS Transcribe on Live and VODs
2. Analyze videos (e.g. filter adult videos or inappropriate videos) using AWS Rekognition
3. Recommend similar VODs as the one being watched or viewed

2.6 Share videos/audio/playlists (even images and documents)

Securely sharing media across users. User A uploads a media object which they share with User B and only User B should be able to access the media object. Further User B may or may not have access to share it to User C.

3. UI Framework SDKs

3.1 Media Player SDKs (Media display)

Configurable native player that can play the live streams and VODs in native frameworks.

1. Native controls such as ⏯, ⏪, ⏮, ⏩, ⏭ for live streaming and VODs
2. Extensible to add reactions/comments/chat

3.2 Streamer SDKs (Ingestion)

1. Enable customers to live stream from their cameras on computer and mobile phones.
2. Make existing streamer applications compatible with Amplify such as OBS
3. Screen Sharing/Streaming capabilities in the SDKs

3.3 Multi-participant video call SDKs

A kind of combination of the above two where customers stream as well as receive from other participants in real time.

1. Multi-tile view of meeting participants for video calls
2. Capture current user’s audio/video and/or screen
3. Extensible to add text chat

4. Current solutions to AWS customers

4.1 Kinesis Video Streams

Kinesis Video Streams(KVS) offers two media solutions, one using a low level media streams and another with webRTC signaling and peer to peer connections.

4.2 AWS Chime SDK

What is Amazon Chime? - Amazon Chime
Amazon Chime is a communications service that transforms online meetings with an application that is secure and comprehensive. Amazon Chime works across your devices so that you can stay connected. You can use Amazon Chime for online meetings, video conferencing, calls, and chat. You can also share content inside and outside of your organization. Amazon Chime helps you to work productively from anywhere.

4.3 Amazon Interactive Video Service (IVS)

IVS provides a managed and hosted end to end live streaming functionality over rtmps protocol. The service provides a video player SDK in all platforms (Web/android/iOS and video.js and JWPlayer integration as well). Streaming is done via RTMPS url and any popular streaming software. No SDKs are provided for streaming OOTB.

4.4 AWS Elemental Services

A set of Media services to create very specialized and highly configurable media workflows for live streaming and VOD.

4.5 Amplify Video Plugin

A plugin for amplify CLI currently maintained by AWS Solutions Architects and enables automated resource creation for VOD and live streaming backed by IVS and AWS Elemental Services.

[kylekirkby]

+1 on the captions support :)

[starpebble]

+1

[djsjr]

Building a flutter app using Amplify and I need VOD capability to stream videos. Most desired (simplest) solution would be the Amplify video plugin but it is not available for flutter as far as I know. I'd like to see that some time soon if possible.

EDIT: It IS available for flutter, but still being developed. A more straightforward way to protect content would be great.

[Thiago-LM]

Would love to see Amazon Polly integration in the Flutter Amplify. Right now I am using it from Android Amplify as per my use case right only includes Android (but may also get to iOS in the future) and it would be great to have to have a Neural TTS option, as the native TTS Android offers may be not so user-friendly.

[codercatdev]

4.1 is the most interesting to me. Building an application like StreamYard that allows multiple users to connect using Kinesis with WebRTC then combining these into a suitable single stream that is then broadcast using Kinesis media streams.

This is an extremely exciting area that you can find more on in the labs https://github.com/awslabs/amplify-video

[wayfarer3130]

For 1.2, video on demand, having a way for the front end to automatically select the right format, and know what formats are available would be very helpful. Also, a lambda method to convert data formats from one type to another, with a bunch of pre-provided methods already available would be helpful.
The particular requirement for the viewer I'm working on is that types that the viewer can handle are provided in an accept header, and then the response format needs to be chosen from the actually available types. For example:
Accept: video/mp4
Accept: video/H265
Accept: video/jxl
where only H265 was available would stream the video/H265.

It would also be nice to be able to encapsulate the response automatically as multipart/related in order to provide additional information - for example:
Accept: multipart/related; contentType=video/H265
would return a multipart/related header with the metadata about the study, followed by the actual video content.

Response type determination should be generic based on the overall type, so for example:
Accept: image/jpeg
Accept: image/jxl
should automatically choose between the two types based on what is available.

[MorCohenAres]

+1 for VOD, especially the ability to control the lifecycle.
Our team has a production setup using the amplify-category-video plugin and that's very helpful.
Sadly, it isn't maintained anymore by the original maintainer.
@Amplifiyer We're interested in contributing to the amplify-category-video repo to keep it maintained at least until this RFC progresses. Already got some pending ideas for changes.
Is that possible somehow? Maybe there is someone to contact in AWS?
This simple PR for example is left with no review for weeks 😞

[abdallahshaban557]

Thank you all for the feedback! We are closing this RFC now.