Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make RTCEncodedVideoFrame and RTCEncodedAudioFrame transferable #238

Closed
wants to merge 3 commits into from
Closed

Make RTCEncodedVideoFrame and RTCEncodedAudioFrame transferable #238

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ae121de
Make RTCEncodedVideoFrame and RTCEncodedAudioFrame transferable
jan-ivar Feb 3, 2025
c073c94
fix bikeshed errors
jan-ivar Feb 3, 2025
f250dc5
Remove ? in [[?FOO]] as it is not (any longer?) valid bikeshed syntax.
jan-ivar Feb 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 55 additions & 26 deletions index.bs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,10 +142,10 @@ The <dfn abstract-op>writeEncodedData</dfn> algorithm is given a |rtcObject| as
1. Set |rtcObject|.`[[lastReceivedFrameCounter]]` to |frame|`[[counter]]`.
1. Let |data| be |frame|.`[[data]]`.
1. Let |serializedFrame| be [$StructuredSerializeWithTransfer$](|frame|, « |data| »).
1. Let |frameCopy| be [$StructuredDeserialize$](|serializedFrame|, |frame|'s [=relevant realm=]).
1. Enqueue |frameCopy| for processing as if it came directly from the encoded data source, by running one of the following steps:
* If |rtcObject| is a {{RTCRtpSender}}, enqueue |frameCopy| to |rtcObject|'s packetizer, to be processed [=in parallel=].
* If |rtcObject| is a {{RTCRtpReceiver}}, enqueue |frameCopy| it to |rtcObject|'s decoder, to be processed [=in parallel=].
1. Let |transferredFrame| be [$StructuredDeserializeWithTransfer$](|serializedFrame|, |frame|'s [=relevant realm=]).
1. Enqueue |transferredFrame| for processing as if it came directly from the encoded data source, by running one of the following steps:
* If |rtcObject| is a {{RTCRtpSender}}, enqueue |transferredFrame| to |rtcObject|'s packetizer, to be processed [=in parallel=].
* If |rtcObject| is a {{RTCRtpReceiver}}, enqueue |transferredFrame| it to |rtcObject|'s decoder, to be processed [=in parallel=].
1. Return [=a promise resolved with=] undefined.

On sender side, as part of [$readEncodedData$], frames produced by |rtcObject|'s encoder MUST be enqueued in |rtcObject|.`[[readable]]` in the encoder's output order.
Expand Down Expand Up @@ -371,7 +371,7 @@ dictionary RTCEncodedVideoFrameMetadata {
<dd>
<p>
An identifier for the encoded frame, monotonically increasing in decode order. Its lower
16 bits match the frame_number of the AV1 Dependency Descriptor Header Extension defined in Appendix A of [[?AV1-RTP-SPEC]], if present.
16 bits match the frame_number of the AV1 Dependency Descriptor Header Extension defined in Appendix A of [[AV1-RTP-SPEC]], if present.
Only present for received frames if the Dependency Descriptor Header Extension is present.
</p>
</dd>
Expand All @@ -381,7 +381,7 @@ dictionary RTCEncodedVideoFrameMetadata {
<dd>
<p>
List of frameIds of frames this frame references.
Only present for received frames if the AV1 Dependency Descriptor Header Extension defined in Appendix A of [[?AV1-RTP-SPEC]] is present.
Only present for received frames if the AV1 Dependency Descriptor Header Extension defined in Appendix A of [[AV1-RTP-SPEC]] is present.
</p>
</dd>
<dt>
Expand Down Expand Up @@ -451,7 +451,7 @@ dictionary RTCEncodedVideoFrameOptions {

// New interfaces to define encoded video and audio frames. Will eventually
// re-use or extend the equivalent defined in WebCodecs.
[Exposed=(Window,DedicatedWorker), Serializable]
[Exposed=(Window,DedicatedWorker), Serializable, Transferable]
interface RTCEncodedVideoFrame {
constructor(RTCEncodedVideoFrame originalFrame, optional RTCEncodedVideoFrameOptions options = {});
readonly attribute RTCEncodedVideoFrameType type;
Expand All @@ -460,7 +460,7 @@ interface RTCEncodedVideoFrame {
};
</pre>

### Constructor ### {#RTCEncodedVideoFrame-members}
### Constructor ### {#RTCEncodedVideoFrame-constructor}
<dl dfn-for="RTCEncodedVideoFrame" class="dictionary-members">
<dt>
<dfn for="RTCEncodedVideoFrame" method>constructor()</dfn>
Expand Down Expand Up @@ -506,11 +506,13 @@ interface RTCEncodedVideoFrame {
{{RTCEncodedVideoFrameMetadata/mimeType}}.
For <a href="https://w3c.github.io/webrtc-svc/">SVC</a>, each spatial layer
is transformed separately.
<p class="note">
</p>
<p class="note">
Since packetizers may drop certain elements, e.g. AV1 temporal delimiter OBUs,
the input to an receive-side transform may be different from the output of
a send-side transform.
</p>
</p>
<p>
The following table gives a number of examples:
</p>
<table class="simple">
Expand Down Expand Up @@ -580,26 +582,41 @@ interface RTCEncodedVideoFrame {
</dd>
</dl>

### Serialization ### {#RTCEncodedVideoFrame-serialization}
### Transfer and Serialization ### {#RTCEncodedVideoFrame-serialization}

{{RTCEncodedVideoFrame}} objects are [=serializable objects=] and [=transferable objects=].

{{RTCEncodedVideoFrame}} objects are serializable objects [[HTML]].
Their [=serialization steps=], given |value|, |serialized|, and |forStorage|, are:

1. If |forStorage| is true, then throw a {{DataCloneError}}.
1. Set |serialized|.`[[type]]` to the value of |value|.{{RTCEncodedVideoFrame/type}}
1. Set |serialized|.`[[type]]` to the value of |value|.{{RTCEncodedVideoFrame/type}}.
1. Set |serialized|.`[[metadata]]` to an internal representation of |value|'s metadata.
1. Set |serialized|.`[[data]]` to |value|.`[[data]]`
1. Set |serialized|.`[[data]]` to a copy of |value|.`[[data]]`.

Their [=deserialization steps=], given |serialized|, |value| and |realm|, are:

1. Set |value|.{{RTCEncodedVideoFrame/type}} to |serialized|.`[[type]]`
1. Set |value|'s metadata to the platform object representation of |serialized|.`[[metadata]]`
1. Set |value|.{{RTCEncodedVideoFrame/type}} to |serialized|.`[[type]]`.
1. Set |value|'s metadata to the platform object representation of |serialized|.`[[metadata]]`.
1. Set |value|.`[[data]]` to |serialized|.`[[data]]`.

Their [=transfer steps=], given |value| and |dataHolder|, are:

1. Set |dataHolder|.`[[type]]` to the value of |value|.{{RTCEncodedVideoFrame/type}}.
1. Set |dataHolder|.`[[metadata]]` to an internal representation of |value|'s metadata.
1. Set |dataHolder|.`[[data]]` to |value|.`[[data]]`.
1. Unset |value|.`[[data]]`.

Their [=transfer-receiving steps=], given |dataHolder| and |value|, are:

1. Set |value|.{{RTCEncodedVideoFrame/type}} to |dataHolder|.`[[type]]`
1. Set |value|'s metadata to the platform object representation of |dataHolder|.`[[metadata]]`.
1. Set |value|.`[[data]]` to |dataHolder|.`[[data]]`.

<p class="note">
The internal form of a serialized RTCEncodedVideoFrame is not observable;
it is defined chiefly so that it can be used with frame cloning in the
[$writeEncodedData$] algorithm and in the {{WindowOrWorkerGlobalScope/structuredClone()}} operation.
transfer is used to hand off frames in the
[$writeEncodedData$] algorithm, and serialization is used in the
{{WindowOrWorkerGlobalScope/structuredClone()}} operation.
An implementation is therefore free to choose whatever method works best.
</p>

Expand Down Expand Up @@ -681,15 +698,15 @@ dictionary RTCEncodedAudioFrameOptions {
RTCEncodedAudioFrameMetadata metadata;
};

[Exposed=(Window,DedicatedWorker), Serializable]
[Exposed=(Window,DedicatedWorker), Serializable, Transferable]
interface RTCEncodedAudioFrame {
constructor(RTCEncodedAudioFrame originalFrame, optional RTCEncodedAudioFrameOptions options = {});
attribute ArrayBuffer data;
RTCEncodedAudioFrameMetadata getMetadata();
};
</pre>

### Constructor ### {#RTCEncodedAudioFrame-members}
### Constructor ### {#RTCEncodedAudioFrame-constructor}
<dl dfn-for="RTCEncodedAudioFrame" class="dictionary-members">
<dt>
<dfn for="RTCEncodedAudioFrame" method>constructor()</dfn>
Expand Down Expand Up @@ -806,20 +823,32 @@ interface RTCEncodedAudioFrame {
</dl>


### Serialization ### {#RTCEncodedAudioFrame-serialization}
### Transfer and Serialization ### {#RTCEncodedAudioFrame-serialization}

{{RTCEncodedAudioFrame}} objects are [=serializable objects=] and [=transferable objects=].

{{RTCEncodedAudioFrame}} objects are serializable objects [[HTML]].
Their [=serialization steps=], given |value|, |serialized|, and |forStorage|, are:

1. If |forStorage| is true, then throw a {{DataCloneError}}.
1. Set |serialized|.`[[metadata]]` to an internal representation of |value|'s metadata.
1. Set |serialized|.`[[data]]` to |value|.`[[data]]`
1. Set |serialized|.`[[data]]` to a copy of |value|.`[[data]]`

Their [=deserialization steps=], given |serialized|, |value| and |realm|, are:

1. Set |value|'s metadata to the platform object representation of |serialized|.`[[metadata]]`
1. Set |value|.`[[data]]` to |serialized|.`[[data]]`.

Their [=transfer steps=], given |value| and |dataHolder|, are:

1. Set |dataHolder|.`[[metadata]]` to an internal representation of |value|'s metadata.
1. Set |dataHolder|.`[[data]]` to |value|.`[[data]]`.
1. Unset |value|.`[[data]]`.

Their [=transfer-receiving steps=], given |dataHolder| and |value|, are:

1. Set |value|'s metadata to the platform object representation of |dataHolder|.`[[metadata]]`.
1. Set |value|.`[[data]]` to |dataHolder|.`[[data]]`.

## Interfaces ## {#RTCRtpScriptTransformer-interfaces}
<pre class="idl">
[Exposed=DedicatedWorker]
Expand Down Expand Up @@ -867,9 +896,9 @@ The <dfn constructor for="RTCRtpScriptTransform" lt="RTCRtpScriptTransform(worke
6. Let |serializedReadable| be the result of [$StructuredSerializeWithTransfer$](|t1|.`[[readable]]`, « |t1|.`[[readable]]` »).
7. Let |serializedWritable| be the result of [$StructuredSerializeWithTransfer$](|t2|.`[[writable]]`, « |t2|.`[[writable]]` »).
8. [=Queue a task=] on the DOM manipulation [=task source=] |worker|'s global scope to run the following steps:
1. Let |transformerOptions| be the result of [$StructuredDeserialize$](|serializedOptions|, the current Realm).
2. Let |readable| be the result of [$StructuredDeserialize$](|serializedReadable|, the current Realm).
3. Let |writable| be the result of [$StructuredDeserialize$](|serializedWritable|, the current Realm).
1. Let |transformerOptions| be the result of [$StructuredDeserializeWithTransfer$](|serializedOptions|, the current Realm).
2. Let |readable| be the result of [$StructuredDeserializeWithTransfer$](|serializedReadable|, the current Realm).
3. Let |writable| be the result of [$StructuredDeserializeWithTransfer$](|serializedWritable|, the current Realm).
4. Let |transformer| be a new {{RTCRtpScriptTransformer}}.
5. Set |transformer|.`[[options]]` to |transformerOptions|.
6. Set |transformer|.`[[readable]]` to |readable|.
Expand Down