The API provides info on smooth, supported and powerEfficient. If some of the info needed to respond on all dimensions is missing, what happens? For example, if all I want is to know if a scalabilityMode is supported, and I omit width, height, bitrate and framerate, it may not be possible to determine whether powerEfficient is supported.

Some context for the discussion. In typical video codecs, the definitions of "levels" (which is signaled in the MIME codecs sub-parameter) already includes some limits for width/height/framerate. For example, in AV1 levels, level 5.0 indicates:

This means, for example, that at this level, if the width of the image is 8192, the height cannot be greater than 1088, and at this resolution, the display frame rate cannot be greater than 30fps. Obviously, if you decrease width or height, you can increase the maximum frame rate.

The same is true for bitrate. Levels define a maximum bitrate. For example, for AV1 level 5.0, the max bitrate for Main tier is 30Mbps (or 100Mbps for High Tier).

I made this simple web page to test the Media Capabilities API in various browsers.

In my tests, like @aboba said, most of the time supported does not depend on width, height, bitrate, and framerate.
For example, I get the following image with Chrome on a Pixel phone, when selecting AVC High Level 4 @ 30fps, 100kbps, max resolution 16K:

Each small rectangle corresponds to a different resolution within the given max resolution. Green means powerEfficient=true, black means powerEfficient=false.
Smoothness can also be impacted but it harder to demonstrate because some browser implementations are currently optimistic about the smoothness unless proven wrong. Some maintain that by buckets (Chrome that depend on resolution and frame rate.

However, I've seen cases where even supported depends on the resolution. For example, on Safari, when setting the VP9 codecs, 120fps, 16K max res, 100kbps, on my Mac, it gives the following:

White rectangles means supported/smooth/powerEfficient. Yellow means supported/powerEfficient. Black means not supported, not smooth, not efficient.
As you can see, the limits seem to depend on the value of width * height.

Hi @cconcolato, thanks for the simple web page.
I am trying out the decodingInfo() api.
How is power efficiency calculated?

As all of you are saying, the reason to have width, height and framerate is to be able to provide a value for smooth and potentially also powerEfficient. Although there is a relation between codec level and these values it's not uniquely defined and it also differs between codecs. Explicitly specifying width, height, and framerate when needed makes the API easier to use in my opinion.

From my understanding, the concern that was raised in the WG meeting, which is also in line with @aboba's comment, is if the API should be possible to use without these values if the user only wants to know if a certain codec and scalabilityMode is supported.

In Chromium powerEfficient is more or less the same as the codec being supported in hardware.

In Chromium powerEfficient is more or less the same as the codec being supported in hardware.
@drkron thanks for the info.
Curious to know if there are any plans to include ABR specific info related to per-segment decoding power drawn or
per-segment residual power when there are multiple tracks with codecs that are smooth, supported and powerEfficient.
Also, any plans to to take into consideration the impact of:

• resolution/content type
• frame rate
• codec + encoding quality
• display technologies
• LTE/Wifi
  that might also impact power efficiency?

I agree w/ @cconcolato and @drkron. Let me try to answer @aboba's question directly.

@aboba wrote

If some of the info needed to respond on all dimensions is missing, what happens? For example, if all I want is to know if a scalabilityMode is supported, and I omit width, height, bitrate and framerate, it may not be possible to determine whether powerEfficient is supported.

Any of codec profile, width, height, and scalabilityMode/spatialScalability may break support for hardware encoding/decoding (powerEfficient). They may break support entirely in the event that you lack a software fallback for this codec and combination of parameters. Profile, width, height, framerate and bitrate are considered for performance (smooth) predictions (aside: Chrome currently ignores considers bitrate). If we lack any of these pieces we can't provide smoothness/power info, so we throw a TypeError.

Given the above, it's important to provide a real width and height alongside scalabilityMode, even if you're not concerned about power efficiency or smoothness. Looking just at 'supported', you could get away with providing arbitrary values for framerate and bitrate. But most folks probably should care about smoothness and power efficiency (optimize the user experience) , so I recommend providing real values for those parameters too.

In the WebRTC meeting it was pointed out that bitrate and framerate are not explicitly controlled by the RTC app (the UA adapts), so it was unclear what users should use. I recommend giving values that represent the best-case scenario for your codec and resolution (i.e. assuming no CPU throttling and network congestion). If you find the config is predicted as smooth=false, try a lower resolution or a different codec.

realted: @drkron, I think the spec should clarify the meaning of width and height when scalabilityMode involves spatial scalability. My leaning is to say this is the width and height for the highest spatial layer (max). WDYT?

Curious to know if there are any plans to include ABR specific info related to per-segment decoding power drawn or
per-segment residual power when there are multiple tracks with codecs that are smooth, supported and powerEfficient.

No plans. UAs typically don't have access to this sort of granular power info.

Also, any plans to to take into consideration the impact of:

Partially answered earlier up. For display tech, LTE, ... no plans. Basically powerEfficient = "hardware acceleration", but we avoided calling it that because the power draw for hardware acceleration vs software codecs is roughly equal at lower resolutions for well optimized software codecs. For example, Chrome has plans to return powerEfficient=true for software vp9 decode at <= 360p.

I think the spec should clarify the meaning of width and height when scalabilityMode involves spatial scalability.
My leaning is to say this is the width and height for the highest spatial layer (max). WDYT?

That's the implicit assumption I would do. Sounds good to clarify that.

It seems possible for scalabilityMode support to depend on width, height and framerate. For example, a spatial scalability mode might only be supported in software. So if queried with a higher resolution/framerate than the software encoder could support, the result might be "not supported" whereas it would be "supported" with a lower resolution/framerate.

Summarizing current state here: I think we're all in agreement. Work that remains is minor updates to non-normative parts of spec to clarify the points below. I'll send a PR (or review one if someone beats me to it :)).

That's the implicit assumption I would do. Sounds good to clarify that.

Ack. Will do.

It seems possible for scalabilityMode support to depend on width, height and framerate.

I agree. Spec could do a better job of calling this out.

Related: w3c/webrtc-svc#49

Discussed in 12 April 2022 Media WG meeting: https://www.w3.org/2022/04/12-mediawg-minutes.html#t02

We've found cases with bluetooth headsets somehow modifying available audio encoder bitrates on macOS, leading to the default MediaRecorder audio bitrate not being accepted sometimes. It should prove a case for impact on supported based on bitrate.

There are multiple issues being discussed here, meaning it's unclear what milestone to assign to this issue or how to resolve it. I suggest moving the discussion of new features like bitrate to a new issue with use cases, and we can discuss scoping there.

To close this issue, I will prepare a PR to clarify the spec per #192 (comment)

Correction to the above. I think the ask is not for a new feature, but for Chrome to consider bitrate when reporting supported, per the following:

(aside: Chrome currently ignores considers bitrate)

Probably better to file in the Chromium tracker to see how feasible that would be for the headset scenario?

#236 should address width and height. Should we also add text about framerate for temporal layers?