SW + Content-DPR: enabling correct intrinsic image size calculations

[igrigorik]

I'm seeing a bunch of img/picture experiments popping up that are using SW to rewrite requests on the fly to fetch optimized image assets, and there is a subtle gotcha that most of them will quickly run into...

<img src="opera-1x.jpg" alt="The Opera House" srcset="opera-2x.jpg 2x, opera-3x.jpg 3x">

UA "remembers" which DPR variant it picks when request is initiated and uses said value to determine the "intrinsic size" once the response is available - e.g. when rendering a 3x resource it needs to "scale down" the received image by a factor of 3, otherwise you'll get a really big and really blurry image.

However, if SW wants to synthesize / serve cached image response.. it doesn't know what DPR value UA is expecting (it only knows the request URL), and it can't tell the UA what asset DPR its returning either - e.g. was this request for a 2x asset request, and what if I'm offline but only have 1x asset that I want to return?

Alternatively, consider the case where we want to use SW to retrofit existing img markup to be DPR friendly - same problem. SW can rewrite requests and fetch correct DPR asset, but it can't communicate the intended DPR value to the UA.. which will result in UA rendering the image with incorrect dimensions.

---

tl;dr: I ran into same problem when working on Client Hints, and proposed solution is to add support for Content-DPR header, which would allow servers + SW to indicate the intended image DPR: http://igrigorik.github.io/http-client-hints/#rfc.section.5.1

We have a prototype implementation in Chrome (slightly outdated, use "DPR" header):

• Enable chrome://flags/#enable-experimental-web-platform-features
• Launch Chrome with --enable-client-hints flag

While this is not a SW "bug", I want to raise this here as I'm sure it'll come up as a gotcha once more people start playing with SW. Should we make a push to ship above header as a standalone feature?

P.S. Landing support for Content-DPR would effectively mean that you can implement Client Hints via ServiceWorker, which is rather nice!

[annevk]

Content-DPR specifically should be done as a feature for <img> elements and such. However, a generic solution would be exposing the Request object in some fashion so developers can add Content-DPR themselves (or similar such headers). Even more generic would be adding some kind of data to Request objects that is structured cloned across environments. Though perhaps headers are sufficient in that regard. (The service worker could create a new request that does not have them if they are not intended to hit the network.)

[igrigorik]

@annevk s/Request/Response/, right? Content-DPR needs to be set on the response to indicate to the UA that it should adjust its image processing logic.

Also, there is the case of fetching a binary blob (with XHR or fetch) and feeding said data to new img element... it seems like we need a way to query+set the DPR value on an img? /cc @yoavweiss

[annevk]

If it is on the response, it seems a service worker could already set such a header.

[igrigorik]

Thinking about this some more, it seems like we need something similar to backingStorePixelRatio for img: ability to query it and override it from JS, plus Content-DPR header to automatically initialize it from response. The use cases are:

• Client has a data blob and wants to use createObjectUrl to inject an image: it may not know the image dimensions, but it may know the DPR. It should be possible to communicate this to the UA.
• Server is responding with an optimized image variant and needs to communicate its DPR ratio, which may be different from what browser thinks it is. The Content-DPR header would initialize the backing store pixel ratio.

@yoavweiss sanity check? If that doesn't crazy, should probably move this discussion to whatwg, or some other place?

[annevk]

Initializing an <img> from a Response would make more sense. That way you also cover potential future extensions.

[igrigorik]

Moving discussion to ricg repo: ResponsiveImagesCG/picture-element#252