weixin_39840729
2021-01-10 19:18 阅读 3

[WIP] ImageData alpha premultiplication

Currently, the ImageData object contains pixel data with un-premultiplied/unassociated alpha, meaning that the red, green, and blue values of each pixel are not multiplied by that pixel's alpha value.

This causes a couple of issues: - In all implementations I'm aware of, canvas data is stored as premultiplied. When CanvasRenderingContext2D.getImageData is called, each canvas pixel's RGB values must be divided by the corresponding alpha value, resulting in performance overhead (division is a relatively expensive operation). In the same way, when CanvasRenderingContext2D.putImageData is called, the image data's RGB values must be multiplied by the corresponding alpha values. - The recommended practice for alpha blending, including blending in 3D applications, is to use premultiplied alpha. If a developer is using premultiplied alpha in a WebGL context, for instance, and wishes to read pixels from the WebGL context to put onto a canvas (via readPixels and putImageData), they must take extra steps to un-premultiply the pixels they have read. - In addition, image processing/compositing algorithms that utilize the ImageData API (perhaps to display a processed image to a canvas element, or to read pixel data from one), and desire proper alpha blending, must manually multiply alpha values after reading pixels from a canvas and then manually divide them again when writing them to one, adding yet more overhead.

This proposed revision modifies the ImageData API such that CanvasRenderingContext2D.getImageData can optionally return premultiplied data, the ImageData object contains a field indicating whether it contains premultiplied data, all methods which create an ImageData object can optionally specify whether it will hold premultiplied data, and CanvasRenderingContext2D.putImageData will take into account whether the source ImageData is premultiplied:


interface mixin CanvasImageData {
  // pixel manipulation
  ImageData createImageData(long sw, long sh, optional ImageDataOptions options = {});
  ImageData createImageData(ImageData imagedata);
  ImageData getImageData(long sx, long sy, long sw, long sh, optional ImageDataOptions options = {});
  void putImageData(ImageData imagedata, long dx, long dy);
  void putImageData(ImageData imagedata, long dx, long dy, long dirtyX, long dirtyY, long dirtyWidth, long dirtyHeight);
};

dictionary ImageDataOptions {
    boolean premultipliedAlpha = false;
};

[Exposed=(Window,Worker),
 Serializable]
interface ImageData {
  constructor(unsigned long sw, unsigned long sh, optional ImageDataOptions options = {});
  constructor(Uint8ClampedArray data, unsigned long sw, optional unsigned long sh, optional ImageDataOptions options = {});

  readonly attribute unsigned long width;
  readonly attribute unsigned long height;
  readonly attribute Uint8ClampedArray data;
  readonly attribute boolean premultipliedAlpha; // (default false)
};

This behaves thusly: - The premultipliedAlpha property indicates that the ImageData's data is premultiplied, and should be treated as such when being put onto a canvas - For compatibility, all methods which construct or return an ImageData object set the premultipliedAlpha attribute to false unless options is present and its premultipliedAlpha property is true. - If it is true, though, getImageData will return image data which has premultiplied alpha, unlike the current behavior which un-premultiplies before returning it. - When you place ImageData onto a canvas with putImageData, it will check the premultipliedAlpha property and: - If premultipliedAlpha === true, use the image data as-is - If premultipliedAlpha === false, multiply the RGB values by the alpha values before putting the image data on the canvas (as is currently done)

I have also specified that a canvas' output bitmap/backing store must be premultiplied. I believe all current implementations ensure this, and all compositors should already support this in order to deal with premultiplied WebGL canvases, but I'm not 100% sure that's the case for all browsers.

A prototype implementation for Firefox is available for those who want to try out this API.

Original changes This implements [these `ImageData` API changes](https://github.com/whatwg/html/issues/5365#issuecomment-600361866), which add an optional options dictionary to the `ImageData` constructor(s), the `CanvasRenderingContext2D.createImageData` method(s), and the `CanvasRenderingContext2D.getImageData` method(s). Currently, this dictionary contains one option, `premultiplied`, which determines whether or not the pixel data contained within the `ImageData` is considered to be premultiplied. See #5365 for the motivation behind this. I have some questions about things I encountered when revising the spec: - Would it be good to elaborate somewhere on the difference between premultiplied and non-premultiplied alpha? - Currently, I'm using the same dictionary, `ImageDataOptions`, for both the methods that create `ImageData` objects (the `ImageData` constructor and `createImageData`) and those that grab existing image data (`getImageData`). One could feasibly imagine a new option that only makes sense for "grab existing data" APIs (e.g. `getImageData`)--for instance, [a `destination` option](https://github.com/whatwg/html/issues/5365#issuecomment-600127024) that writes the data to an existing `ImageData` object. Should I create two options dictionaries, `GetImageDataOptions` and `ImageDataOptions`, or does just one suffice for now? - I'm treating the `ImageDataOptions` parameter as opaque all the way into the "create an ImageData object" section, which currently reads > When the user agent is required to create an ImageData object, given a positive integer number of rows rows, a positive integer number of pixels per row pixelsPerRow, an optional Uint8ClampedArray source, *and additional ImageDataOptions options* it must run these steps: Would it make more sense to expose `premultiplied` as its own unique option there, and make it read more like: > When the user agent is required to create an ImageData object, given a positive integer number of rows rows, a positive integer number of pixels per row pixelsPerRow, an optional Uint8ClampedArray source, *and a boolean alpha premultiplication flag premultiplied* it must run these steps: - Some colors are only representable if the canvas' backing buffer is premultiplied. Should a premultiplied backing buffer be a spec requirement, or should it be left to the implementations to decide? I also have [an updated prototype implementation for Firefox](https://github.com/whatwg/html/files/4349323/premul_imagedata_v2.patch.txt).
  • [ ] At least two implementers are interested (and none opposed):
  • [ ] Tests are written and can be reviewed and commented upon at:
  • [ ] Implementation bugs are filed:
  • Chrome: …
  • Firefox: …
  • Safari: …

/acknowledgements.html ( diff ) /canvas.html ( diff )

该提问来源于开源项目:whatwg/html

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享

8条回答 默认 最新

  • weixin_39661881 weixin_39661881 2021-01-10 19:18

    Would it be good to elaborate somewhere on the difference between premultiplied and non-premultiplied alpha?

    I think so, unless canvas experts think the normative text is obviously interoperably implementable. (The normative text appears to be "Pixel values must be, or must have already been, premultiplied by alpha if options's premultiplied is true. Otherwise, they must not be premultiplied by alpha.", which to me seems pretty ambiguous. How do I know if the pixels are already premultiplied? What is "alpha"?)

    Should I create two options dictionaries, GetImageDataOptions and ImageDataOptions, or does just one suffice for now?

    Just one seems fine for now!

    Would it make more sense to expose premultiplied as its own unique option there, and make it read more like:

    I tend toward "yes". Then the dictionary type only shows up in the public API. This is just a style thing though; it's fine to pass the dictionary through the layers as you've currently done.

    Some colors are only representable if the canvas' backing buffer is premultiplied. Should a premultiplied backing buffer be a spec requirement, or should it be left to the implementations to decide?

    I don't have the canvas expertise to answer this question, but in general, if there are potential interop problems caused by leaving things to implementations, then we should not leave things to implementations. (But, my lack of expertise prevents me from knowing whether there would be interop problems from this.)

    点赞 评论 复制链接分享
  • weixin_39840729 weixin_39840729 2021-01-10 19:18

    I have a couple questions about the process going forward.

    • What's the best way to gauge/gather implementer interest? Should I simply ask around in mailing lists + Matrix chats?
    • Is there a specific order in which certain things should be done? I'm assuming something like:
    • Get implementer interest
    • Add tests
    • File bugs
    • Get review
    • Merge?
    点赞 评论 复制链接分享
  • weixin_39609650 weixin_39609650 2021-01-10 19:18

    https://whatwg.org/working-mode#changes might help. There's no specific order, it's a checklist that needs to be met before things are standardized (i.e., merged). Perhaps the comments from and in #5365 count as support from Chrome and Firefox? (But yeah, asking in various places helps. For Mozilla there's also https://github.com/mozilla/standards-positions though that's not always used for smaller API changes.) Hope that helps.

    Note that you need to use your name when signing https://participate.whatwg.org/agreement, not a pseudonym. See also https://github.com/whatwg/sg/issues/93.

    点赞 评论 复制链接分享
  • weixin_39840729 weixin_39840729 2021-01-10 19:18

    Note that you need to use your name when signing https://participate.whatwg.org/agreement, not a pseudonym. See also whatwg/sg#93.

    Is there any way to do this without having my full name linked to my username in a publicly-viewable JSON file?

    点赞 评论 复制链接分享
  • weixin_39840729 weixin_39840729 2021-01-10 19:18

    I'm going over this again and I think I'll need to tighten up the semantics/specified behavior around premultiplied alpha. I have a few questions about current implementations and how strict the specification should be, and was hoping you could provide answers to some of them and/or point me to others who may also be able to provide some useful input.

    • As far as I know, all current browsers represent canvas' backing stores in a premultiplied format. Should the specification mandate this?
    • Currently, if you write pixels into a canvas with putImageData and then read them back out with getImageData, alpha premultiplication and subsequent un-premultiplication will cause the color values to differ if the alpha < 255. With premultiplied ImageData objects, this lossy step will not occur. Should the specification mandate that putting premultiplied color values into a canvas and pulling them back out again will produce identical results, or that there is a maximum acceptable error for how much they differ? I know of at least one browser (Brave) which slightly randomizes color values returned from getImageData, but there are many web applications that could subtly break, or even become impossible to properly implement, if browsers are explicitly allowed to do this sort of thing.
    • If a canvas holds "superluminant" color values (premultiplied colors where one or more of the color channel values are greater than the alpha value), and you ask for un-premultiplied image data, how should those color values be represented? I'm leaning towards "saturating"/"clamping" behavior (if dividing a color channel value by the alpha results in a value > 255, then set the channel's value to 255); should that be mandated?
    • In general, how much of the canvas color manipulation behavior should be left to implementations? In more recent years, there have been concerns about implementation-specific behavior being an avenue for fingerprinting, and this has been demonstrated with things like canvas text rendering differences. It would also be beneficial for web developers if implementation-specific behavior was kept to a minimum so that pixel manipulation is less fraught with potential differences across platforms. However, certain platforms may also have specific limitations that could prevent them from implementing canvas pixel manipulation behavior as described in the specification. How much detail should the specification go into in order to balance these needs?
    点赞 评论 复制链接分享
  • weixin_39601511 weixin_39601511 2021-01-10 19:18
    1. We should spec that backing stores for canvas2d RCs (not canvas in general) are alpha-premult.
    2. That behavior should implicitly handled by (1). For fuzzed outputs, we can test that results are within a range.
    3. IIRC this is undefined results right now, and should be discussed separately from this bug IMO.
    4. This also sounds like scope creep we should punt on here for now.
    点赞 评论 复制链接分享
  • weixin_39840729 weixin_39840729 2021-01-10 19:18
    1. That behavior should implicitly handled by (1). For fuzzed outputs, we can test that results are within a range.

    If it is implicitly handled, is there a downside to having it be explicitly handled, just so no browsers get the idea to do something subtly different?

    1. IIRC this is undefined results right now, and should be discussed separately from this bug IMO.

    My understanding is that it's undefined because up until this proposal there's been no existing code path that allows you to put superluminant pixels on a canvas, and thus that state was unreachable and didn't need to be defined.

    I know you can read superluminant pixels from a WebGL context with readPixels but those are always read as premultiplied, never converted. Is there a way to hit an "unpremultiply superluminant pixel" code path currently?

    点赞 评论 复制链接分享
  • weixin_39601511 weixin_39601511 2021-01-10 19:18
    1. That behavior should implicitly handled by (1). For fuzzed outputs, we can test that results are within a range.

    If it is implicitly handled, is there a downside to having it be _ex_plicitly handled, just so no browsers get the idea to do something subtly different?

    I don't think implementations can get this wrong if they do (1) right, so no. Just test it, worry about redundantly speccing it.

    1. IIRC this is undefined results right now, and should be discussed separately from this bug IMO.

    My understanding is that it's undefined because up until this proposal there's been no existing code path that allows you to put superluminant pixels on a canvas, and thus that state was unreachable and didn't need to be defined.

    I know you can read superluminant pixels from a WebGL context with readPixels but those are always read as premultiplied, never converted. Is there a way to hit an "unpremultiply superluminant pixel" code path currently?

    WebGL with premultAlpha:true can send 1/0/0/0.5 premult data into drawImage, yeah. We see browser differences in compositing there too, but IIRC it'd be fairly prohibitively slow to standardize, or even to detect in the general case.

    点赞 评论 复制链接分享

相关推荐