weixin_39840729
2021-01-10 19:31 阅读 4

ImageData alpha premultiplication

This is outdated! See #5371 for current proposal

Currently, the ImageData object contains pixel data with un-premultiplied/unassociated alpha, meaning that the red, green, and blue values of each pixel are not multiplied by that pixel's alpha value.

This causes a couple of issues: - In all implementations I'm aware of, canvas data is stored as premultiplied. When CanvasRenderingContext2D.getImageData is called, each canvas pixel's RGB values must be divided by the corresponding alpha value, resulting in performance overhead (division is a relatively expensive operation). In the same way, when CanvasRenderingContext2D.putImageData is called, the image data's RGB values must be multiplied by the corresponding alpha values. - The recommended practice for alpha blending, including blending in 3D applications, is to use premultiplied alpha. If a developer is using premultiplied alpha in a WebGL context, for instance, and wishes to read pixels from the WebGL context to put onto a canvas (via readPixels and putImageData), they must take extra steps to un-premultiply the pixels they have read. - In addition, image processing/compositing algorithms that utilize the ImageData API (perhaps to display a processed image to a canvas element, or to read pixel data from one), and desire proper alpha blending, must manually multiply alpha values after reading pixels from a canvas and then manually divide them again when writing them to one, adding yet more overhead.

It would be helpful to add a premultiplied property to ImageData objects:


interface ImageData {
  constructor(unsigned long sw, unsigned long sh, optional boolean premultiplied);
  constructor(Uint8ClampedArray data, unsigned long sw, optional unsigned long sh, optional boolean premultiplied);

  readonly attribute unsigned long width;
  readonly attribute unsigned long height;
  readonly attribute Uint8ClampedArray data;
  attribute boolean premultiplied; // (default false)
};

interface mixin CanvasImageData {
  // pixel manipulation
  ImageData createImageData(long sw, long sh, optional boolean premultiplied);
  ImageData createImageData(ImageData imagedata);
  ImageData getImageData(long sx, long sy, long sw, long sh, optional boolean premultiplied);
  void putImageData(ImageData imagedata, long dx, long dy);
  void putImageData(ImageData imagedata, long dx, long dy, long dirtyX, long dirtyY, long dirtyWidth, long dirtyHeight);
};

The way I could see this behaving is: - The premultiplied value indicates that the ImageData's data is premultiplied, and should be treated as such when being put onto a canvas - For compatibility, all methods which construct or return an ImageData object set the premultiplied attribute to false unless the optional premultiplied argument is true. - If it is true, though, return image data which has premultiplied alpha - When you place this ImageData onto a canvas with putImageData, it will check the premultiplied property and: - If premultiplied == true and the canvas' backing store is premultiplied, use the image data as-is - If premultiplied == true and the canvas' backing store is not premultiplied, divide the RGB values by the alpha values before putting the image data onto the canvas - If premultiplied == false and the canvas' backing store is premultiplied, multiply the RGB values by the alpha values before putting the image data on the canvas - If premultiplied == false and the canvas' backing store is not premultiplied, use the image data as-is - You can set the premultiplied property of an ImageData object after constructing it-- this only changes the way it will behave in future putImageData calls, and does not modify the data in any way.

These APIs are quite rough, but hopefully they convey the intended functionality.

该提问来源于开源项目:whatwg/html

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享

19条回答 默认 最新

  • weixin_39661881 weixin_39661881 2021-01-10 19:31

    /cc /canvas

    点赞 评论 复制链接分享
  • weixin_39604350 weixin_39604350 2021-01-10 19:31

    One other request we've also been consistently getting is the ability to pass an existing ImageData to getImageData to be updated instead of creating a new one.

    点赞 评论 复制链接分享
  • weixin_39840729 weixin_39840729 2021-01-10 19:31

    I've created a quick-and-dirty Firefox patch for those who want to play around with the API. It's rough around the edges but might be helpful for prototyping.

    点赞 评论 复制链接分享
  • weixin_39601511 weixin_39601511 2021-01-10 19:31

    It's an interesting choice for premultiplied to be non-readonly, but I think that might be fine. Overall this is a good idea.

    点赞 评论 复制链接分享
  • weixin_39661881 weixin_39661881 2021-01-10 19:31

    API design critique: avoid boolean arguments and use options dictionaries instead.

    Is there a strong motivation to allow changing premultiplied after the fact? Otherwise keeping ImageData objects immutable would be better.

    点赞 评论 复制链接分享
  • weixin_39840729 weixin_39840729 2021-01-10 19:31

    I suppose it would be nicer to make the premultiplication state immutable. If a developer needs to change it, they can always create a new ImageData with a different premultiplication state that shares the same underlying buffer.

    Maybe an options dictionary could look something like:

    
    enum AlphaAssociation { "straight", "premultiplied" };
    
    dictionary ImageDataOptions {
      AlphaAssociation alphaAssociation = "straight";
    };
    

    Not so sure about the name-- would simply alpha be better? This could also open the door to an "opaque" AlphaAssociation, which I could see representing data laid out as RGBRGB... with no alpha values (although reinterpreting that buffer as alpha-containing data would be a whole different can of worms).

    点赞 评论 复制链接分享
  • weixin_39661881 weixin_39661881 2021-01-10 19:31

    Sorry, to be clear a boolean inside an options dictionary is fine. (Although maybe an enum is nicer, I dunno, that's up to you.) The principle is to avoid new ImageData(10, 10, true) and instead have new ImageData(10, 10, { premultiplied: true }). new ImageData(10, 10, { alphaAssociation: "premultiplied" }) may also be nice; I don't have a preference between those two.

    点赞 评论 复制链接分享
  • weixin_39601511 weixin_39601511 2021-01-10 19:31

    I think a bool is better than alphaAssociation. It's less clear to me now than it was before. We should use existing terms of art.

    点赞 评论 复制链接分享
  • weixin_39604350 weixin_39604350 2021-01-10 19:31

    I agree the bool is better.

    点赞 评论 复制链接分享
  • weixin_39840729 weixin_39840729 2021-01-10 19:31

    So the consensus seems to be on something like:

    
    dictionary ImageDataOptions {
        boolean premultiplied = false;
    };
    
    interface ImageData {
      constructor(unsigned long sw, unsigned long sh, optional ImageDataOptions options = {});
      constructor(Uint8ClampedArray data, unsigned long sw, optional unsigned long sh, optional ImageDataOptions options = {});
    
      readonly attribute unsigned long width;
      readonly attribute unsigned long height;
      readonly attribute Uint8ClampedArray data;
      attribute boolean premultiplied; // (default false)
    };
    
    
    interface mixin CanvasImageData {
      // pixel manipulation
      ImageData createImageData(long sw, long sh, optional ImageDataOptions options = {});
      ImageData createImageData(ImageData imagedata);
      ImageData getImageData(long sx, long sy, long sw, long sh, optional ImageDataOptions options = {});
      void putImageData(ImageData imagedata, long dx, long dy);
      void putImageData(ImageData imagedata, long dx, long dy, long dirtyX, long dirtyY, long dirtyWidth, long dirtyHeight);
    };
    

    Thoughts on this?

    点赞 评论 复制链接分享
  • weixin_39601511 weixin_39601511 2021-01-10 19:31

    I like the ImageDataOptions and ImageData, but CanvasImageData should be a different proposal.

    IE think I see what you're wanting to do with CanvasImageData instead of using a canvas2d context, but you'll need to compel the usecase for this better.

    点赞 评论 复制链接分享
  • weixin_39601511 weixin_39601511 2021-01-10 19:31

    ^

    点赞 评论 复制链接分享
  • weixin_39840729 weixin_39840729 2021-01-10 19:31

    The CanvasImageData interface already exists-- I only modified the createImageData and getImageData signatures to use the new ImageDataOptions.

    点赞 评论 复制链接分享
  • weixin_39601511 weixin_39601511 2021-01-10 19:31

    Oh sorry, I forgot about that weird canvas mixin thing. (I find its name confusing) Sounds good to me.

    点赞 评论 复制链接分享
  • weixin_39723441 weixin_39723441 2021-01-10 19:31

    Nice proposal. The fact that the new attribute is visible on the ImageData interface means that this is easily feature-detectable and won't run into the same problems as #4248 .

    Two comments:

    1) It would be ideal if the name "premultipliedAlpha" could be used in order to have parity with the same-named WebGL context creation attribute: https://www.khronos.org/registry/webgl/specs/latest/1.0/#WEBGLCONTEXTATTRIBUTES .

    2) It would be preferred to have the premultipliedAlpha attribute be immutable on the ImageData. Simply reinterpreting the contents of the Uint8Array will almost never work correctly if alpha != 255. This means that changing the attribute from false to true or vice versa essentially requires updating the contents of the Uint8Array, and operations like this should be restricted to the construction of a new ImageData. Further, it should be specified that constructing an ImageData from another one where the premultipliedAlpha creation attribute differs is a lossy operation.

    点赞 评论 复制链接分享
  • weixin_39840729 weixin_39840729 2021-01-10 19:31

    Further, it should be specified that constructing an ImageData from another one where the premultipliedAlpha creation attribute differs is a lossy operation.

    How can such an operation occur? Passing an existing ImageData object to createImageData returns a new ImageData filled with transparent black, and creating an ImageData object which shares its buffer with another should, in my opinion, simply reinterpret the data.

    点赞 评论 复制链接分享
  • weixin_39723441 weixin_39723441 2021-01-10 19:31

    Further, it should be specified that constructing an ImageData from another one where the premultipliedAlpha creation attribute differs is a lossy operation.

    How can such an operation occur? Passing an existing ImageData object to createImageData returns a new ImageData filled with transparent black, and creating an ImageData object which shares its buffer with another should, in my opinion, simply reinterpret the data.

    Sorry, my mistake, I didn't remember how createImageData(ImageData) worked. You're right, since the spec says that the only way to create an ImageData initially containing some externally-produced data is by passing in Uint8ClampedArray to the ImageData constructor, it won't / can't modify the data.

    Making the premultipliedAlpha attribute read-only still makes this proposal easier to reason about in my opinion. Reinterpreting a previously non-premultipliedAlpha ImageData as premultipledAlpha is confusing.

    点赞 评论 复制链接分享
  • weixin_39601511 weixin_39601511 2021-01-10 19:31

    I sort of like that it's not readonly, since that lets you "recast" it without having to copy or something. It's lesser usecase, but I think it's real.

    点赞 评论 复制链接分享
  • weixin_39840729 weixin_39840729 2021-01-10 19:31

    I sort of like that it's not readonly, since that lets you "recast" it without having to copy or something. It's lesser usecase, but I think it's real.

    Even if premultipliedAlpha is immutable, you can still do something like:

    js
    const premultiplied = new ImageData(1000, 1000, {premultipliedAlpha: true});
    
    const nonPremultiplied = new ImageData(
      premultiplied.data,
      premultiplied.width,
      premultiplied.height,
      {premultipliedAlpha: false}
    );
    

    to "recast" the data.

    点赞 评论 复制链接分享

相关推荐