weixin_39821604
2021-01-09 20:03 阅读 3

Memory leak when setting source from imageData

I'm trying to get a "stream" of base64 encoded jpeg images drawn on a canvas with some effects.

Approach 1

I first tried to set the targets source property to a image element:

 js
var seriously = new Seriously();
var target = seriously.target('#image-canvas');
seriously.go();

var image = new Image();
target.source = image;

function onRecieveImage(data) {
  image.src = 'data:image/jpeg;base64,' + data;
}

The onRecieveImage is called around 15 times/s.

This only showed the first loaded image. If i added the image to the page i could see the live stream in the img tag.

Approach 2

I'm drawing the image on a second canvas and then setting the imageData from that canvas to my seriously target:

 js
var canvas = document.getElementById('image-canvas');

var helperCanvas = document.createElement('canvas');
var helperContext = helperCanvas.getContext('2d');
helperCanvas.width = canvas.width;
helperCanvas.height = canvas.height;

var seriously = new Seriously();
var target = seriously.target(canvas);
seriously.go();

var image = new Image();
image.onload = function() {
  helperContext.drawImage(this, 0, 0);
  var imageData = helperContext.getImageData(0, 0, helperCanvas.width, helperCanvas.height);
  target.source = imageData;
};

function onRecieveImage(data) {
  image.src = 'data:image/jpeg;base64,' + data;
}

This works as intended, however this approach introduces a major memory leak, and Chrome crashes within a few minutes.

When taking heap snapshots in the Chrome dev tools i can see a raise in memory usage by around 0.1mb every 2-4 second until Chrome crashes.

Removing the target.source = imageData; inside the onload function and showing the helperCanvas on the document will not introduce any memory leaks.

该提问来源于开源项目:brianchirls/Seriously.js

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享

7条回答 默认 最新

  • weixin_39880623 weixin_39880623 2021-01-09 20:03

    Thanks for writing in such detail, Christofer. These are two separate issues, so I'll take them one at a time.

    Approach 1

    Seriously does not update a source node after changing the src of the image. But it should, so I'll fix that. Shouldn't be too hard.

    Approach 2

    When you set target.source to the ImageData object, it implicitly creates a new source node there. And when you set it to another ImageData object, it does not destroy the original node, because it's impossible for Seriously to know if you might need it again. For example:

     javascript
        var seriously = new Seriously();
        var target = seriously.target(canvas);
        target.source = helperContext.getImageData(0, 0, helperCanvas.width, helperCanvas.height);
        var firstSourceNode = target.source; //returns a node
        target.source = helperContext.getImageData(0, 0, helperCanvas.width, helperCanvas.height);
        var secondSourceNode = target.source; //returns a node
    
        //We need the original node again
        target.source = firstSourceNode;
    

    So you've got a whole bunch of source nodes, each with its very own ImageData, GPU texture and FBO. If you really need to do it this way, you want to destroy old source nodes when you're done with them. Like this:

     javascript
        if (target.source) {
            // release GPU resources and source media (video, canvas, ImageData, array, whatever)
            // for garbage collection
            target.source.destroy();
        }
        target.source = imageData;
    

    Generally, using ImageData is almost never the best option. You're usually better off just building the source node based on the canvas. And remember to call .update() on that node before you draw it, because Seriously doesn't have any other way to know when a canvas is ready for a redraw.

    点赞 评论 复制链接分享
  • weixin_39821604 weixin_39821604 2021-01-09 20:03

    Thanks Brian for the fast reply! And thanks for an amazing plugin!

    I just tried out your suggestion with using a canvas based source node and calling the update method after drawing to my canvas. It's been running for around 30minutes now and the heap snapshots are steadily at 6.2MB!

    The solution would definitely be a lot cleaner if the source node was updated when the image src was updated. Are you creating a new issue for that so i can subscribe for updates? :)

    I'm posting my working code for anybody else who might run into the same problem:

     js
    var canvas = document.getElementById('image-canvas');
    var seriously = new Seriously();
    var target = seriously.target(canvas);
    
    // A canvas which will be updated every time the image is loaded
    var sourceCanvas = document.createElement('canvas');
    var sourceContext = sourceCanvas.getContext('2d');
    sourceCanvas.width = canvas.width;
    sourceCanvas.height = canvas.height;
    target.source = sourceCanvas;
    
    // An image which will act as the reciever of the stream. The
    // image will be drawn onto the sourceCanvas.
    var image = new Image();
    image.onload = function() {
      sourceContext.drawImage(this, 0, 0);
      target.source.update();
    };
    
    seriously.go();
    
    function onRecieveImage(data) {
      image.src = 'data:image/jpeg;base64,' + data;
    }
    
    点赞 评论 复制链接分享
  • weixin_39821604 weixin_39821604 2021-01-09 20:03

    I've been experiment a bit more now, and i just realized i could do the exact same thing with an source node based on an image as with the canvas.

    This removes alot of overhead and makes it alot cleaner. If you're making the source node update when the image has changed, it should probably be in the onload event to always get the latest frame.

    Heres my updated code for anyone curious:

     js
    var seriously = new Seriously();
    var target = seriously.target('#image-canvas');
    
    // An image which will act as the reciever of the stream.
    // Everytime a new image is loaded we will update the source
    // node.
    var image = new Image();
    image.onload = function() {
      target.source.update();
    };
    target.source = image;
    
    seriously.go();
    
    function onRecieveImage(data) {
      image.src = 'data:image/jpeg;base64,' + data;
    }
    
    点赞 评论 复制链接分享
  • weixin_39880623 weixin_39880623 2021-01-09 20:03

    Sweet. If you went 30 minutes without any detectable memory growth, then maybe I can close #17.

    Yeah, I'll make a new issue for the fix, and I'll reference this ticket, so you'll hopefully get a notification. Thanks again for your contributions.

    Care to tell us what you're building?

    点赞 评论 复制链接分享
  • weixin_39821604 weixin_39821604 2021-01-09 20:03

    Yeah, i'm pretty sure you can close #17. I had it going for over an hour with no growth in memory usage, but i've only tested it in latest Chrome.

    I'm building sort of a simple greenscreen Android app, but only using the phone as a remote camera. So i've got a webserver and a websocket server in the app, which i host the website on and transfer camera preview images to the browser.

    When the images reach the browser, i apply the chroma key effect using the code posted above (the code above is without the chroma effect just for keeping the example simple, but except that its the exact same code).

    When the preview looks good, i can capture a photo using camera controls on the website, which will be saved to the external memory card of the phone, transmitted via a websocket to the website, and lastly, a hidden seriously canvas will apply the same chroma key effect to the full-size image.

    点赞 评论 复制链接分享
  • weixin_39880623 weixin_39880623 2021-01-09 20:03

    That makes a lot of sense. I find that my mobile phone cameras have been much better than built-in webcams, not to mention portability. I would probably like to play with that.

    Have you considered using WebRTC peer connection for preview instead of streaming over websockets? Might be a lot harder to implement, but i would think the latency would be much lower.

    点赞 评论 复制链接分享
  • weixin_39821604 weixin_39821604 2021-01-09 20:03

    Yeah, the resulting image quality is extremely good. I was pretty impressed that Chrome managed to handle the 8mp image without any issues.

    Thats a very good idea! However the bottleneck at the moment is getting the camera preview in a usable format. I'm currently grabbing a bitmap from the surface the camera preview is drawn onto, which brings the whole preview down to 15fps.

    If i've got more time i'll probably implement something like grabbing the raw preview image and process that in a different thread, which should speed things up alot.

    The latency from the phone to the browser via the websocket is pretty much instant. However i'm only on a 480x320 resolution and the camera supports up to 1920x1080 at 30fps. So if i'm able to get the preview into a working image format without slowing the phone down, i'll have to look into WebRTC!

    点赞 评论 复制链接分享

相关推荐