2020-12-08 20:44 阅读 3

Warp OutOfMaskSampleError

It seems that we have a bug when using the warp_to_mask() method. If both images are masked, then we almost always get an OutOfMaskSampleError. This error is raised by the sample() method of MaskedImage which checks the sampled warping points. It is not very clear if this error is raised because of numerical accuracy or not. However, it wasn't a problem in menpo0.4.4. The error is raised by both PiecewiseAffine and ThinPlateSplines.

The error can be reproduced as:

import menpo.io as mio
from menpo.transform import PiecewiseAffine, ThinPlateSplines

takeo = mio.import_builtin_asset('takeo.ppm')
takeo = takeo.as_masked()

lenna = mio.import_builtin_asset('lenna.png')
lenna = lenna.as_masked()

and then for PiecewiseAffine

pwa_lenna_to_takeo = PiecewiseAffine(lenna.landmarks['LJSON'].lms, takeo.landmarks['PTS'].lms)
warped_takeo_to_lenna_tps = takeo.warp_to_mask(lenna.mask, pwa_lenna_to_takeo)

and for ThinPlateSplines

tps_lenna_to_takeo = ThinPlateSplines(lenna.landmarks['LJSON'].lms, takeo.landmarks['PTS'].lms)
warped_takeo_to_lenna_tps = takeo.warp_to_mask(lenna.mask, tps_lenna_to_takeo)

This error also caused a bug on the instance() method of AAM and ATM from menpofit. The straightforward way to fix it is to warp after calling as_unmasked() on the image or by setting the mask all true.

However, we need to decide whether this check is important or should be removed.


  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享

4条回答 默认 最新

  • weixin_39636608 weixin_39636608 2020-12-08 20:44

    I think the bottom line is that you can't have it both ways. You either sample the mask in a way that is out of bounds in the other image (which cause unexpected behaviour as you sample bad values from the mask e.g NaN), or you raise an error when you sample out of bounds.

    The reason I added this explicit error is because it was very confusing when you sampled out of bounds because it often caused a vectorization error when building models such as AAMs. Now, at least, the error is explicit.

    I think the current behaviour is fine as long as make a small tutorial that shows people why this behaviour is the case, and that by doing .as_unmasked(copy=False) you incur almost no overhead for the warp which ignores the existing mask.

    点赞 评论 复制链接分享
  • weixin_39949889 weixin_39949889 2020-12-08 20:44

    I think the problem here arises because in the example given by Nontas you would expect both images to have a consistent mask given the one-to-one correspondence between their landmarks and the fact that their masks are theoretically masking the same equivalent domain. Because of discretization at pixel level this ends up not being true and the error is raised... I believe this check is useful but was introduced to check other situations, e.g. when both masks are not masking equivalent domains, and perhaps we could do something about the little numerical differences in this particular case.

    点赞 评论 复制链接分享
  • weixin_39636608 weixin_39636608 2020-12-08 20:44

    No, they aren't the same, because Lena and Takeo have different image sizes and different face shapes, so the masks of each image aren't equivalent. I think I will create a small visual example that demonstrates this - it's good for the documentation anyway.

    点赞 评论 复制链接分享
  • weixin_39636608 weixin_39636608 2020-12-08 20:44

    I never made that example... But the point remains, that is expected behaviour.

    点赞 评论 复制链接分享