weixin_39877504
weixin_39877504
2021-01-12 07:27

Premature end of JPEG file not reported correctly.

I have a problem with JPEG files that have been truncated (or not correctly sent) from my camera to my FTP server. Not all JPEG pictures are corrupted, but I want to detect the problem in VIPS. According to the documentation for vips_jpegload(), it should be possible to set "fail" to TRUE, so that, to quote: "the JPEG reader fail on any warnings. This can be useful for detecting truncated files". Unfortunately, this does not seem to be the case as the test program below and attached test picture show. When failis set to FALSE, then vips reports a problem on the command line. As far as I can see, the report is only produced when vips_shutdown() is called. When failis set to TRUE, then no report is produced from VIPS.

I have found a possible workaround that I will try - to examine the contents of vips_error_buffer() after the first operation on the image - avg() (which I do anyway). I can understand, that due to the way that VIPS caches operations on an image, the problem with the truncated file is not noticed until after the return from VImage::new_from_file(). It would however be nice to have a way of detecting such problems earlier or having them signalled with an exception.

Best wishes

David Singleton


/*
 * vipstestbadfiles_main.cpp
 *
 *  Created on: 23.06.2016
 *      Author: David Singleton
 */
#include <stdio.h>
#include <unistd.h>
#include <string>
#include <vips>

#define RC_OK 0

using namespace vips;

int main(int argv, char ** argc) {
   printf("vipstestbadfiles_main started\n");
   int iRc = vips_init(argc[0]);
   if (RC_OK != iRc) {
      throw VError("unable to start VIPS");
   }
   vips_leak_set(TRUE);
   {
      std::string inDirName =
            "D:/Workarea/Cplusplus/VipsTestBadFiles/testdaten/";
      std::string outDirName = "D:/Workarea/Cplusplus/VipsTestBadFiles/out/";
      const char * szTestFileName = "006E07875254_20160623015311.jpg";
      std::string testFilePath = inDirName + szTestFileName;
      std::string outFilePath = outDirName + szTestFileName;
      unlink(outFilePath.c_str());
      VImage testImage;
      const char * szError;
      do {
         try {
            gboolean bFail = TRUE;
            testImage = VImage::new_from_file(testFilePath.c_str(),
                  VImage::option()->set("fail", bFail));
            szError = vips_error_buffer();
            if (strlen(szError) > 0) {
               printf("VImage::new_from_file: error text = %s\n", szError);
            } else {
               printf("VImage::new_from_file: no error info\n");
            }
         } catch (const VError & ex) {
            printf("Caught exception %s opening %s\n", ex.what(),
                  testFilePath.c_str());
            break;
         }
         bool bTestSolution = false;
         if (bTestSolution) {
            double avg = testImage.avg();
            szError = vips_error_buffer();
            if (strlen(szError) > 0) {
               printf("testImage.avg(): error text = %s\n", szError);
               vips_error_clear();
            } else {
               printf("testImage.avg(): no error info\n");
            }
         }
         try {
            testImage.write_to_file(outFilePath.c_str());
            szError = vips_error_buffer();
            if (strlen(szError) > 0) {
               printf("VImage::write_to_file: error text = %s\n", szError);
            } else {
               printf("VImage::write_to_file: no error info\n");
            }
         } catch (const VError & ex) {
            printf("Caught exception %s writing %s\n", ex.what(),
                  outFilePath.c_str());
            break;
         }
      } while (false);
   }
   vips_shutdown();
   printf("vipstestbadfiles_main ended\n");
   return 0;
}

</vips></string></unistd.h></stdio.h>

006e07875254_20160623015311

该提问来源于开源项目:libvips/libvips

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 复制链接分享
  • 邀请回答

10条回答

  • weixin_39596668 weixin_39596668 3月前

    Sorry David, this dropped off my radar. I'll have a look.

    点赞 评论 复制链接分享
  • weixin_39596668 weixin_39596668 3月前

    I think I've fixed this in git master. I now see:

    
    john:~/pics$ head -c 10000 k2.jpg > k2_broken.jpg
    john:~/pics$ vips copy k2_broken.jpg x.v
    vips warning: VipsJpeg: read gave 2 warnings
    vips warning: VipsJpeg: Premature end of JPEG file
    john:~/pics$ echo $?
    0
    john:~/pics$ vips copy k2_broken.jpg[fail] x.v
    vips warning: linecache: error reading tile 0x48: VipsJpeg: Premature end of JPEG file
    VipsJpeg: read gave 2 warnings
    VipsJpeg: Premature end of JPEG file
    VipsJpeg: read gave 2 warnings
    
    
    vips warning: linecache: error reading tile 0x56: VipsJpeg: out of order read at line 48
    john:~/pics$ echo $?
    1
    

    Thanks for reporting this bug!

    点赞 评论 复制链接分享
  • weixin_39877504 weixin_39877504 3月前

    Hello John,

    It has taken some time for me to look at your solution - sorry.

    I have just downloaded and built the git master version of libvips and tested it with my slightly updated test program. I now get a VError exception the first time that I try to use the damaged picture. This is when I call the avg() function on the testimage. The exception text is:

    vips__region_start: start function failed for image D:/Workarea/Cplusplus/VipsTestBadFiles/testdaten/006E07875254_20160623015311.jpg

    There is still no warning when the file is read, but considering how vips caches operations, this is probably the best that one can hope for. The exception text is also not so helpful in allowing the user to identify the cause of the problem. He/she knows only that vips had a problem with the specified file, not why.

    Best wishes

    David Singleton

    点赞 评论 复制链接分享
  • weixin_39877504 weixin_39877504 3月前

    I have now done some debugging in vips and have noticed the following: - When the avg() function is called, the function read_jpeg_generate() is called. Here, the jpeg read error is noticed (jpeg2vips.c Line 995) and an appropriate error message (Premature end of jpeg file) is written to the vips_error_buffer. - As the stack is unwound, in tilecache.cpp Line 705, vips_warn is then called and the vips_error_buffer is cleared (loosing the 'Premature end of jpeg file' message?) - The thread then ends. - The error is detected in VImage::call_option_string() (VImage.cpp Line 490) and the contents of the vips_error_buffer (now containing the not very specific error message) are written to the VError exception.

    I hope this helps you a little with a better exception text.

    点赞 评论 复制链接分享
  • weixin_39596668 weixin_39596668 3月前

    I fiddled a bit, it seems better now. With this test program:

     c++
    // compile with
    // g++ read.cpp `pkg-config vips-cpp --cflags --libs`
    
    #include <vips>
    #include <stdio.h>
    
    using namespace vips;
    
    int
    main( int argc, char **argv )
    {
        VIPS_INIT( argv[0] );
    
        try {
            VImage fred = VImage::new_from_file( argv[1] );
    
            printf( "avg = %g\n", fred.avg() );
        }
        catch( VError err ) {
            printf( "exception <>\n", err.what() );
        }
    
        return( 0 );
    }
    </stdio.h></vips>

    I see:

    
    $ ./a.out ~/pics/broken.jpg[fail] 
    vips warning: linecache: error in tile 0 x 192
    vips warning: linecache: error in tile 0 x 200
    exception <<vipsjpeg: premature end of jpeg file vipsjpeg: out order read at line vips__region_start: start function failed for image>>
    </vipsjpeg:>

    The start function failed message is probably not useful, that should go too.

    About 50% of the time it seems to trigger an out-of-order read that makes vips try to resync its threads, causing a lengthy delay. I'll see if I can fix that too.

    点赞 评论 复制链接分享
  • weixin_39877504 weixin_39877504 3月前

    Hello John,

    Thanks for the fix. I have run my test program on it and it works as expected. As you say, the start_function failed message does not bring anything extra useful. It does not however cause me a problem. My test program did not notice any problems with delays, but if there is a problem there, it should probably be fixed.

    There is unfortunately one further problem, When the exception is thrown and caught, there is now a memory leak! The displayed vips text in my test program is:

    11 objects alive: 0) VipsSequential (0x600122e40), count=1 VipsSequential (sequential), check sequential access, sequential in=((VipsImage) 0x6001274d0) out=( (VipsImage_) 0x600127660) tile-height=8 access=((VipsAccess) VIPS_ACCESS_SEQUENTIAL_UNBUFFERED) - 1) VipsForeignLoadJpegFile (0x60011f840), count=1 VipsForeignLoadJpegFile (jpegload), load jpeg from file (.jpg, .jpeg, .jpe), priority=50, is_a, get_ flags, get_flags_filename, header, load, jpegload filename="D:/Workarea/Cplusplus/VipsTestBadFiles/t estdaten/006E07875254_20160623015311.jpg" out=((VipsImage_) 0x600124020) flags=((VipsForeignFlags) V IPS_FOREIGN_SEQUENTIAL) fail=TRUE - 2) VipsAvg (0x60011f8e0), count=1 VipsAvg (avg), find image average, avg in=((VipsImage_) 0x600124020) - 3) VipsImage (0x6001241b0) 224 bytes, count=1 VipsImage (image), image class, 640x480 uchar, 3 bands, srgb, jpegload 4) VipsImage (0x6001277f0), count=1 VipsImage (image), image class, 640x480 uchar, 3 bands, srgb 5) VipsImage (0x600127660), count=1 VipsImage (image), image class, 640x480 uchar, 3 bands, srgb 6) VipsCopy (0x60012c020), count=1 VipsCopy (copy), copy an image, copy in=((VipsImage_) 0x600124020) out=((VipsImage_) 0x6001241b0) - 7) VipsImage (0x600124020) 1136 bytes, count=2 VipsImage (image), image class, 640x480 uchar, 3 bands, srgb, jpegload 8) VipsImage (0x6001274d0), count=3 VipsImage (image), image class, 640x480 uchar, 3 bands, srgb 9) VipsImage (0x600127340) 1360 bytes, count=1 VipsImage (image), image class, 640x480 uchar, 3 bands, srgb 10) VipsLineCache (0x60013f8d0), count=1 VipsLineCache (linecache), cache an image as a set of lines, linecache in=((VipsImage_) 0x6001274d0) out=((VipsImage_) 0x6001277f0) tile-height=8 access=((VipsAccess) VIPS_ACCESS_SEQUENTIAL_UNBUFFERED ) - 1 VipsArea alive 0x600123ac0 count = 2, bytes = 8 memory: 1 allocations, 921616 bytes files: 0 open memory: high-water mark 1.11 MB_

    I could guess that, after the problem is detected, the allocated objects are not associated with the image, so that they are not then deleted when the VImage goes out of scope. Alternatively, they could be deleted in the error handling.

    点赞 评论 复制链接分享
  • weixin_39596668 weixin_39596668 3月前

    I tried a test C program:

     C
    /* compile with
     * gcc read.c `pkg-config vips --cflags --libs`
     */
    
    #include <vips>
    #include <stdio.h>
    
    int
    main( int argc, char **argv )
    {
        VipsImage *im;
        double d;
    
        VIPS_INIT( argv[0] );
    
        if( !(im = vips_image_new_from_file( argv[1], NULL )) )
            vips_error_exit( NULL ); 
        if( vips_avg( im, &d, NULL ) ) {
            g_object_unref( im );
            vips_error_exit( NULL ); 
        }
        g_object_unref( im );
    
        printf( "avg = %g\n", d ); 
    
        return( 0 );
    }
    </stdio.h></vips>

    And that runs without any leaks, so it looks like a problem in the C++ binding. I'll see if I can find out some more.

    点赞 评论 复制链接分享
  • weixin_39596668 weixin_39596668 3月前

    I found a missing unref in the cpp binding, it seems to work now.

    点赞 评论 复制链接分享
  • weixin_39877504 weixin_39877504 3月前

    Hello John,

    Thanks for the quick response. I have now run my test program with your tip (master) revision. The memory leaks are not there any more, so the problem is fixed. As far as I am concerned, this issue can be closed.

    Thanks again

    David Singleton

    点赞 评论 复制链接分享
  • weixin_39596668 weixin_39596668 3月前

    Great!

    点赞 评论 复制链接分享