2013-12-13 14:29
浏览 84


I have a number of LZO-compressed log files on Amazon S3, which I want to read from PHP. The AWS SDK provides a nice StreamWrapper for reading these files efficiently, but since the files are compressed, I need to decompress the content before I can process it.

I have installed the PHP-LZO extension which allows me to do lzo_decompress($data), but since I'm dealing with a stream rather than the full file contents, I assume I'll need to consume the string one LZO compressed block at a time. In other words, I want to do something like:

$s3 = S3Client::factory( $myAwsCredentials );

$stream = fopen("s3://my_bucket/my_logfile", 'r');

$compressed_data = '';
while (!feof($stream)) {
    $compressed_data .= fread($stream, 1024);

    // TODO: determine if we have a full LZO block yet
    if (contains_full_lzo_block($compressed_data)) {

        // TODO: extract the LZO block
        $lzo_block = get_lzo_block($compressed_data);

        $input = lzo_decompress( $lzo_block );
        // ...... and do stuff to the decompressed input


The two TODOs are where I'm unsure what to do:

  1. Inspecting the data stream to dtermine whether I have a full LZO block yet
  2. Extracting this block for decompression

Since the compression was done by Amazon (s3distCp) I don't have control over the block size, so I'll probably need to inspect the incoming stream to determine how big the blocks are -- is this a correct assumption?

(ideally, I'd use a custom StreamFilter directly on the stream, but I haven't been able to find anyone who has done that before)

  • 点赞
  • 写回答
  • 关注问题
  • 收藏
  • 邀请回答

1条回答 默认 最新

  • doubeng9567
    doubeng9567 2013-12-16 13:31

    Ok executing a command via PHP can be done in many different ways, something like:

    $command = 'gunzip -c /path/src /path/dest';
    $escapedCommand = escapeshellcmd($command);

    or also

    shell_exec('gunzip -c /path/src /path/dest');

    will do the work. Now it's a matter of what command to execute, under Linux there's a nice command line tool called lzop which extracts orcompresses lzop files.

    You can use it via something like:

    lzop -dN sources.lzo

    So you final code might be something as easy as:

    shell_exec('lzop -dN s3://my_bucket/my_logfile');
    点赞 评论