在PHP中解压缩LZO流

I have a number of LZO-compressed log files on Amazon S3, which I want to read from PHP. The AWS SDK provides a nice StreamWrapper for reading these files efficiently, but since the files are compressed, I need to decompress the content before I can process it.

I have installed the PHP-LZO extension which allows me to do lzo_decompress($data), but since I'm dealing with a stream rather than the full file contents, I assume I'll need to consume the string one LZO compressed block at a time. In other words, I want to do something like:

$s3 = S3Client::factory( $myAwsCredentials );
$s3->registerStreamWrapper();

$stream = fopen("s3://my_bucket/my_logfile", 'r');

$compressed_data = '';
while (!feof($stream)) {
    $compressed_data .= fread($stream, 1024);

    // TODO: determine if we have a full LZO block yet
    if (contains_full_lzo_block($compressed_data)) {

        // TODO: extract the LZO block
        $lzo_block = get_lzo_block($compressed_data);

        $input = lzo_decompress( $lzo_block );
        // ...... and do stuff to the decompressed input
    }

}
fclose($stream);

The two TODOs are where I'm unsure what to do:

  1. Inspecting the data stream to dtermine whether I have a full LZO block yet
  2. Extracting this block for decompression

Since the compression was done by Amazon (s3distCp) I don't have control over the block size, so I'll probably need to inspect the incoming stream to determine how big the blocks are -- is this a correct assumption?

(ideally, I'd use a custom StreamFilter directly on the stream, but I haven't been able to find anyone who has done that before)

duanmen1887
duanmen1887 通过系统命令解压缩工作会很好-如果有人可以发布一个有效的例子,我很乐意接受这个作为答案
6 年多之前 回复
doucanrui1735
doucanrui1735 如何使用一些系统命令解压缩lzo文件,而不是在PHP下从头开始实现解压缩?我搜索了phplzo库,但没有找到。
6 年多之前 回复

1个回答



通过PHP执行命令可以通过多种方式完成,例如:</ p>

   $ command ='gunzip -c / path / src / path / dest'; 
$ escapedCommand = escapeshellcmd($ command);
system($ escapedCommand);
</ code> </ pre>

或者</ p>

  shell_exec('gunzip -c / path / src / path / dest'); 
</ code> </ pre>

将完成工作。
现在要执行什么命令,在Linux下有一个很好的命令行工具lzop,它提取或压缩lzop文件。</ p>

你可以 通过以下方式使用它:</ p>

  lzop -dN sources.lzo 
</ code> </ pre>

所以你最后的代码可能是某种东西 一样简单:</ p>

  shell_exec('lzop -dN s3:// my_bucket / my_logfile'); 
</ code> </ pre>
</ div>

展开原文

原文

Ok executing a command via PHP can be done in many different ways, something like:

$command = 'gunzip -c /path/src /path/dest';
$escapedCommand = escapeshellcmd($command);
system($escapedCommand);

or also

shell_exec('gunzip -c /path/src /path/dest');

will do the work. Now it's a matter of what command to execute, under Linux there's a nice command line tool called lzop which extracts orcompresses lzop files.

You can use it via something like:

lzop -dN sources.lzo

So you final code might be something as easy as:

shell_exec('lzop -dN s3://my_bucket/my_logfile');

duancheng1955
duancheng1955 看起来很有希望。 我明天会试一试!
6 年多之前 回复
Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
立即提问