douxin0251
2015-05-19 19:46
浏览 70
已采纳

处理大卷曲响应 - PHP

I wrote a PHP script that makes HTTP POST request using curl and does the following,

  • Prepare post variables
  • Initialize curl
  • Set client cookie to use in request
  • Set POST variables as query string
  • Set other curl options
  • Execute curl

Here is the code:

    $ch = curl_init ( $url );

    curl_setopt ( $ch, CURLOPT_COOKIE, "cookie=cookie");
    curl_setopt ( $ch, CURLOPT_POST, 1);
    curl_setopt ( $ch, CURLOPT_POSTFIELDS, $post_string);
    curl_setopt ( $ch, CURLOPT_FOLLOWLOCATION, 1);
    curl_setopt ( $ch, CURLOPT_HEADER, 0);
    curl_setopt ( $ch, CURLOPT_RETURNTRANSFER, 1);

    $response = curl_exec( $ch );
    // this point
    extr ( $response, $param_1, $param_2);

Problem is, sometimes the response is larger than 1GB, so the PHP code pauses until, full response is received (shown in code as // this point), and if there is malformed HTML receive, PHP generates error so, all thing here needs to do from beginning.

Here is rest of the functions:

function extr($string = '',$a,$b)
{
    $doc = new DOMDocument;
    @$doc -> loadHTML($string);
    $table = $doc -> getElementById('myTableId');

    if(is_object($table)):
    foreach ($table->getElementsByTagName('tr') as $record)
    {
        $rec = array();
        foreach ($record->getElementsByTagName('td') as $data)
        {
            $rec[] = $data -> nodeValue;
        }
        if ($rec)
        {
            put_data($rec);
        }
    }
    else:
    {
        echo 'Skipped: Param1:'.$a.'-- Param2: '.$b.'<br>';
    }
    endif;
}

function put_data($one = array())
{
    $one = json_encode($one) . "
";
    file_put_contents("data.json", $one, FILE_APPEND);
}

ini_set('max_execution_time', 3000000);
ini_set('memory_limit', '-1');

The alternative i can think of is process data as it received, if possible , using curl, or continue previous curl request from the previous state.

Is there any possible workaround for this?

Do i need to switch to any other language than PHP for this ?

  • 写回答
  • 好问题 提建议
  • 追加酬金
  • 关注问题
  • 邀请回答

2条回答 默认 最新

相关推荐 更多相似问题