dongtiao2105 2013-07-03 09:56
浏览 9

Zend Framework 1.12中的Http客户端太慢了

I want to send ~50 requests to different pages on the same domain and then, I'm using DOM object to gain urls to articles.

The problem is that this number of requests takes over 30 sec.

for ($i = 1; $i < 51; $i++)
    $url = ''.$i.'/';             

    $client = new Zend_Http_Client($url);
    $response = $client->request();
    $dom = new Zend_Dom_Query($response); // without this two lines, execution is also too long
    $results = $dom->query('li');         //

Is there any way to speed this up?

  • 写回答

2条回答 默认 最新

  • doujiunai2169 2013-07-04 09:40

    It's a generel problem by design - not the code itself. If you're doing a for-loop over 50 items each opening an request to an remote uri, things get pretty slow since every requests waits until responde from the remote uri. e.g.: a request takes ~0,6 sec to been completed, multiple this by 50 and you get an exection time of 30 seconds!

    Other problem is that most webserver limits its (open) connections per client to an specific amount. So even if you're able to do 50 requests simultaneously (which you're currently not), things won't speed up measurably.

    In my option there is only one solution (without any deep going changes): Change the amout of requests per exection. Make chunks from e.g. only 5 - 10 per (script)-call and trigger them by an external call (e.g. run them by cron).

    Todo: Build a wrapper function which is able to save the state of its current run ("i did request 1 - 10 at my last run, so now I have to call 11 - 20) into a file or database and trigger this function by an cron.

    Example Code (untested) for better declaration;

    private static $_chunks = 10; //amout of calls per run
    public function cronAction() {
        $lastrun = //here get last run parameter saved from local file or database
    private function crawl($lastrun) {
        $limit = $this->_chunks + $lastrun;
        for ($i = $lastrun; $i < limit; $i++)
            [...] //do stuff here
        //here set $lastrun parameter to new value inside local file / database
    本回答被题主选为最佳回答 , 对您是否有帮助呢?



  • ¥15 有人懂怎么做大模型的客服系统吗?卡住了卡住了
  • ¥20 firefly-rk3399上启动卡住了
  • ¥15 如何删除这个虚拟音频
  • ¥50 hyper默认的default switch
  • ¥15 网站打不开,提示502 Bad Gateway
  • ¥20 基于MATLAB的绝热压缩空气储能系统代码咨询
  • ¥15 R语言建立随机森林模型出现的问题
  • ¥15 中级微观经济学,生产可能性边界问题
  • ¥15 TCP传输时不同网卡传输用时差异过大
  • ¥15 请各位看看我写的属于什么算法,或者有更正确的写法?