dongpu5874 2016-08-18 20:06
浏览 61

限制每秒获得php curl请求以获取网站页面上的所有链接

I'm trying to get all links at concrete web-page. I need only 'a' tags with concrete parameter. But firstly, as far as I know, I have to download the whole page.

I use this (mostly not mine) code:

<?php
function file_get_contents_curl($url) 
{
    $ch = curl_init();
    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //Устанавливаем параметр, чтобы curl возвращал данные, вместо того, чтобы выводить их в браузер.
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_FORBID_REUSE, true);
    $data = curl_exec($ch);
    curl_close($ch);
    return $data;
}
$startUrl = 'address';
$data = file_get_contents_curl($startUrl);
echo($data);
?>

By this I'm getting error "Too many requests". The question is: can I change the amount of requests for finging elements of links array? I think about curl_multi, but as far as I understand, it assumes that I already have the array and only need to make multiple threads.

Help, please.

  • 写回答

0条回答 默认 最新

    报告相同问题?

    悬赏问题

    • ¥15 如何用stata画出文献中常见的安慰剂检验图
    • ¥15 c语言链表结构体数据插入
    • ¥40 使用MATLAB解答线性代数问题
    • ¥15 COCOS的问题COCOS的问题
    • ¥15 FPGA-SRIO初始化失败
    • ¥15 MapReduce实现倒排索引失败
    • ¥15 ZABBIX6.0L连接数据库报错,如何解决?(操作系统-centos)
    • ¥15 找一位技术过硬的游戏pj程序员
    • ¥15 matlab生成电测深三层曲线模型代码
    • ¥50 随机森林与房贷信用风险模型