PHP cURL或Multi Curl还是其他什么？

Question

I have a PHP script which loops through a database table of URLs and call a PHP cURL for each URL, scrape the data and then save the data into a different database. Code:

$results = $wpdb->get_results("SELECT * FROM urls ORDER BY id DESC");
foreach($results as $row) {
    //PHP CURL STUFF HERE
}

The above script is run using cPanel's CRON every 12 minutes. My problem is a) the URLs needs to be scraped more often to get better results, because sometime the loop takes time to complete and some URls are not covered in the 12 minutes time.

Now.. Is the above strategy the only strategy or there is any better solution such as Multi Curl? I was thinking if I can have something like script.php?url=1 and then there is a dedicated CRON or something similar for this URL so I can track and check whats going on with this exact URL. Is this possible? If yes, how ? If there are 300 URLs in the database, I can't create 300 CRONs for it :(

Looking forward to your opinions :)=

Answer 1

Calling PHP multiple time is not a good solution. Why can't you do 300 request in 12 min? (thats really slow)

You need to add a timeout for example

curl_setopt($ch, CURLOPT_TIMEOUT, 10); //Each url will take 10 sec max

and then display a log with the URL in it to find slow requests Use curl_errno() to detect errors

Code 28 is timeout.