PHP中的非阻塞卷曲请求

Question

So I have this function for making non-blocking curl requests. It works fine on what I've tested so far (small amounts of requests). But I need this to scale up to thousands of requests (maybe max 10,000). My issue is that I don't want to run into issues with too many parallel requests running at once.

What would you suggest to rate-limit the requests? Usleep? Requests in batches? The function is below:

function poly_curl($requests){

            $queue = curl_multi_init(); 
            $curl_array = array(); 
            $count = 0;
            foreach($requests as $request) 
            { 
                $curl_array[$count] = curl_init($request); 
                curl_setopt($curl_array[$count], CURLOPT_RETURNTRANSFER, true); 
                curl_multi_add_handle($queue, $curl_array[$count]); 
                $count++;
            } 

            $running = NULL; 
            do { 

               curl_multi_exec($queue,$running); 

            } while($running > 0); 

            $res = array(); 
            $count = 0;
            foreach($requests as $request) 
            { 
                $res[$count] = curl_multi_getcontent($curl_array[$count]); 
                $count++;
            } 

            $count = 0;
            foreach($requests as $request){ 
                curl_multi_remove_handle($queue, $curl_array[$count]); 
                $count++;
            } 
            curl_multi_close($queue);        
            return $res; 
    }

Answer 1

You may want to implement something that is called Exponential Backoff (wikipedia).

Basically, it is an algorithm that allows you dynamically scale your processes depending on some feedback.

You define a rate in your application, and on the first time-out, error, or anything you decide, you decrease this rate until the request finishes.

You may implement it easily using the HTTP response code for example.