too long

Question

I have a controller method that takes a long time (40 minutes) to complete. the method retrieve 3500 domain name and for each one it should search registrar name, expire that and also check if http server is running or not. This is the reason it's taking so long;

this is my code, it works, but slow :

public function sync_domains(){

        // get the domain name list
        $list = $this->Domain->find('all');

        // search for registrar name, expire date and web server status (up|down)
        foreach ($list as $key => $record) {
            $hostname = $record['Domain']['domain_name'];
            $domain = array();

            // get registrar name and expire date
            $whois = $this->whois($hostname);

            $domain['Domain']['registrar'] = $whois['registrar'];
            $domain['Domain']['expire_date'] = $whois['expire_date'];

            // get the web status
            $domain['Domain']['web_status'] = $this->httpStatus($hostname);

            $newData[] = $domain;
        }


        // save new data
        if(!empty($newData)){
            if($this->DomainStatus->saveMany($newData)){
                $this->Flash->success(__(count($newData).' Domains has been added successfully'), array('key' => 'success'));
            }else{
                $this->Flash->danger(__('An Error occured while saving data'), array('key' => 'danger'));
            }
        }

        // redirect to referer
        return $this->redirect($this->referer());
    }

is there any way to use multithreading to launch whois and httpstatus functions for multiple domain name at the same time ?

thanks,

Answer 1

If it's a one-off job, a common approach is to divide the database results in batches and manually execute them, either via command line or simply using your web browser and accessing each batch in a different tab (or on different browsers).

An example implementation:

public function sync_domains($batch){

    $numberOfItemsPerBatch=500; 
    // get the domain name list
    $list = $this->Domain->find('all',[
        'limit'=>$numberOfItemsPerBatch,
        'offset'=>$batch*$numberOfItemsPerBatch
        ]
    );

    //process data

You would then access the following URLs concurrently:

http://example.com/my_controller/sync_domains/0
http://example.com/my_controller/sync_domains/1
http://example.com/my_controller/sync_domains/2
etc...

This will reduce your processing time from 40 to 5-6 minutes. Make sure your max_execution_time is set to allow this.

If you try to use your browser, you will notice that you are not able to load a new view if a previous one is still being processed. This is due to PHP Session Locks. An easy workaround is to use Chrome Incognito Mode or Firefox's Private Browsing to access your site concurrently.

A simpler option is perhaps to just open a shell and run

$ wget http://example.com/my_controller/sync_domains/0 &
$ wget http://example.com/my_controller/sync_domains/1 &
$ wget http://example.com/my_controller/sync_domains/2 &
etc...