I am using scrapy and scrapyd to crawl some content. I have 28 crawlers that run, but only 8 at a time. Each crawler takes from 10 min to several hours to complete. So im looking for a way to order them correctly, in order to minimize the time the server is active.
I already gather information of how long each crawl takes, so it's only the minimization problem, or how to formulate it.
The script is started using php, so the solutions should preferably run in php.
分享 The best way I've found is to set them up as cronjobs to execute at specific times. I have around 30 cronjobs configured to start at various times meaning you can set specific times per scrap.
Executing a PHP cmmand by cronjob at 5pm every day:
* 17 * * * php /opt/test.php
If you execute the scrapy python command via cronjob, its:
* 17 * * * cd /opt/path1/ && scrapy crawl site1
If your using virtualenv for you python then its
* 17 * * * source /opt/venv/bin/activate && cd /opt/path1/ && scrapy crawl site1
