I'm having an issue with a script I've written. I'm doing some basic screen scraping and saving it to a WordPress database. The script functions as it should but when I'm trying to do multiple pages (like around 20 pages), it gives me a 500 error after running a big chunk of the script. On the script page I included this:
set_time_limit(0);
ignore_user_abort(1);
I have also included max_input_time = 0
in the php.ini file in the root of my server which in theory should prevent it from timing out but I am connecting to two other URLs so I'm curious if it may have something to do with one of those connections. Basically, it goes like this:
Script in folder on server -> Queries info from a URL and pulls it back into my server -> Analyzes the data -> Sends it to WordPress database on my server but in a different directory
Is there any way to determine where the timing out is happening? My assumption is that it's not happening on the script page but with a generic 500 error, I'm not sure how to troubleshoot deeper to identify what's happening.
Thanks for your help!
EDIT
I turned on error reporting and here's the error I received:
[Thu Jul 03 21:29:12 2014] [8808155] [fcgid:warn] (104)Connection reset by peer: [client 50.184.161.228:51983] mod_fcgid: error reading data from FastCGI server
[Thu Jul 03 21:29:12 2014] [8808155] [fcgid:warn] (104)Connection reset by peer: [client 50.184.161.228:51983] mod_fcgid: ap_pass_brigade failed in handle_request_ipc function
The IP address in the error is not my IP address nor is the IP address of the site that I'm scraping from so I'm not sure where that came in.
I'm not sure if this is related to server configuration which obviously I would be limited to updating since I'm on a GoDaddy shared server.
I found this article:
Which I suppose answers my question. I guess I can't update the settings. Now with that said, is there a way that I can have the script crawl five pages, rest, and crawl five more, etc. I've used sleep()
but I assume that that still is part of running the 'script' so it's not like it's stopping and restarting, it's just continuing the script but pausing for a second?