I have a script that is causing timeouts, some of the time. It takes a while to run, and I'll explain what it does: We have a fairly small number of users (put it at 20), and we manage inventory for all of those users. Inventories are sent via ftp by a third party every morning (say 6:00 AM or so) as .csv files. Inventories include item descriptions, and then a variable length list of URLs for images. Our system is required to download any images that it doesnt already have (which 99.9% of the time only happens when a new item is in the inventory feed). Usually the inventory feeds are 95% the same, since most of the inventory doesn't sell from one day to the next.
The tricky part is that every morning our system looks at every inventory item and cross-checks each item's image list against the new feed. If images don't exist, it brings the new ones over using a CURL operation.
As you can imagine, depending on the day, this could be a rather time-consuming operation. I have it on a cron job. If I run it manually, it takes anywhere between 1-5 minutes depending on the load, and sometimes (as in, once in every 5 tries) it gives an "internal server error" with no explanation whatsoever.
I am using the set_time_limit(0) directive first thing in the file so I'm wondering if there's something else I need to do to ensure that it's not going to time out? Or do you guys think there's a possibility that failed transfers could be causing issues and making the script die on certain occasions? Like maybe a failed transfer that's handled badly-- I don't know. Rather than post all the code I'm wondering if I could get some ideas first since the script is pretty involved and I don't want to waste anyones time.
Any ideas are welcome. I can't think of why it's intermittently not working. For the record, if I manually run it twice, it always works the second time, but I think that's because the first run already handled most of the downloads...
