I'm acquiring image objects from a remote server, then attempting to upload them to Rackspace's Cloud Files using their API. Wondering a) how I can make this process more efficient, and b) assuming I'll need to purchase more memory, what a reasonable amount of RAM might be to accomplish this task (current development server is just 512MB).
In executing the script I'm:
- Querying my local database for a set of ids (around 1 thousand)
- For each id, querying a remote server, which returns between 10-20 image objects, each image is 25-30k
- Create a Cloud Files container, based on the id from my db
- For each image object returned from the remote server, create an image object in my container, and write image data to that object
- Update row in local db with datetime of added images
This executes relatively quickly on a small set of ids, however 100 (so 700-1k images) can take 5-10min, and anything more than that seems to run indefinitely. Have tried the following, with little success:
- using php's set_timeout to kill the script after a couple minutes, figuring that'd purge memory allocated to execution, allowing me to pick up where I left off and work through the set is smaller pieces. However this error is never thrown
- unset the array key containing the image object after it's uploaded (not just the reference inside the loop).
PHP's memory_limit is set to 128MB, and running 'tops' command I see that user 'www-data' was consuming 16% of memory resources. However that no longer appears in the list of users, but I continue to see this:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2400 mysql 20 0 161m 8220 2808 S 0 1.6 11:12.69 mysqld
...but the TIME+ never changes. I see that there is still 1 task running, yet these values never change:
Mem: 508272k total, 250616k used, 257656k free, 4340k buffers
Apologies for the lengthy post - not entirely sure what (if any of that) is useful. This is not my area of expertise so grasping at straws a little. Thanks in advance for your help.
