I have a php script that is reading a remote CSV file, and adding products to a database based on the contents of the CSV file. At present there are about 2800 lines (products) but the script keeps stopping at line 1388.
The code I used is as follows:
while(($data = fgetcsv($fopen, 0, ",")) !== false):
//stuff is done here...
endwhile;
I have set the php memory limit to 64M and even tried 128M. I also set the max_execution_time to 60mins. I have also tried altering the code as follows:
while(($data = fgetcsv($fopen, 1000, ",", '')) !== false):
//stuff is done here...
endwhile;
That DID result in more lines being parsed, BUT the data was then incorrect, i.e. image columns were becoming description columns etc. I assume that has to do with adding as my line ending. I tried , no luck. Lastly, I also added the auto_detect_line_endings as true in the ini.
Can anyone suggest reasons as to why my data is being cut short?
Regards, Simon
EDIT
I have noticed something interesting. I have a MySQL insert on each line that is looped over in the above code. Now, the last record in my database is the FIRST row in the CSV file, does this mean the file is being parsed from the last line up??
These seem to be the rows at or near the break:
W-3066, I Love Love Cheap And Chic, Moschino, 3.4 oz,EDT Spray,Women,,"Introduced by the design house of Moschino, I love love has a blend of grapefruit, orange, lemon, red currant, tea rose, cinnamon leaves, musk, cedar and tonka wood. It is recommended for daytime wear.",http://www.perfume-worldwide.com/products/Women/Final/W-3066large.jpg,0,0,0,8011003991457
W-3070, Adidas Floral Dream, Adidas, 1.7 oz,EDT Spray,Women,,"Introduced in 2008, the notes are bergamot, lily, rose, tonka bean and vanilla.",http://www.perfume-worldwide.com/products/Women/Final/W-3070large.jpg,0,0,0,3412244310024
W-3071, Adidas Fruity Rhythm, Adidas, 1.7 oz,EDT Spray,Women,,"Introduced in 2008, the notes are black currant, raspberry, cyclamen, freesia and musk.",http://www.perfume-worldwide.com/products/Women/Final/W-3071large.jpg,0,0,0,3412244510004
SOLUTION
As it turns out, it worked out a lot better for me to copy the file to my server, and work off the copy. The steps I followed are as follows:
- I read the contents of the remote file using
file_get_contents()
- I then used
iconv(
) function to re-encode data to UTF-8 - I made a temp file using
fopen()
,fwrite()
andfclose()
functions, contents of the file was the encoded data above - I set the permissions of the file to 0750 using the
chmod()
function - I then applied the
fgetcsv()
function to my temp file - Did all that needed to be done
- Deleted the temp file once done, using
unlink()
function
That did the trick. So, I suspect half the issue was actually the remote server timing out, and the other half encoding issues.
Thank you to everyone for all the nudges in the right direction