I have to create a big csv export file of more than 400 MB with PHP. First drafts of the export file and the PHP code allow some guesses about the performance.
To avoid extremely long processing times, I should focus on creating the export file efficiently and avoid PHP array
-operations as they are too slow in that case. "Create the file efficiently" means: append big blocks of text to other big blocks in the file, each big block being created quickly.
Unfortunately, the "big blocks" are rather rectangles than lines. Building my export file will start with lots of line beginnings, like this:
Title a, Title b, Title c
"2014", "07", "01"
"2014", "07", "02"
...
Then I would have to to add a "rectangle" of text to the right of the line beginnings:
Title a, Title b, Title c, extention 1, extention 2, extention 3
"2014", "07", "01", "23", "1", "null"
"2014", "07", "02", "23", "1", "null"
...
If I have to do this line by line, it will slow me down again. So I'm hoping for a way to add "rectangles" in a file, just as you can in some text editors. Also helpful would be a concrete experience with huge text buffers in PHP, could also work.
As it is not my hosting, I'm not sure if I have permissions to invoke sed/akw.
So the question is: Can sb advice from experience how to handle big csv files in PHP efficiently (file block operations, file "rectangle" operations) or just how to handle big string buffers in PHP efficiently? There seems to be no framework for string buffers.
Thank you for your attention :-)
Note: This is not a duplicate of this: https://stackoverflow.com/questions/19725129/creating-big-csv-file-in-windows-apache2-php