dongmimeng5500 2018-06-13 03:40
浏览 91
已采纳

PHP / Symfony:写入CSV大文件

Thank you for helping.

My Spec:

  1. I have a very large CSV file
  2. I need to process the file and output another large CSV file

My environment:

  1. php7.0
  2. Symfony 3.4 framework

My current solution:

I am using Symfony Serializer component to read my file, I then process my file and output the file. All in memory. Operation takes around 20 minutes to complete.

I run the script from a Symfony command. I have a service to read the file and a service to export the file.

An improved solution 1:

  1. I can load the CSV file into a database table like explained here, with "LOAD DATA LOCAL INFILE" sql query. Very fast operation
  2. I can then process my data and save it to another table
  3. Then I would use "SELECT ... INTO OUTFILE 'file_name'" to output the file

Advantage: -SQL centered - No memory problem

Disavantage: -DB operations during processing might be expensive

An improved solution 2:

  1. I can read the CSV file line by line
  2. I process the line
  3. I write the line to the output file

Adv: No memory issue Disav: Could take a LOT of time.

An improved solution 3:

  1. I can load the CSV file into a database table like explained here, with "LOAD DATA LOCAL INFILE" sql query. Very fast operation
  2. I can then process my data in chunks of 1000 and write them to a file

What solution would you use? Or do you have any better solution?

  • 写回答

1条回答 默认 最新

  • ds34222 2018-06-13 04:21
    关注

    solution 2, without a database. Read from csv, process and output to csv just like someone mentioned in the comments, use fgetcsv() and fputcsv(). Going line by line should hardly consume any memory and it will remove the need for a database in between. The issue with these types of operations is the sequential nature of reading csv files as streams, eventually the speed of the process will boil down to the speed of the operation on the data in between the read and write operations. Using a database in between will just slow the entire process down and be somewhat wasteful.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题?

悬赏问题

  • ¥15 slaris 系统断电后,重新开机后一直自动重启
  • ¥15 51寻迹小车定点寻迹
  • ¥15 谁能帮我看看这拒稿理由啥意思啊阿啊
  • ¥15 关于vue2中methods使用call修改this指向的问题
  • ¥15 idea自动补全键位冲突
  • ¥15 请教一下写代码,代码好难
  • ¥15 iis10中如何阻止别人网站重定向到我的网站
  • ¥15 滑块验证码移动速度不一致问题
  • ¥15 Utunbu中vscode下cern root工作台中写的程序root的头文件无法包含
  • ¥15 麒麟V10桌面版SP1如何配置bonding