致命错误：允许内存耗尽（wordpress +插件）CSV导入

Question

Hello fellow IT fanatics, this is my first post so I am going to try not to mess up ...

I am currently trying to import 500 000 CSV lines into a database using a plugin for Wordpress I bought (basically a php script).

This is the error I get :

     ( ! ) Fatal error: Allowed memory size of 268435456 bytes exhausted (tried to allocate  70 bytes) in E:\UwAmp\www\wp-includes\option.php on line 397
    Call Stack
    #   Time    Memory  Function    Location
    1   0.0009  425192  {main}( )   ..\admin.php:0
    2   1.2649  60016232    do_action( )    ..\admin.php:211
    3   1.2649  60017160    call_user_func_array ( )    ..\plugin.php:470
    4   1.2649  60017176    AitImport->display_plugin_admin_page( ) ..\plugin.php:470
    5   1.2658  60091912    include_once( 'E:\UwAmp\www\wp-content\plugins\wp-import- plugin\views\admin.php' ) ..\class-ait-import.php:220
    6   1.2659  60091968    AitImport->import_terms_csv( )  ..\admin.php:41
    7   240.4112    261278688   wp_insert_term( )   ..\class-ait-import.php:505
    8   240.4547    261280928   clean_term_cache( ) ..\taxonomy.php:2520 
    9   240.4919    261281832   _get_term_hierarchy( )  ..\taxonomy.php:3226
    10  240.9745    266577072   update_option( )    ..\taxonomy.php:3373
    11  240.9746    266577176   add_option( )   ..\option.php:274

I've tried checking the logs in phpMyAdmin here is what I get (these 3 lines get repeated), there is no END statement, may this be the cause of my problem ?

 mysql-bin.000029    27293   Query   1   27398   BEGIN 
 mysql-bin.000029    27398   Query   1   28471   use `lecoindesadresses`; UPDATE `wp_options` SET `... 
 mysql-bin.000029    28471   Xid     1   28502   COMMIT /* xid=199 */ 

This is what I have at line 505 :

    if (...) {
    ...
    } else {
        $term_id = wp_insert_term($title, $tax, $attrs);
    }

Here is what the plugin does :
1 - Parse one line of the CSV input file
2 - Check with 3 select if the slug doesn't already exist
3 - Ignore if it exists or call the wp_insert_term if it doesn't

My RAM is fine while it finds already existing slugs (Apache uses around 91Mb of RAM) and when it starts inserting the RAM goes up by 5Mb every second until it reaches around 256Mb. I then get this fatal error. It successfully inserts around 300 to 400 items every time but it lowers everytime I launch the script again.

I am trying all this in local atm (tried on my server, same error). Allowing more memory to apache won't solve the problem as the RAM usage keeps growing very fast.

You can ask for more specific details if you have any idea, thanks in advance !

Answer 1

PHP is allocated 256mb of RAM (268435456 bytes), that should be enough for most operations. Imports like these, if implemented naively, can eat up a lot of memory.

Since you say you bought this plugin, is there any chance you can debug it? Look for memory leaks, variables that are never released, a global heap, that kind of thing.

---

You can also split the CSV file into smaller files so the script doesn't try to allocate as much memory. This is the pragmatic solution.