解析50GB xml文件，导致内存不足致命错误

Question

I am parsing one xml file with the size near about 50GB, using below code. getting Fatal error: Out of memory (allocated 524288) (tried to allocate 5000000001 bytes)

    $xml_parser = xml_parser_create();
    xml_parser_set_option($xml_parser, XML_OPTION_CASE_FOLDING, false);
    xml_set_element_handler($xml_parser, "startElement", "endElement");
    xml_set_character_data_handler($xml_parser, "characterData");
    if (!($fp = fopen($file, "r"))) {
        die("could not open XML input");
    }

    $e=4096;
    $file_content = fread($fp, 5000000000);

    xml_parse($xml_parser, $file_content, feof($fp));
    echo "<pre>";
    print_r($valuess);
    if(is_array($valuess)){
      mysql_connect('********', '*****', '*******');
      mysql_select_db("wp534");

    for($i=0;$i<count($valuess);$i++)
    {
    $sql = "INSERT INTO wp_xmldata (RegistreringNummerNummer, KoeretoejMaerkeTypeNavn, KoeretoejModelTypeNavn, KoeretoejVariantTypeNavn, KoeretoejTypeTypeNavn, KoeretoejOplysningFoersteRegistreringDato, DrivkraftTypeNavn, SynResultatSynsDato) values ";
    $items=array();
    $valuesArr=array();    
    //echo ($i+1).'  ------------------------<br/>';

      for($j=0;$j<count($names[$i]);$j++)
      {  //echo $names[$i][$j].'=>'.$valuess[$i][$j]."</br>";
                $a=explode("ns:",$names[$i][$j]); 
                $items[$a[1]]=$valuess[$i][$j];

      }
                $valuesArr[] = "('$items[RegistreringNummerNummer]','$items[KoeretoejMaerkeTypeNavn]','$items[KoeretoejModelTypeNavn]','$items[KoeretoejVariantTypeNavn]','$items[KoeretoejTypeTypeNavn]','$items[KoeretoejOplysningFoersteRegistreringDato]','$items[DrivkraftTypeNavn]','$items[SynResultatSynsDato]')";

    $sql .= implode(',', $valuesArr);
    /******start code for append data**********/
    /*$select="select * from wp_xmldata where RegistreringNummerNummer = '$items[RegistreringNummerNummer]'";
    //echo $fetch11;
    $select_result=mysql_query($select);
    if(mysql_num_rows($select_result) == 0)
    { //echo $sql;
      mysql_query($sql);
    }*/
    /******end code for append data**********/
    //mysql_query($sql);
    //echo $sql; 

    }

        enter code here

    }

    xml_parser_free($xml_parser);
    fclose($fp);

Anybody have a suggestion for this?

Thanks, Ankit Sanghvi

Answer 1

Your problem is not yet with the XML parser here, it's more fundamentally:

 $file_content = fread($fp, 5000000000);

AFAIK in PHP strings are limited to 2GB in size. You try to obtain 50GB into that string. That simply does not work. Please consult the documentation which data does fit into which data-type. For strings you can obtain the information here:

Note: string can be as large as up to 2GB (2147483647 bytes maximum)

Source: http://php.net/manual/en/language.types.string.php

You should therefore understand that xml_parse must not operate on the whole string at once, it allows to parse chunk by chunk. Here again, you should read the manual, understand what which usage those functions are meant and then write your code.

Especially as you've running against the wall already. Time to change the code and rewrite it with a pulling logic. Or instead, take XMLReader, perhaps with XMLReaderIterator extension.