可以使用cURL_setopt和PHP编写大型XML文件而不使用fwrite吗？

Question

I am having an almost identical problem to this one, posted in '09. It doesn't appear to be successfully solved, and I have been poring over related questions all day & implementing suggestions I have found.

I am attempting to pull dynamic (changes daily) data from an XML source (not a file) and insert into a mySQL database. When I attempt to pull the data without fwrite, the data returns without tags/keys in the output. Thus, I cannot properly assign the data to specific database fields.

I decided to write the information to a file and go from there, but the file aborts/"completes" at 334kb every single time. Unfortunately, I am a cURL novice and don't have the chops to see exactly what the issue is. Also, I am using vqMod for OpenCart to do this, otherwise I'd be using straight up PHP.

What am I missing?

Also, is there a way to pull the xml as an array instead of as a string (thereby potentially bypassing the fwrite step)? Or, should I be writing to some other file type?

Any suggestions or a redirect to a solved question I've missed are appreciated.

This is the relevant code, commented portions are fixes I've attempted:

$curl = curl_init();
    $fp = fopen('dir/file.xml' , "w" );
    //w+ will not download any information from the url - file is created but empty.

    //AS IS, downloads first 334KB of file then lands on a blank page
    //and a 500 error when any of these other options are implemented. 
    curl_setopt($curl, CURLOPT_URL, 'http://www.url.com');
    curl_setopt($curl, CURLOPT_FILE, $fp);
    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);

    //  curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    //  curl_setopt($curl, CURLOPT_TIMEOUT,  300);
    //  curl_setopt($curl, CURLOPT_NOPROGRESS, false);
    //  curl_setopt($curl, CURLOPT_RANGE, '0-1000');


    //  $data = array();
    $data = curl_exec($curl);

    fwrite($fp, $data);

    curl_close($curl);
    fclose($fp);

Update: Attempted to use simplexml_load_string instead of fwrite() to pull one product's information, but am still having limited success. Example of XML I am using:

        <?xml version="1.0"?>
      <response>
        <root>
          <part>
            <![CDATA[PARTNUM]]>
          </part>
          <errorcode>0</errorcode>
          <errormsg></errormsg>
          <special>N</special>
          <description>
            <![CDATA[]]>
          </description>
          <price>75</price>
          <weight>1.02</weight>
          <webpurch>Y</webpurch>
          <altnum>
            <![CDATA[ALT-NUM]]>
          </altnum>
          <active>Y</active>
          <redo>
            <![CDATA[]]>
          </redo>
          <codes> 
              <code>
                <fieldname>
                  <![CDATA[Mfr Part No]]>
                </fieldname>
                <fieldvalue>
                  <![CDATA[PARTNUM]]>
                </fieldvalue>
               </code>
               <code>
                <fieldname>
                  <![CDATA[Special Code 1]]>
                </fieldname>
                <fieldvalue>
                  <![CDATA[XYZ123]]>
                </fieldvalue>
               </code> 
            </codes>
          <customtag>N</customtag>
          <onhand>0</onhand>
          <notes>
            <![CDATA[PRODUCT-SPECIFIC NOTE]]>
          </notes>
          <mfr>
            <mfr_name>
              <![CDATA[MFR]]>
            </mfr_name>
          </mfr>
          <altpartnums>
            <altnum>
              <![CDATA[PARTNUM.12]]>
            </altnum>
          </altpartnums>
          <gtrue>N</gtrue>
          <group>
            <![CDATA[GROUP NAME]]>
          </group>
          <categories>
            <cat>294</cat>
            <cat>475</cat>
          </categories>
        </root>
      </response>

This is an example of the $data return for multiple products when I do not use fwrite(): 0 N 75 1.02 Y Y N 0 N 294 475 0 N 288 12 Y Y Y 18 Y 222 456 3786

Answer 1

I got it to work sans fwrite() using this code:

        if(extension_loaded('curl')){
                    $curl = curl_init();
                    curl_setopt($curl, CURLOPT_URL, 'http://www.url.com');
                    curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
                    curl_setopt($curl, CURLOPT_TIMEOUT, 30);
                    curl_setopt($curl, CURLOPT_NOPROGRESS, FALSE);
                    curl_setopt($curl, CURLOPT_MAXCONNECTS, 1);
                    curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 0); 

                    $newparts = curl_exec($curl);

                    $xml = simplexml_load_string($newparts);

                foreach($xml->modpart as $item){

                      if(($item->active == 'Y') || ($item->active == 'YES')){
                          $status = '1';
                      } else {
                          $status = '0';
                      }


                if(!empty($item->mfr)){ 
                      $rs   = $this->db->query("SELECT manufacturer_id FROM ".DB_PREFIX."manufacturer WHERE name = '".$item->mfr->mfr_name."'");
                      $mfr_id = $rs->row['manufacturer_id'];
                }

                 if(!empty($item->codes)){
                          if($item->codes->code->fieldname == 'Mfr Part No'){
                              $mpn = $item->codes->code->fieldvalue;
                          } 
                  }
                      $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product (model, sku, mpn, quantity, date_available, sort_order, manufacturer_id, price, weight, status, date_added) 
                          VALUES ( 
                              '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."', 
                              '".mysql_real_escape_string($item->sku)."', 
                               '".$mpn."', 
                              '".mysql_real_escape_string($item->mpn)."', 
                               NOW(), 1000,
                              '".$mfr_id."',
                               '".mysql_real_escape_string($item->price)."', 
                               '".mysql_real_escape_string($item->weight)."', 
                               '".$status."', 
                                 NOW() 
                              )");    

                    $pr = $this->db->query("SELECT product_id FROM ".DB_PREFIX."product WHERE sku = '".$item->part."'");
                    $product_id = $pr->row['product_id'];

                    if(isset($product_id)){

                              $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_description (product_id, language_id, name, meta_keyword, meta_description, description, tag)
                                      VALUES(                                                       
                                              '" . (int)$product_id . "',
                                              '1',
                                              '" . $item->part. "',
                                              '" . $item->part. "',
                                              '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."', 
                                              '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."', 
                                              '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."' 
                                      )");


                            $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_to_store SET product_id = '" . (int)$product_id . "', store_id = '0'");

                            $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "url_alias SET query = 'product_id=" . (int)$product_id . "', keyword = '" .$item->part . "'");


                        foreach($item->cats as $cats){
                            $category = $cats->cat;
                                foreach($category as $category_id){
                                $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_to_category SET product_id = '" . (int)$product_id . "', category_id = '" . (int)$category_id . "'");
                                } 
                        } 
                             $this->cache->delete('product');
                    } 
            } 

            curl_close($curl);

    } 

I still have had no luck with the timeout issue, but at least I can get my products into the database by some means. I will post the timeout solution if/when I find it.