刮不止一页

Question

I am trying to scape data (Name, varietal, format and price) from this site https://aabalat.com/wine/country/france. I have made an array by name $urls and I push every link in the array. For each new curl session, I will get 20 new data about wine. I need to capture format at first and push to the array as you can see on my code below. When I print $french_wines_formats_matches it work successfully. But when I want to print $french_wines_format_array it is not working very well.

I am new in scraping and I am not much experience with that.

    // Array contains 197 links
$urls = array();
array_push($urls, "https://aabalat.com/wine/country/france");


// This for loop makes others links
for($i = 1; $i < 5; $i++)
{
  $urls[] = "https://aabalat.com/wine/country/france?page=".$i;
}

// echo "<pre>";
// print_r($urls);
// echo "</pre>";

$french_wines_array = array();
$french_wines_title_array = array();
$french_wines_varietal_array = array();
$french_wines_format_array = array();
$french_wines_price_array = array();

// Repeat curl session until url exists.
foreach($urls as $url)
{
  $curl = curl_init();
  curl_setopt($curl, CURLOPT_URL, $url);

  curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($curl, CURLOPT_VERBOSE, true);

  $output = curl_exec($curl);
  $info = curl_getinfo($curl);
  $err = curl_error($curl);
  $ern = curl_errno($curl);

  $french_wine_formats_pattern = '!<span class="wine-list-item-format">(.*)</span>!mi';
  preg_match_all($french_wine_formats_pattern, $output, $french_wines_formats_matches);

  foreach($french_wines_formats_matches[0] as $french_wines_formats_match)
  {
    $french_wines_format_array[] = $french_wines_formats_match;
  }

  echo "<pre>";
  print_r($french_wines_format_array);
  echo "</pre>";

curl_close($curl);
sleep(rand(2, 5));

}

Answer 1

Your code and regex seem to work (I tried them). I was unable to replicate your cURL call. Try the following instead of just $output = curl_exec($curl), see if you catch any cURL errors:

    if(!$output = curl_exec($curl)){
        if (curl_error($ch)) {
            die(curl_error($ch));
        }
    }

Finally, I tried a simple file_get_contents() and that seemed to work:

    $url = "https://aabalat.com/wine/country/france";
    $output= file_get_contents($url);