I'm currently trying to download satellite images from esa's Copernicus / Sentinel project with curl. Unfortunately the download keeps stopping at around 90% and the php script returns an Internal Server Error (500).
Therefore I would like to resume the download at a specific byte number. It seems that the esa server just ignores the http-range-header (CURLOPT_RANGE) and CURLOPT_RESUME_FROM doesn't change anything either.
If I use Google Chrome to download the file manually, the download also interrupts but continues after some time.
So, if Google Chrome can resume the download, curl should be able to do that, too. I would appreciate any help on how to do that.
Some details:
The file I'm trying to download is here (420MB), to access it you need to register at scihub.esa.int/dhus/.
Content-Type is application/octet-stream
My code:
$save_file = fopen($save_filepath, "w+");
$open_file = curl_init(str_replace(" ","%20", $url));
curl_setopt($open_file, CURLOPT_USERPWD, $username.":".$password);
curl_setopt($open_file, CURLOPT_TIMEOUT, 300);
curl_setopt($open_file, CURLOPT_FILE, $save_file);
curl_setopt($open_file, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($open_file, CURLOPT_PROGRESSFUNCTION, "trackprogress");
curl_setopt($open_file, CURLOPT_NOPROGRESS, false);
curl_exec($open_file);
curl_close($open_file);
fclose($save_file);
It works perfectly for smaller files (I've tested it with some images and pdf-files) and I can also download most of the satellite image (the first 380MB are downloaded). I tried to increase the timeout value, too, but the script terminates long before the 5 minutes are reached.
I tried curl_setopt($open_file, CURLOPT_RESUME_FROM, 1048576);
and curl_setopt($open_file, CURLOPT_RANGE, "1048576-");
but the file always starts with the same bytes.
EDIT:
I can't answer my question, but for this specific case I found a workaround. So, if anybody reads this and also wants to download these satellite images with cURL by chance, here is what I did:
When downloading not just the image file, but the zip-file with some additional data, the download still keeps stopping, however with curl_setopt($open_file, CURLOPT_RESUME_FROM, $bytes_already_loaded);
it is possible to skip the bytes which had previously been loaded and resume the download (which isn't possible for the image file). Thus, use this link instead of the image file.