I want to curl a large file to Hoop using php. If I do a normal php file upload there's headers prepended to the file.
When I try this:
$url = http://hoop:14000/filename?op=create&user.name=root
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, array("file" => "@" . $this->filepath));
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/octet-stream', 'Expect:'));
$content = curl_exec($ch);
The file on hoop will have these headers:
------------------------------f0f063939ed8
Content-Disposition: form-data; name="file"; filename="phpbsA4ty"
Content-Type: application/octet-stream
{binary data here........}
I'm guessing it needs to be raw post data. So I can get it to work like this:
$url = http://hoop:14000/filename?op=create&user.name=root
$fileData = file_get_contents($this->filepath);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fileData);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/octet-stream', 'Expect:'));
$content = curl_exec($ch);
But large files cause memory errors:
PHP Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 8388608 bytes)
Is there a way to post raw files without loading the file into memory?
I can do it on the command line using the instructions from the Hoop documentation
curl -X POST -c ~/.hoopauth "http://<HOOP_HOST>:14000/<PATH>?op=create[&<OPTION>]*" \ --data-binary @data.txt --header "content-type: application/octet-stream"
from http://cloudera.github.com/hoop/docs/latest/HttpRestApi.html