用php将文件大量插入couchbase数据库 - 如何？

Question

I am experimenting actually a little bit with couchbase server.

I have tried to read a mysql database table, build a document from each data row and then inserting the document with an id which I generate with

uniqid('table_name'); 

via cUrl, method is POST.

This so far works pretty good, until the script has inserted roundabout 7050 documents. Then an exception is thrown -> "No buffer space".

Until now I was not able to fix this, so I decided to collect i.e. 50 rows of data build a json_encode(d) string and POST it again via cUrl.

This worked so far if I don't set the id - but I can't figure out how to set the id of the inserted documents.

Actually I try to send my documents in a format like this:

{"docs": {
   "_id": {
          "geodata_de_54476f7e6adc57.14196038": {
              "table": "geodata_de",
              "country": "DE",
              "postal_code": "01945",
              "place_name": "Lindenau",
              "state_name": "Brandenburg",
              "state_code": "BB",
              "province_name": "",
              "province_code": "00",
              "community_name": "Oberspreewald-Lausitz",
              "community_code": "12066",
              "lat": "51.4",
              "lng": "13.7333",
              "Xco": "3861.1",
              "Yco": "943.614",
              "Zco": "4979.07"
           } 
       },  ...
   }

}

but this just inserts ONE document with the above object.

Maybe there is someone here who can point me the right direction.

Answer 1

I would use the Couchbase PHP SDK to insert these documents instead of using curl. http://docs.couchbase.com/developer/php-2.0/storing.html

Also for CB, you do not have to set the ID in the document itself. it depends. I might take a look at instead using the ID you have in your example ("geodata_de_54476f7e6adc57.14196038") and put it as the key for the object in Couchbase. Then you do not necessarily need the _id. The key in Couchbase can be up to 250 bytes of data and you can make it meaningful to your application so you can do lookup by key extremely fast.

Another option is, if you wrote your docs to the filesystem, you could also use cbdocloader utility which is specifically for bulk loading docs. If you are on linux it is in /opt/couchbase/bin/tools/cbdocloader.