I'm writing yii2 command controller, that exports mongodb data to sphinx (as csv). MongoDB controller contains about 9M lines, and I've added a while cycle.
- public function actionExportSphinx()
- {
- $query = new MongoQuery;
- $count = (int) $query->select(['_id'])
- ->from('test')
- ->count();
- $i = 0;
-
- while($i < $count) {
- $rows = self::getExportData($i);
- self::printMongoRow($rows);
- unset($rows);
- $i += 100000;
- }
- }
Each next cycle loop is more slower than previous one. And on $i = 500000 mongo excepts timeout... I know, that it is a mongo skip problem, but I don't know any solutions of this.
UPD: Added self::getExportData() and self::printMongoRow() methods.
- public function printMongoRow($rows)
- {
- foreach ($rows as $row) {
- printf("%d,\"%s\",\"%s\",\"%s\",\"%s\",%d,%d,%d
- ",
- $row['_id'],
- (isset($row['lastname']) ? str_replace('"', "", $row['lastname']) : ""),
- (isset($row['firstname']) ? str_replace('"', "", $row['firstname']) : ""),
- (isset($row['middlename']) ? str_replace('"', "", $row['middlename']) : ""),
- (isset($row['town']) ? str_replace('"', "", $row['town']['title']) : ""),
- (isset($row['birthday']) ? $row['birthday'] : 0),
- (isset($row['birthmonth']) ? $row['birthmonth'] : 0),
- (isset($row['birthyear']) ? $row['birthyear'] : 0)
- );
- }
- }
-
- public function getExportData($i)
- {
- $query = new MongoQuery;
- $query->select(['lastname', 'firstname', 'middlename', 'town', 'birthday', 'birthmonth', 'birthyear'])
- ->from('test')
- ->limit(100000)
- ->offset($i);
- return $query->all();
- }