I'm writing yii2 command controller, that exports mongodb data to sphinx (as csv). MongoDB controller contains about 9M lines, and I've added a while cycle.
public function actionExportSphinx()
{
$query = new MongoQuery;
$count = (int) $query->select(['_id'])
->from('test')
->count();
$i = 0;
while($i < $count) {
$rows = self::getExportData($i);
self::printMongoRow($rows);
unset($rows);
$i += 100000;
}
}
Each next cycle loop is more slower than previous one. And on $i = 500000 mongo excepts timeout... I know, that it is a mongo skip problem, but I don't know any solutions of this.
UPD: Added self::getExportData() and self::printMongoRow() methods.
public function printMongoRow($rows)
{
foreach ($rows as $row) {
printf("%d,\"%s\",\"%s\",\"%s\",\"%s\",%d,%d,%d
",
$row['_id'],
(isset($row['lastname']) ? str_replace('"', "", $row['lastname']) : ""),
(isset($row['firstname']) ? str_replace('"', "", $row['firstname']) : ""),
(isset($row['middlename']) ? str_replace('"', "", $row['middlename']) : ""),
(isset($row['town']) ? str_replace('"', "", $row['town']['title']) : ""),
(isset($row['birthday']) ? $row['birthday'] : 0),
(isset($row['birthmonth']) ? $row['birthmonth'] : 0),
(isset($row['birthyear']) ? $row['birthyear'] : 0)
);
}
}
public function getExportData($i)
{
$query = new MongoQuery;
$query->select(['lastname', 'firstname', 'middlename', 'town', 'birthday', 'birthmonth', 'birthyear'])
->from('test')
->limit(100000)
->offset($i);
return $query->all();
}