在一个巨大的站点中使用PHP的所有文件的列表

Question

I need to do a list of all files on a server from an other server.

I don't have access to PHP config like maximum timeout of the remote server. The maximum timeout could be very short like 30s. In some case, the following code gives a Timeout issue, because the iterator don't have enough time to get all the files.

public function getStructure($path)
    {
        $structure = new \stdClass();
        $structure->dirs = array();
        $structure->files = array();
        $iterator = new \RecursiveIteratorIterator(
            new \RecursiveDirectoryIterator($path), \RecursiveIteratorIterator::SELF_FIRST);
        foreach ($iterator as $file)
        {
            if ($file->isDir())
            {
                $structure->dirs[] = $file->getRealpath();
            }
            else
            {
                $structure->files[] = $file->getRealpath();
            }
        }

        return $structure;
    }

I'm looking for a way to get the structure in multiple calls. Something like : myremotesite.com/api/v1/structrue?start=xxxx where start is the point where the last call stop.

Thanks for your help

Answer 1

sounds like you need a dir with save/resume functionality... i would probably implement it in SQLite, due to its synchronous-by-default nature.

since i was bored.. UNTESTED, but should work in theory. DO NOT try to implement beginTransaction() / commit() optimizations to this code, that would defeat the whole "synchronous and tolerates crashing at any moment" part of the code;

<?php 
//will return bool(true) when it's finished creating the database.
//should be timeout/unstable system resistant, 
//relying on SQLite's syncronous-by-default nature.
function dirToSQLite($dir,$sqldb){
    if(!is_readable($dir)){
        throw new InvalidArgumentException('argument 1 must be a readable dir, but is not readable.');      
    }
    if(!is_dir($dir)){
        throw new InvalidArgumentException('argument 1 is not a valid dir');
    }
$db=new PDO('sqlite:'.$sqldb,'','',array(PDO::ATTR_EMULATE_PREPARES => false,PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION));
$db->exec('CREATE TABLE IF NOT EXISTS `dir` (`id` INTEGER PRIMARY KEY,`path` TEXT UNIQUE,`type` TEXT);');
$db->query('INSERT OR IGNORE INTO `dir` (`path`,`type`) VALUES('.$db->quote($dir).',\'dirUnexplored\');');
$stm=$db->prepare('INSERT INTO `dir` (`path`,`type`) VALUES(:path,:type);');
$stmExplored=$db->prepare('UPDATE `dir` SET `type` = \'dir\' WHERE id = ? ');
$path='';
$type='';
$stm->bindParam(':path',$path,PDO::PARAM_STR);
$stm->bindParam(':type',$type,PDO::PARAM_STR);
while(true){
    $found=false;
    foreach($db->query('SELECT `id`,`path` FROM `dir` WHERE `type` = \'dirUnexplored\'') as $res)
    {
        $found=true;
        $di=new DirectoryIterator($res['path']);
        foreach($di as $file){
            if($file->isDot()){
                continue;
            } else
            if($file->isLink()){
                $type='link';
            } else
            if($file->isDir()){
                $type='dirUnexplored';
            } else
            if($file->isFile()){
                $type='file';
            } else
            {
                $type='unknown';
            }
            $path=$file->getPathname();
            $stm->execute();
        }
        $stmExplored->execute(array($res['id']));
    }
    if(!$found){
        break;
    }
}
return true;
}
if(true===dirToSqlite('/home/foo','homefoo.db3')){
echo "finished!";
}else {
throw new Exception();
}

then just keep calling that url until it returns the string "finished!", then you can download the SQLite database directly, no php involved in the download.