douwen4125 2011-02-03 16:49
浏览 120
已采纳

MySQL,使用LIMIT 0,1800到LIMIT 72000,18000循环(40次)来创建平面文件,必须是更加DB的友好方式吗?

I'm creating some flat files from my MySQL database in a php job. Each file is 225kb. I create 40 files.

Basically, I'm running a PHP script which calls a query, the first time it has LIMIT 0, 1800. It then loops and runs 40 times and the last query uses LIMIT 72000,1800.

In the loop I sleep for 0.7 seconds. The whole process takes 45 seconds.

Heres some debug info I produced.

Query took 0.03 second for ../sitemap-0.xml done!
Query took 0.06 second for ../sitemap-1800.xml done!
..snip ..
Query took 0.9 second for ../sitemap-70200.xml done!
Query took 0.9 second for ../sitemap-72000.xml done!
Took: 44.7057

You'll notice the queries take longer and longer as the job runs, this must be the amount of data that the query needs to look at to determine the position of the LIMIT. As a result the CPU usage increases with each query.

The maximum cpu times limit on the server is 60 seconds.

My prime consideration is to keep cpu time and database query time as low as possible. Having high values is unacceptable.

Running the query over and over seems a bit waste of resources.

Is there a better way to do this ?

  • 写回答

3条回答 默认 最新

  • dongzhang1875 2011-02-03 16:55
    关注

    The better solution would be to use unbuffered results (so it returns the result without waiting for all of the results to finish transferring) with no limit.

    So, if you're using the mysql extension (using mysql_unbuffered_query ):

    $sql = 'SELECT a, bunch, of, data FROM a_big_table WHERE some_condition';
    $result = mysql_unbuffered_query($sql);
    $data = array();
    $count = 0;
    while ($row = mysql_fetch_assoc($result)) {
        $count++;
        $data[] = $row;
        if ($count >= 1800) {
            storeData($data);
            $data = array();
            $count = 0;
        }
    }
    if ($count > 0) {
        storeData($data);
    }
    

    Where the function storeData actually writes the files.

    The benefit of this, is two fold. First, the query only executes once, so you're not re-running things multiple times. Second, it's unbuffered so you can start fetching results immediately rather than waiting for the whole query to finish.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(2条)

报告相同问题?

悬赏问题

  • ¥15 本题的答案是不是有问题
  • ¥15 关于#r语言#的问题:(svydesign)为什么在一个大的数据集中抽取了一个小数据集
  • ¥15 C++使用Gunplot
  • ¥15 这个电路是如何实现路灯控制器的,原理是什么,怎么求解灯亮起后熄灭的时间如图?
  • ¥15 matlab数字图像处理频率域滤波
  • ¥15 在abaqus做了二维正交切削模型,给刀具添加了超声振动条件后输出切削力为什么比普通切削增大这么多
  • ¥15 ELGamal和paillier计算效率谁快?
  • ¥15 蓝桥杯单片机第十三届第一场,整点继电器吸合,5s后断开出现了问题
  • ¥15 file converter 转换格式失败 报错 Error marking filters as finished,如何解决?
  • ¥15 Arcgis相交分析无法绘制一个或多个图形