dortmundbvb0624 2015-12-29 14:51
浏览 44
已采纳

有效地列出目录中包含很多条目的文件

I need to recursively read a directory structure, but I also need to perform an additional step once I have read through all entries for each directory. Therefore, I need to write my own recursion logic (and can't use the simplistic filepath.Walk routine). However, the ioutil.ReadDir and filepath.Glob routines only return slices. What if I'm pushing the limits of ext4 or xfs and have a directory with files numbering into the billions? I would expect golang to have a function that returns an unsorted series of os.FileInfo (or, even better, raw strings) over a channel rather than a sorted slice. How do we efficiently read file entries in this case?

All of the functions cited above seem to rely on readdirnames in os/dir_unix.go, and, for some reason, it just makes an array when it seems like it would've been easy to spawn a gothread and and push the values into a channel. There might have been sound logic to do this, but it's not clear what it is. I'm new to Go, so I also could've easy missed some principle that's obvious to everyone else.

This is the sourcecode, for convenience:

func (f *File) readdirnames(n int) (names []string, err error) {
    // If this file has no dirinfo, create one.
    if f.dirinfo == nil {
        f.dirinfo = new(dirInfo)
        // The buffer must be at least a block long.
        f.dirinfo.buf = make([]byte, blockSize)
    }
    d := f.dirinfo

    size := n
    if size <= 0 {
        size = 100
        n = -1
    }

    names = make([]string, 0, size) // Empty with room to grow.
    for n != 0 {
        // Refill the buffer if necessary
        if d.bufp >= d.nbuf {
            d.bufp = 0
            var errno error
            d.nbuf, errno = fixCount(syscall.ReadDirent(f.fd, d.buf))
            if errno != nil {
                return names, NewSyscallError("readdirent", errno)
            }
            if d.nbuf <= 0 {
                break // EOF
            }
        }

        // Drain the buffer
        var nb, nc int
        nb, nc, names = syscall.ParseDirent(d.buf[d.bufp:d.nbuf], n, names)
        d.bufp += nb
        n -= nc
    }
    if n >= 0 && len(names) == 0 {
        return names, io.EOF
    }
    return names, nil
}
  • 写回答

1条回答 默认 最新

  • drmq16019 2015-12-29 15:20
    关注

    ioutil.ReadDir and filepath.Glob are just convenience functions around reading directory entries.

    You can read directory entries in batches by directly using the Readdir or Readdirnames methods, if you supply an n argument > 0.

    For something as basic as reading directory entries, there's no need to add the overhead of a goroutine and channel, and also provide an alternate way to return the error. You can always wrap the batched calls with your own goroutine and channel pattern if you wish.

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题?

悬赏问题

  • ¥15 目详情-五一模拟赛详情页
  • ¥15 有了解d3和topogram.js库的吗?有偿请教
  • ¥100 任意维数的K均值聚类
  • ¥15 stamps做sbas-insar,时序沉降图怎么画
  • ¥15 买了个传感器,根据商家发的代码和步骤使用但是代码报错了不会改,有没有人可以看看
  • ¥15 关于#Java#的问题,如何解决?
  • ¥15 加热介质是液体,换热器壳侧导热系数和总的导热系数怎么算
  • ¥100 嵌入式系统基于PIC16F882和热敏电阻的数字温度计
  • ¥15 cmd cl 0x000007b
  • ¥20 BAPI_PR_CHANGE how to add account assignment information for service line