I need to recursively read a directory structure, but I also need to perform an additional step once I have read through all entries for each directory. Therefore, I need to write my own recursion logic (and can't use the simplistic filepath.Walk
routine). However, the ioutil.ReadDir
and filepath.Glob
routines only return slices. What if I'm pushing the limits of ext4 or xfs and have a directory with files numbering into the billions? I would expect golang to have a function that returns an unsorted series of os.FileInfo
(or, even better, raw strings) over a channel rather than a sorted slice. How do we efficiently read file entries in this case?
All of the functions cited above seem to rely on readdirnames
in os/dir_unix.go, and, for some reason, it just makes an array when it seems like it would've been easy to spawn a gothread and and push the values into a channel. There might have been sound logic to do this, but it's not clear what it is. I'm new to Go, so I also could've easy missed some principle that's obvious to everyone else.
This is the sourcecode, for convenience:
func (f *File) readdirnames(n int) (names []string, err error) {
// If this file has no dirinfo, create one.
if f.dirinfo == nil {
f.dirinfo = new(dirInfo)
// The buffer must be at least a block long.
f.dirinfo.buf = make([]byte, blockSize)
}
d := f.dirinfo
size := n
if size <= 0 {
size = 100
n = -1
}
names = make([]string, 0, size) // Empty with room to grow.
for n != 0 {
// Refill the buffer if necessary
if d.bufp >= d.nbuf {
d.bufp = 0
var errno error
d.nbuf, errno = fixCount(syscall.ReadDirent(f.fd, d.buf))
if errno != nil {
return names, NewSyscallError("readdirent", errno)
}
if d.nbuf <= 0 {
break // EOF
}
}
// Drain the buffer
var nb, nc int
nb, nc, names = syscall.ParseDirent(d.buf[d.bufp:d.nbuf], n, names)
d.bufp += nb
n -= nc
}
if n >= 0 && len(names) == 0 {
return names, io.EOF
}
return names, nil
}