I am attempting to do what I originally thought would be pretty simple. To wit:
For every file in a list of input files:
- open the file with png.Decode()
- scan every pixel in the file and test to see if it is "grey".
- Return the percentage of "grey" pixels in the image.
This is the function I am calling:
func greyLevel(fname string) (float64, string) {
f, err := os.Open(fname)
if err != nil {
return -1.0, "can't open file"
}
defer f.Close()
i, err := png.Decode(f)
if err != nil {
return -1.0, "unable to decode"
}
bounds := i.Bounds()
var lo uint32 = 122 // Low grey RGB value.
var hi uint32 = 134 // High grey RGB value.
var gpix float64 // Grey pixel count.
var opix float64 // Other (non-grey) pixel count.
var tpix float64 // Total pixels.
for x := bounds.Min.X; x < bounds.Max.X; x++ {
for y := bounds.Min.Y; y < bounds.Max.Y; y++ {
r, g, b, _ := i.At(x, y).RGBA()
if ((r/255)-1 > lo && (r/255)-1 < hi) &&
((g/255)-1 > lo && (g/255)-1 < hi) &&
((b/255)-1 > lo && (b/255)-1 < hi) {
gpix++
} else {
opix++
}
tpix++
}
}
return (gpix / tpix) * 100, ""
}
func main() {
srcDir := flag.String("s", "", "Directory containing image files.")
threshold := flag.Float64("t", 65.0, "Threshold (in percent) of grey pixels.")
flag.Parse()
dirlist, direrr := ioutil.ReadDir(*srcDir)
if direrr != nil {
log.Fatalf("Error reading %s: %s
", *srcDir, direrr)
}
for f := range dirlist {
src := path.Join(*srcDir, dirlist[f].Name())
level, msg := greyLevel(src)
if msg != "" {
log.Printf("error processing %s: %s
", src, msg)
continue
}
if level >= *threshold {
log.Printf("%s is grey (%2.2f%%)
", src, level)
} else {
log.Printf("%s is not grey (%2.2f%%)
", src, level)
}
}
}
The files are relatively small (960x720, 8-bit RGB)
I am calling ioutil.ReadDir() to generate a list of files, looping over the slice and calling greyLevel().
After about 155 files (out of a list of >4000) the script panics with:
runtime: memory allocated by OS not in usable range
runtime: out of memory: cannot allocate 2818048-byte block (534708224 in use)
throw: out of memory
I figure there is something simple I am missing. I thought that Go would de-allocate the memory allocated in greyLevels() but I guess not?
Follow up:
After inserting runtime.GC() after every call to greyLevels, the memory usage evens out. Last night I was teting to about 800 images then stopped. Today I let it run over the entire input set, approximately 6800 images.
After 1500 images, top looks like this:
top - 10:30:11 up 41 days, 11:47, 2 users, load average: 1.46, 1.25, 0.88
Tasks: 135 total, 2 running, 131 sleeping, 1 stopped, 1 zombie
Cpu(s): 49.8%us, 5.1%sy, 0.2%ni, 29.6%id, 15.0%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 3090304k total, 2921108k used, 169196k free, 2840k buffers
Swap: 3135484k total, 31500k used, 3103984k free, 640676k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28474 mtw 20 0 2311m 1.8g 412 R 99 60.5 16:48.52 8.out
And remained steady after processing another 5000 images.