The following code declares two arrays, and then iterates over stdin ( just blindly iterates over the file - no interaction with the arrays ).
This is causing continuous increase in memory.
However, if I just declare two arrays and sleep - there is no increase in memory.
Similarly, if I just iterate over stdin - there is no increase in memory.
But together ( apart from the memory allocated for the arrays) there is a continuous increase.
I measure this by looking at the RES memory using top tool.
I have commented out the first few lines in func doSomething() to show that there is no memory increase when it is commented. Uncommenting the lines and running will cause an increase.
NOTE: This was run on go 1.4.2, 1.5.3 and 1.6
NOTE: You will need to recreate this on a machine with at least 16GB RAM as I have observed it only on the array size of 1 billion.
package main
import (
"bufio"
"fmt"
"io"
"os"
)
type MyStruct struct {
arr1 []int
arr2 []int
}
func (ms *MyStruct) Init(size int, arr1 []int, arr2 []int) error {
fmt.Printf("initializing mystruct arr1...
")
ms.arr1 = arr1
if ms.arr1 == nil {
ms.arr1 = make([]int, size, size)
}
fmt.Printf("initializing mystruct arr2...
")
ms.arr2 = arr2
if ms.arr2 == nil {
ms.arr2 = make([]int, size, size)
}
fmt.Printf("done initializing ...
")
for i := 0; i < size; i++ {
ms.arr1[i] = 0
ms.arr2[i] = 0
}
return nil
}
func doSomething() error {
fmt.Printf("starting...
")
fmt.Printf("allocating
")
/* NOTE WHEN UNCOMMENTED CAUSES MEMORY INCREASE
ms := &MyStruct{}
size := 1000000000
ms.Init(size, nil, nil)
*/
fmt.Printf("finished allocating..%d %d
", len(ms.arr1), len(ms.arr2))
fmt.Printf("reading from stdin...
")
reader := bufio.NewReader(os.Stdin)
var line string
var readErr error
var lineNo int = 0
for {
if lineNo%1000000 == 0 {
fmt.Printf("read %d lines...
", lineNo)
}
lineNo++
line, readErr = reader.ReadString('
')
if readErr != nil {
fmt.Printf("break at %s
", line)
break
}
}
if readErr == io.EOF {
readErr = nil
}
if readErr != nil {
return readErr
}
return nil
}
func main() {
if err := doSomething(); err != nil {
panic(err)
}
fmt.Printf("done...
")
}
- Is this an issue with my code ? Or is the go system doing something unintended ?
- If its the latter, how can I go about debugging this ?
To make it easier to replicate here are pastebin files for good case ( commented portion of the above code) and bad case ( with uncommented portion )
wget http://pastebin.com/raw/QfG22xXk -O badcase.go
yes "1234567890" | go run badcase.go
wget http://pastebin.com/raw/G9xS2fKy -O goodcase.go
yes "1234567890" | go run goodcase.go
