大量分配的数据块导致巨大的GC性能问题

Question

I have just noticed that if I allocate a huge memory block in programm. GC will eat all programm time.

Here is POC.

https://gist.github.com/martende/252f403f0c17cb489de4

func main() {
    //////////////// !!!!!!!
    /* If I uncomment 2 lines below programm runs fast */
    nodesPool := make([]int, 300e6, 300e6)
    _ = nodesPool
    //////////////////////7

    file, _ := os.Open("result.txt")
    defer file.Close()

    reader := bufio.NewReader(file)

    var lastLinkIdx = 1 // Dont use first element use 0 as saver

    cnt:=  0
    totalB := 0

    for {
       l, err := reader.ReadString('
')
       if err == io.EOF {
    fmt.Println("EOF")
        break
    }

    cnt+=1      
    totalB+=len(l)

    lines := strings.Split(l, ":")
    nodeId,_ := strconv.Atoi(lines[0])
    _ = nodeId

    linkIdsStr  := strings.Split(lines[1], ",")
    var ii = len(linkIdsStr)
    _ = ii
    /*      ... */
}

fmt.Println("pool ",cnt,totalB,lastLinkIdx)


}

I think that GC tries somehow move the huge memory block , is it actually possible to allocate memory out of GC but leave GC for all other libraries becuase even ReadLine need it.

Here is profiling with memory block.

Total: 1445 samples
     428  29.6%  29.6%      722  50.0% runtime.sweepone
     375  26.0%  55.6%      375  26.0% markroot
     263  18.2%  73.8%      263  18.2% runtime.xadd
     116   8.0%  81.8%      116   8.0% strings.Count
      98   6.8%  88.6%      673  46.6% strings.genSplit
      34   2.4%  90.9%       44   3.0% runtime.MSpan_Sweep
      25   1.7%  92.7%      729  50.4% MCentral_Grow
      17   1.2%  93.8%       19   1.3% syscall.Syscall
       9   0.6%  94.5%        9   0.6% runtime.memclr
       9   0.6%  95.1%        9   0.6% runtime.memmove

Here is profiling without memory block.

  98  27.0%  27.0%       98  27.0% strings.Count
  93  25.6%  52.6%      228  62.8% strings.genSplit
  45  12.4%  65.0%       45  12.4% scanblock
  24   6.6%  71.6%       28   7.7% runtime.MSpan_Sweep
  13   3.6%  75.2%       74  20.4% runtime.mallocgc
  12   3.3%  78.5%       12   3.3% runtime.memclr
   8   2.2%  80.7%        8   2.2% MHeap_ReclaimList
   8   2.2%  82.9%       11   3.0% syscall.Syscall
   6   1.7%  84.6%       44  12.1% MHeap_Reclaim
   6   1.7%  86.2%        6   1.7% markonly

Answer 1

Dmitry Vyukov of the Go team says this is a Go runtime performance issue you can trigger with a huge allocation, and that as a workaround, "you can collect the large object as soon as it become[s] dead and increase GOGC right after that."

Broadly, the GitHub issue says that the runtime creates a lot of memory-management structures (spans) that it then keeps around indefinitely and has to sweep on every GC. Going by the issue tags, a fix is targeted for Go 1.5.

His sample with workaround is:

package main

import (
    "runtime"
    "runtime/debug"
)

var x = make([]byte, 1<<20)
var y []byte
var z []byte

func main() {
    y = make([]byte, 1<<30)
    y = nil
    runtime.GC()
    debug.SetGCPercent(1000)
    for i := 0; i < 1e6; i++ {
        z = make([]byte, 8192)
    }
}

(Some comments are about a totally different answer and code sample focused on avoiding allocations that I've edited out. There's no way to "tell" StackOverflow this is a new answer, so they remain.)