I want to monitor memory usage of my golang program and clean up some internal cache if system lacks of free memory.
The problem is that HeapAlloc / HeapInuse / HeapReleased stats aren't always add up properly (to my understanding).
I'm looking at free system memory (+ buffers/cache) - the value that is shown as available by free utility:
$ free
total used free shared buff/cache available
Mem: 16123232 409248 15113628 200 600356 15398424
Swap: 73242180 34560 73207620
And also I look at HeapIdle - HeapReleased, that, according to comments in https://godoc.org/runtime#MemStats,
HeapIdle minus HeapReleased estimates the amount of memory
that could be returned to the OS, but is being retained by
the runtime so it can grow the heap without requesting more
memory from the OS.
Now the problem: sometimes Available + HeapInuse + HeapIdle - HeapReleased exceeds total amount of system memory. Usually it happens when HeapIdle is quite high and HeapReleased is neither close to HeapIdle nor to zero:
# Start of test
Available: 15379M, HeapAlloc: 49M, HeapInuse: 51M, HeapIdle: 58M, HeapReleased: 0M
# Work is in progress
# Looks good: 11795 + 3593 = 15388
Available: 11795M, HeapAlloc: 3591M, HeapInuse: 3593M, HeapIdle: 0M, HeapReleased: 0M
# Work has been done
# Looks good: 11745 + 45 + 3602 = 15392
Available: 11745M, HeapAlloc: 42M, HeapInuse: 45M, HeapIdle: 3602M, HeapReleased: 0M
# Golang released some memory to OS
# Looks good: 15224 + 14 + 3632 - 3552 = 15318
Available: 15224M, HeapAlloc: 10M, HeapInuse: 14M, HeapIdle: 3632M, HeapReleased: 3552M
# Some other work started
# Looks SUSPICIOUS: 13995 + 1285 + 2360 - 1769 = 15871
Available: 13995M, HeapAlloc: 1282M, HeapInuse: 1285M, HeapIdle: 2360M, HeapReleased: 1769M
# 5 seconds later
# Looks BAD: 13487 + 994 + 2652 - 398 = 16735 - more than system memory
Available: 13487M, HeapAlloc: 991M, HeapInuse: 994M, HeapIdle: 2652M, HeapReleased: 398M
# This bad situation holds for quite a while, even when work has been done
# Looks BAD: 13488 + 14 + 3631 - 489 = 16644
Available: 13488M, HeapAlloc: 10M, HeapInuse: 14M, HeapIdle: 3631M, HeapReleased: 489M
# It is strange that at this moment HeapIdle - HeapReleased = 3142
# > than 2134M of used memory reported by "free" utility.
$ free
total used free shared buff/cache available
Mem: 16123232 2185696 13337632 200 599904 13621988
Swap: 73242180 34560 73207620
# Still bad when another set of work started
# Looks BAD: 13066 + 2242 + 1403 = 16711
Available: 13066M, HeapAlloc: 2240M, HeapInuse: 2242M, HeapIdle: 1403M, HeapReleased: 0M
# But after 10 seconds it becomes good
# Looks good: 11815 + 2325 + 1320 = 15460
Available: 11815M, HeapAlloc: 2322M, HeapInuse: 2325M, HeapIdle: 1320M, HeapReleased: 0M
I do not understand from where this additional "breathing" 1.3GB (16700 - 15400) of memory comes from. Used swap space remained the same during the whole test.
