是什么导致malloc忽略交换空间？

Question

I have setup a 4GB swapfile using dphys-swapfile additionally to the 1GB physical memory on a Raspberry Pi3 B+. The default was 100MB. I am running a go application, whose memory usage sometime peaks at about 1.2GB. However, even having 4GB swap memory, this peak causes an OOM error:

runtime: out of memory: cannot allocate 137035776-byte block (1053458432 in use)
fatal error: out of memory

runtime stack:
runtime.throw(0x6a2e8e, 0xd)
        /usr/local/go/src/runtime/panic.go:616 +0x60
runtime.largeAlloc(0x82ab0fc, 0x101, 0x11160b60)
        /usr/local/go/src/runtime/malloc.go:828 +0xdc
runtime.mallocgc.func1()
        /usr/local/go/src/runtime/malloc.go:721 +0x38
runtime.systemstack(0x0)
        /usr/local/go/src/runtime/asm_arm.s:349 +0x80
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1175

Why does malloc ignore the available swap space?

root@raspberrypi:~# sysctl vm.swappiness
vm.swappiness = 100
root@raspberrypi:~# sysctl vm.overcommit_memory
vm.overcommit_memory = 1
root@raspberrypi:~# free
              total        used        free      shared  buff/cache   available
Mem:         949452       36448      847880        1148       65124      862884
Swap:       4194300       30464     4163836
root@raspberrypi:~# cat /proc/meminfo
MemTotal:         949452 kB
MemFree:          847880 kB
MemAvailable:     862900 kB
Buffers:           10476 kB
Cached:            46848 kB
SwapCached:          872 kB
Active:            32140 kB
Inactive:          37728 kB
Active(anon):      10008 kB
Inactive(anon):     3692 kB
Active(file):      22132 kB
Inactive(file):    34036 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4194300 kB
SwapFree:        4163836 kB
Dirty:                20 kB
Writeback:             0 kB
AnonPages:         12072 kB
Mapped:            13024 kB
Shmem:              1148 kB
Slab:              18452 kB
SReclaimable:       7816 kB
SUnreclaim:        10636 kB
KernelStack:         888 kB
PageTables:         1140 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     4669024 kB
Committed_AS:     114252 kB
VmallocTotal:    1114112 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
CmaTotal:           8192 kB
CmaFree:            6780 kB

Setting vm.swappiness to 1, 10, 60, 100 or vm.overcommit_memory has no effect on this OOM.

My understanding was that OS would try to swap as much data as possible from the physical memory to avoid an OOM crash. But in this case it seems to just ignore it. What could be the reason?

ulimit values for completeness:

root@raspberrypi:~# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7345
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7345
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Answer 1

You may be hitting memory fragmentation if you're running a 32-bit OS. It is very possible to reach a point where you can't even allocate a couple of megabytes because every possible virtual address is filled with something.

This is especially bad with languages that use a lot of threads like Java, .NET or Go. Each thread gets its own stack which is placed somewhere that it can grow if it needs to.

And combined with ASLR for security randomization, 32-bit memory gets fragmented very quickly.

You could also be hitting the maximum map count. This even affects 64-bit systems but I haven't seen it happen with Go. Rust hits it a lot. Linux defaults this to 64k maps. I've increased it on my Linux systems:

$ cat /etc/sysctl.d/20-max-maps.conf 
vm.max_map_count = 262144