My Go code uses several hundreds of goroutines. A runtime error could occur from time to time. But when error occurs, it will simply print out the stack traces of all goroutines, making it impossible to debug?
How to locate where the program breaks?
Sry I didn't post stack traces earlier, I didn't know how to print stderr to stack and the output is too long so I can't view all of it.
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x141edce pc=0x141edce]
runtime stack:
runtime: unexpected return pc for runtime.sigpanic called from 0x141edce
stack: frame={sp:0x7ffbffffa9f0, fp:0x7ffbffffaa40} stack=[0x7ffbff7fbb80,0x7ffbffffabb0)
00007ffbffffa8f0: 00007ffbffffa960 000000000042b58c <runtime.dopanic_m+540>
00007ffbffffa900: 000000000042b031 <runtime.throw+129> 00007ffbffffa9d0
00007ffbffffa910: 0000000000000000 000000000097f880
00007ffbffffa920: 010000000042bae8 0000000000000004
00007ffbffffa930: 000000000000001f 000000000141edce
00007ffbffffa940: 000000000141edce 0000000000000001
00007ffbffffa950: 00000000007996e6 000000c420302180
00007ffbffffa960: 00007ffbffffa988 00000000004530ac <runtime.dopanic.func1+60>
00007ffbffffa970: 000000000097f880 000000000042b031 <runtime.throw+129>
00007ffbffffa980: 00007ffbffffa9d0 00007ffbffffa9c0
00007ffbffffa990: 000000000042af5a <runtime.dopanic+74> 00007ffbffffa9a0
00007ffbffffa9a0: 0000000000453070 <runtime.dopanic.func1+0> 000000000097f880
00007ffbffffa9b0: 000000000042b031 <runtime.throw+129> 00007ffbffffa9d0
00007ffbffffa9c0: 00007ffbffffa9e0 000000000042b031 <runtime.throw+129>
00007ffbffffa9d0: 0000000000000000 000000000000002a
00007ffbffffa9e0: 00007ffbffffaa30 000000000043fb1e <runtime.sigpanic+654>
00007ffbffffa9f0: <000000000079dce7 000000000000002a
00007ffbffffaa00: 00007ffbffffaa30 000000000041f08e <runtime.greyobject+302>
00007ffbffffaa10: 000000c420029c70 000000000097f880
00007ffbffffaa20: 000000000045247d <runtime.markroot.func1+109> 000000c420a69b00
00007ffbffffaa30: 00007ffbffffaad8 !000000000141edce
00007ffbffffaa40: >000000c42160ca40 000000c4206d8000
00007ffbffffaa50: 0000000000000c00 000000c41ff4f9ad
00007ffbffffaa60: 000000c400000000 00007efbff5188f8
00007ffbffffaa70: 000000c420029c70 0000000000000052
00007ffbffffaa80: 0000000021e84000 00007ffbffffaab0
00007ffbffffaa90: 0000000000002000 0000000000000c00
00007ffbffffaaa0: 000000c422b00000 000000c420000000
00007ffbffffaab0: 00007ffbffffaad8 0000000000421564 <runtime.(*gcWork).tryGet+164>
00007ffbffffaac0: 000000c41ffc939f 000000c4226eb000
00007ffbffffaad0: 000000c4226e9000 00007ffbffffab30
00007ffbffffaae0: 000000000041e527 <runtime.gcDrain+567> 000000c4206d8000
00007ffbffffaaf0: 000000c420029c70 0000000000000000
00007ffbffffab00: 7ffffffffff8df47 00007ffc0001fc30
00007ffbffffab10: 00007ffbffffab70 0000000000000000
00007ffbffffab20: 000000c420302180 0000000000000000
00007ffbffffab30: 00007ffbffffab70 00000000004522c0 <runtime.gcBgMarkWorker.func2+128>
runtime.throw(0x79dce7, 0x2a)
/usr/lib/go-1.10/src/runtime/panic.go:616 +0x81
runtime: unexpected return pc for runtime.sigpanic called from 0x141edce
stack: frame={sp:0x7ffbffffa9f0, fp:0x7ffbffffaa40} stack=[0x7ffbff7fbb80,0x7ffbffffabb0)
00007ffbffffa8f0: 00007ffbffffa960 000000000042b58c <runtime.dopanic_m+540>
00007ffbffffa900: 000000000042b031 <runtime.throw+129> 00007ffbffffa9d0
00007ffbffffa910: 0000000000000000 000000000097f880
00007ffbffffa920: 010000000042bae8 0000000000000004
00007ffbffffa930: 000000000000001f 000000000141edce
00007ffbffffa940: 000000000141edce 0000000000000001
00007ffbffffa950: 00000000007996e6 000000c420302180
00007ffbffffa960: 00007ffbffffa988 00000000004530ac <runtime.dopanic.func1+60>
00007ffbffffa970: 000000000097f880 000000000042b031 <runtime.throw+129>
00007ffbffffa980: 00007ffbffffa9d0 00007ffbffffa9c0
00007ffbffffa990: 000000000042af5a <runtime.dopanic+74> 00007ffbffffa9a0
00007ffbffffa9a0: 0000000000453070 <runtime.dopanic.func1+0> 000000000097f880
00007ffbffffa9b0: 000000000042b031 <runtime.throw+129> 00007ffbffffa9d0
00007ffbffffa9c0: 00007ffbffffa9e0 000000000042b031 <runtime.throw+129>
00007ffbffffa9d0: 0000000000000000 000000000000002a
00007ffbffffa9e0: 00007ffbffffaa30 000000000043fb1e <runtime.sigpanic+654>
00007ffbffffa9f0: <000000000079dce7 000000000000002a
00007ffbffffaa00: 00007ffbffffaa30 000000000041f08e <runtime.greyobject+302>
00007ffbffffaa10: 000000c420029c70 000000000097f880
00007ffbffffaa20: 000000000045247d <runtime.markroot.func1+109> 000000c420a69b00
00007ffbffffaa30: 00007ffbffffaad8 !000000000141edce
00007ffbffffaa40: >000000c42160ca40 000000c4206d8000
00007ffbffffaa50: 0000000000000c00 000000c41ff4f9ad
00007ffbffffaa60: 000000c400000000 00007efbff5188f8
00007ffbffffaa70: 000000c420029c70 0000000000000052
00007ffbffffaa80: 0000000021e84000 00007ffbffffaab0
00007ffbffffaa90: 0000000000002000 0000000000000c00
00007ffbffffaaa0: 000000c422b00000 000000c420000000
00007ffbffffaab0: 00007ffbffffaad8 0000000000421564 <runtime.(*gcWork).tryGet+164>
00007ffbffffaac0: 000000c41ffc939f 000000c4226eb000
00007ffbffffaad0: 000000c4226e9000 00007ffbffffab30
00007ffbffffaae0: 000000000041e527 <runtime.gcDrain+567> 000000c4206d8000
00007ffbffffaaf0: 000000c420029c70 0000000000000000
00007ffbffffab00: 7ffffffffff8df47 00007ffc0001fc30
00007ffbffffab10: 00007ffbffffab70 0000000000000000
00007ffbffffab20: 000000c420302180 0000000000000000
00007ffbffffab30: 00007ffbffffab70 00000000004522c0 <runtime.gcBgMarkWorker.func2+128>
runtime.sigpanic()
/usr/lib/go-1.10/src/runtime/signal_unix.go:372 +0x28e