mongoimport在Mac Sierra 10.12.2上引发随机MSpanList_Insert错误

Question

My goal is importing a dump from Modulus into my local MongoDB. I have a script that took all of the JSON files and imported them one by one. The script worked great until I installed Sierra. Here is the script:

#!/bin/bash
FILES=/Users/username/Documents/ModulusDumps/1-12-17/*
for f in $FILES
do
  filename="${f##*/}"
  basename="${filename%.[^.]*}"
  echo "$basename $filename"
  mongoimport --db meteor --collection $basename --type json --file $f --drop -h 127.0.0.1:3001
done

Now, after the Sierra install, the script still runs through the JSON files and works, but at random places in the import I get the following:

failed MSpanList_Insert 0x7a0100 0x142f1b79b2cc2 0x0 fatal error: MSpanList_Insert

runtime stack: runtime.MSpanList_Insert(0x427610, 0x7a0100) /usr/local/Cellar/go/1.4.2/libexec/src/runtime/mheap.c:692 +0x8f runtime.MHeap_Alloc(0x4275a0, 0x1, 0x10000000002, 0xd9e9) /usr/local/Cellar/go/1.4.2/libexec/src/runtime/mheap.c:240 +0x66 runtime.MCentral_CacheSpan(0x42e118, 0x4ba3e8) /usr/local/Cellar/go/1.4.2/libexec/src/runtime/mcentral.c:85 +0x167 runtime.MCache_Refill(0x4b5960, 0xc200000002, 0x4ba3e8) /usr/local/Cellar/go/1.4.2/libexec/src/runtime/mcache.c:90 +0xa0

goroutine 54 [running]: runtime.switchtoM() /usr/local/Cellar/go/1.4.2/libexec/src/runtime/asm_amd64.s:198 fp=0xc209723878 sp=0xc209723870 runtime.mallocgc(0x5, 0x0, 0x3, 0xc20924dff0) /usr/local/Cellar/go/1.4.2/libexec/src/runtime/malloc.go:147 +0x1bb fp=0xc209723928 sp=0xc209723878 runtime.rawstring(0x5, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/Cellar/go/1.4.2/libexec/src/runtime/string.go:195 +0x93 fp=0xc209723958 sp=0xc209723928 runtime.slicebytetostring(0xc208275afc, 0x5, 0x49, 0x0, 0x0) /usr/local/Cellar/go/1.4.2/libexec/src/runtime/string.go:69 +0x4b fp=0xc2097239e8 sp=0xc209723958 github.com/mongodb/mongo-tools/common/json.maybeUnquote(0xc208275afc, 0x5, 0x49, 0x0, 0x0, 0xc208106001) /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/common/json/decode.go:1170 +0x88 fp=0xc209723a28 sp=0xc2097239e8 github.com/mongodb/mongo-tools/common/json.(*decodeState).objectInterface(0xc208625f00, 0xe) /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/common/json/decode.go:1076 +0x14e fp=0xc209723ac0 sp=0xc209723a28 github.com/mongodb/mongo-tools/common/json.(*decodeState).valueInterface(0xc208625f00, 0x0, 0x0) /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/common/json/decode.go:978 +0x84 fp=0xc209723b10 sp=0xc209723ac0 github.com/mongodb/mongo-tools/common/json.(*decodeState).bsonDInterface(0xc208625f00, 0x0, 0x0, 0x0) /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/common/json/decode.go:1044 +0x25a fp=0xc209723bf0 sp=0xc209723b10 github.com/mongodb/mongo-tools/common/json.(*decodeState).bsonDocument(0xc208625f00, 0x0, 0x0, 0x0) /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/common/json/decode.go:358 +0x65 fp=0xc209723c18 sp=0xc209723bf0 github.com/mongodb/mongo-tools/common/json.(*decodeState).unmarshalBsonD(0xc208625f00, 0x0, 0x0, 0x0, 0x0, 0x0) /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/common/json/decode.go:190 +0x151 fp=0xc209723c70 sp=0xc209723c18 github.com/mongodb/mongo-tools/common/json.UnmarshalBsonD(0xc208275900, 0x245, 0x245, 0x0, 0x0, 0x0, 0x0, 0x0) /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/common/json/decode.go:107 +0x173 fp=0xc209723ce0 sp=0xc209723c70 github.com/mongodb/mongo-tools/mongoimport.JSONConverter.Convert(0xc208275900, 0x245, 0x245, 0x604e, 0x0, 0x0, 0x0, 0x0, 0x0) /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/mongoimport/json.go:142 +0x7b fp=0xc209723dd8 sp=0xc209723ce0 github.com/mongodb/mongo-tools/mongoimport.(*JSONConverter).Convert(0xc208ac3b40, 0x0, 0x0, 0x0, 0x0, 0x0) :8 +0xcd fp=0xc209723e28 sp=0xc209723dd8 github.com/mongodb/mongo-tools/mongoimport.(*importWorker).processDocuments(0xc2080ac040, 0xc2080ac000, 0x0, 0x0) /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/mongoimport/common.go:380 +0x166 fp=0xc209723f68 sp=0xc209723e28 github.com/mongodb/mongo-tools/mongoimport.func·002(0xc208086120, 0xc2080c6060, 0xc2080d85a0) /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/mongoimport/common.go:268 +0xa7 fp=0xc209723fc8 sp=0xc209723f68 runtime.goexit() /usr/local/Cellar/go/1.4.2/libexec/src/runtime/asm_amd64.s:2232 +0x1 fp=0xc209723fd0 sp=0xc209723fc8 created by github.com/mongodb/mongo-tools/mongoimport.streamDocuments /private/tmp/mongodb20150321-60760-11xdj8i/mongodb-src-r3.0.1/src/github.com/mongodb/mongo-tools/.gopath/src/github.com/mongodb/mongo-tools/mongoimport/common.go:275 +0x490

Plus some additional output... I've been searching all day and found that this error is not new in Sierra and is related to the Go language. Something about 1.4 version not being supported or not supporting Sierra. I've installed Go 1.7 and tested it, but to no avail. Same error in random places during the mongoimport. Any help would be appreciated!!

Answer 1

I solved similar issue after cleaning up the data and running 'mongoimport' again using new mongo version 3.7.4 on macOS Sierra 10.12.6.

Before solving the issue:

I was using mongo version 3.0.7 (with macOS Sierra 10.12.6). I have observed similar issue today when I tried to import a large JSON file. Here is the sample command and error:

./mongoimport --host localhost --port 27017 --db mydb --collection access --type json  --file /Users/data.json

2017-08-31T08:08:29.376+0530    connected to: localhost:27017
failed MSpanList_Insert 0x1341998 0x2a98efb0db8fa 0x0
fatal error: MSpanList_Insert

runtime stack:
runtime.MSpanList_Insert(0x433940, 0x1341998)
    /usr/local/go/src/runtime/mheap.c:692 +0x8f
runtime.MHeap_Alloc(0x433660, 0x1, 0x10000000009, 0xd9e9)
    /usr/local/go/src/runtime/mheap.c:240 +0x66
runtime.MCentral_CacheSpan(0x43a9b8, 0xc20c2e9c88)
    /usr/local/go/src/runtime/mcentral.c:85 +0x167
runtime.MCache_Refill(0x600960, 0x9, 0x8013417f8)
    /usr/local/go/src/runtime/mcache.c:90 +0xa0

goroutine 20 [running]:
runtime.switchtoM()
    /usr/local/go/src/runtime/asm_amd64.s:198 fp=0xc20c2e9368 sp=0xc20c2e9360
runtime.mallocgc(0x80, 0x0, 0x3, 0xc208531718)
    /usr/local/go/src/runtime/malloc.go:178 +0x849 fp=0xc20c2e9418 sp=0xc20c2e9368
runtime.rawmem(0x80, 0x80)
    /usr/local/go/src/runtime/malloc.go:371 +0x39 fp=0xc20c2e9440 sp=0xc20c2e9418

./mongoimport --version
mongoimport version: 3.0.7
git version: 134c548992e8248c7a7c53777a652cbb2490ab6c

How it got fixed: I removed the mongo data under /data/db/*. It will be some other path as per your configuration.

sudo rm -rf /data/db/*

Then I downloaded & installed new mongo version 3.7.4 from https://www.mongodb.com/dr/fastdl.mongodb.org/osx/mongodb-osx-ssl-x86_64-3.4.7.tgz/download

drwxr-xr-x   8 root  root  272 Aug 31 07:38 mongodb-osx-x86_64-3.0.7
drwxr-xr-x   7 root  root  238 Aug 31 07:39 mongodb-osx-x86_64-3.4.7
lrwxr-xr-x   1 root  root   24 Aug 31 07:39 mongo -> mongodb-osx-x86_64-3.4.7

Check the mongoimport versoin

./mongoimport  --version
mongoimport version: r3.4.7
git version: cf38c1b8a0a8dca4a11737581beafef4fe120bcd
Go version: go1.7
   os: darwin
   arch: amd64
   compiler: gc
OpenSSL version: OpenSSL 0.9.8zh 14 Jan 2016

I was successfully able to import the large file (in GBs) using 'mongoimport' 3.4.7.

Note: if you can't cleanup the data same as I did because of any reason. You can try installing new mongo separately and try 'mongoimport' using new version.

I also tried the 'mongoimport' using older version after cleaning up the data but it didn't work. Using latest 'mongoimport' helped to solve my issue.

Hope it helps.