We have been using a filesystem build in Golang, seaweedfs, and our own disk's file system was XFS,
a few days back we got an error as
[ 42.236828] XFS (sda5): xfs_do_force_shutdown(0x8) called from line 118 of file /build/linux-lts-wily-JUyKGw/linux-lts-wily-4.2.0/fs/xfs/xfs_bmap_util.c. Return address = 0xffffffffc02e35e4
[ 42.237271] XFS (sda5): Corruption of in-memory data detected. Shutting down filesystem
[ 42.237320] XFS (sda5): Please umount the filesystem and rectify the problem(s)
so, we mounted and unmounted the disk and then It needs cleaning, so we repaired it with xfs_repair -L /dev/sda5
there was nothing in lost+found folder but after that, our seaweed master server started to crash, As We asked a question to Chris Lu, who made the seaweedFS, he mentioned the files are corrupt, in filer, actually leveldb files. the error was something like this
I1101 12:42:19 31159 volume.go:110] loading index file /storage/1082.idx readonly false
F1101 12:42:19 31159 filer_server.go:53] Can not start filer in dir /storage/filer : leveldb/storage: corrupted or incomplete meta file
goroutine 21 [running]:
github.com/chrislusf/seaweedfs/weed/glog.stacks(0x1214600, 0x0, 0x0, 0x0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:767 +0xb8
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).output(0x11f59e0, 0xc800000003, 0xc8200faf00, 0x11cb17f, 0xf, 0x35, 0x0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:718 +0x259
github.com/chrislusf/seaweedfs/weed/glog.(*loggingT).printf(0x11f59e0, 0xc800000003, 0xe57d80, 0x22, 0xc82004dd10, 0x2, 0x2)
/root/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:656 +0x1d4
github.com/chrislusf/seaweedfs/weed/glog.Fatalf(0xe57d80, 0x22, 0xc82004dd10, 0x2, 0x2)
/root/go/src/github.com/chrislusf/seaweedfs/weed/glog/glog.go:1149 +0x5d
github.com/chrislusf/seaweedfs/weed/server.NewFilerServer(0xc8200659e0, 0xd8b9b0, 0x7, 0x22b8, 0xc8200eb740, 0x13, 0xc82018d280, 0xe, 0x0, 0x0, ...)
after that, we used a tool to repair leveldb files, https://github.com/rchunping/leveldb-tools, It repaired the files without any error, and seaweedfs Master started with filer, with no errors like the mentioned above, But as when everything is working fine, filer should respond on port 8888, but when we access port 8888, it gave error like this,
017/11/02 01:50:41 http: panic serving 39.36.53.157:46124: leveldb: internal key "\x00\x01d,", len=4: invalid length
goroutine 2498 [running]:
net/http.(*conn).serve.func1(0xcd3aaf8080)
/usr/lib/go/src/net/http/server.go:1389 +0xc1
panic(0xaf99a0, 0xcfefca8210)
/usr/lib/go/src/runtime/panic.go:426 +0x4e9
github.com/syndtr/goleveldb/leveldb.internalKey.assert(0xc8712930ed, 0x4, 0x20)
/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:115 +0x22c
github.com/syndtr/goleveldb/leveldb.internalKey.ukey(0xc8712930ed, 0x4, 0x20, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:120 +0x46
github.com/syndtr/goleveldb/leveldb.(*iComparer).Compare(0xc852e100a0, 0xc8712930ed, 0x4, 0x20, 0xcfefca81d0, 0xc, 0xc, 0xc850f5cf60)
/root/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:36 +0x4a
github.com/syndtr/goleveldb/leveldb/table.(*block).seek.func1(0x1, 0x8cce81)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:68 +0x278
sort.Search(0x3, 0xcfefc44bd0, 0xe8)
/usr/lib/go/src/sort/search.go:66 +0x52
github.com/syndtr/goleveldb/leveldb/table.(*block).seek(0xc87231c1c0, 0x7f8204327100, 0xc852e100a0, 0x0, 0x3, 0xcfefca81d0, 0xc, 0xc, 0xcdcd40, 0x443815, ...)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:69 +0xa9
github.com/syndtr/goleveldb/leveldb/table.(*blockIter).Seek(0xcfefb9e5a0, 0xcfefca81d0, 0xc, 0xc, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:229 +0x10e
github.com/syndtr/goleveldb/leveldb/table.(*Reader).newBlockIter(0xc8387461a0, 0xc87231c1c0, 0x7f81f57e7168, 0xcb79534140, 0xc87026d3b0, 0x1, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:736 +0x1db
github.com/syndtr/goleveldb/leveldb/table.(*Reader).NewIterator(0xc8387461a0, 0xc87026d3b0, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:804 +0x356
github.com/syndtr/goleveldb/leveldb.(*tOps).newIterator(0xc86b6ad350, 0xc85214bf90, 0xc87026d3b0, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table.go:411 +0x1a5
github.com/syndtr/goleveldb/leveldb.(*version).getIterators(0xc844da5cc0, 0xc87026d3b0, 0x0, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/version.go:253 +0x1a7
github.com/syndtr/goleveldb/leveldb.(*DB).newRawIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0xc87026d3b0, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:41 +0xdc
github.com/syndtr/goleveldb/leveldb.(*DB).newIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0x16571aa28, 0xcfefc45550, 0x0, 0x4ee651)
/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:79 +0x229
github.com/syndtr/goleveldb/leveldb.(*DB).NewIterator(0xc8201f4780, 0xcfefc45550, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/db.go:882 +0x205
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FileListInLevelDb).ListFiles(0xc82009c138, 0xcf00000000, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/files_in_leveldb.go:64 +0x48c
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FilerEmbedded).ListFiles(0xc849337300, 0xcfefca8144, 0x1, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, 0x0, ...)
/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/filer_embedded.go:63 +0xf1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).listDirectoryHandler(0xc8201f6000, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:40 +0x6b1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).GetOrHeadHandler(0xc8201f6000, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380, 0xffffffffffffff01)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:50 +0x150
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc8201f6000, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:10 +0xf6
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).(github.com/chrislusf/seaweedfs/weed/server.filerHandler)-fm(0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server.go:60 +0x3e
net/http.HandlerFunc.ServeHTTP(0xc82ddf86d0, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
/usr/lib/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc82017eb40, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
/usr/lib/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc852de9f00, 0x7f81e2155190, 0xcbbe8221a0, 0xc820156380)
/usr/lib/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xcd3aaf8080)
/usr/lib/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
/usr/lib/go/src/net/http/server.go:2137 +0x44e
2017/11/02 01:51:12 http: panic serving 39.36.53.157:46128: leveldb: internal key "\x00\x01d,", len=4: invalid length
goroutine 2467 [running]:
net/http.(*conn).serve.func1(0xc921cf6780)
/usr/lib/go/src/net/http/server.go:1389 +0xc1
panic(0xaf99a0, 0xcf62fea290)
/usr/lib/go/src/runtime/panic.go:426 +0x4e9
github.com/syndtr/goleveldb/leveldb.internalKey.assert(0xc8712930ed, 0x4, 0x20)
/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:115 +0x22c
github.com/syndtr/goleveldb/leveldb.internalKey.ukey(0xc8712930ed, 0x4, 0x20, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:120 +0x46
github.com/syndtr/goleveldb/leveldb.(*iComparer).Compare(0xc852e100a0, 0xc8712930ed, 0x4, 0x20, 0xcf62fea250, 0xc, 0xc, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:36 +0x4a
github.com/syndtr/goleveldb/leveldb/table.(*block).seek.func1(0x1, 0x8cce81)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:68 +0x278
sort.Search(0x3, 0xca5cc78bd0, 0xe8)
/usr/lib/go/src/sort/search.go:66 +0x52
github.com/syndtr/goleveldb/leveldb/table.(*block).seek(0xc87231c1c0, 0x7f8204327100, 0xc852e100a0, 0x0, 0x3, 0xcf62fea250, 0xc, 0xc, 0xcdcd40, 0x443815, ...)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:69 +0xa9
github.com/syndtr/goleveldb/leveldb/table.(*blockIter).Seek(0xce24069b30, 0xcf62fea250, 0xc, 0xc, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:229 +0x10e
github.com/syndtr/goleveldb/leveldb/table.(*Reader).newBlockIter(0xc8387461a0, 0xc87231c1c0, 0x7f81f57e7168, 0xcb795cc1f8, 0xc87024edb0, 0x1, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:736 +0x1db
github.com/syndtr/goleveldb/leveldb/table.(*Reader).NewIterator(0xc8387461a0, 0xc87024edb0, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:804 +0x356
github.com/syndtr/goleveldb/leveldb.(*tOps).newIterator(0xc86b6ad350, 0xc85214bf90, 0xc87024edb0, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table.go:411 +0x1a5
github.com/syndtr/goleveldb/leveldb.(*version).getIterators(0xc844da5cc0, 0xc87024edb0, 0x0, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/version.go:253 +0x1a7
github.com/syndtr/goleveldb/leveldb.(*DB).newRawIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0xc87024edb0, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:41 +0xdc
github.com/syndtr/goleveldb/leveldb.(*DB).newIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0x16571aa28, 0xca5cc79550, 0x0, 0x4ee651)
/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:79 +0x229
github.com/syndtr/goleveldb/leveldb.(*DB).NewIterator(0xc8201f4780, 0xca5cc79550, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/db.go:882 +0x205
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FileListInLevelDb).ListFiles(0xc82009c138, 0xcf00000000, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/files_in_leveldb.go:64 +0x48c
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FilerEmbedded).ListFiles(0xc849337300, 0xcf62fea1c4, 0x1, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, 0x0, ...)
/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/filer_embedded.go:63 +0xf1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).listDirectoryHandler(0xc8201f6000, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:40 +0x6b1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).GetOrHeadHandler(0xc8201f6000, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0, 0xffffffffffffff01)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:50 +0x150
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc8201f6000, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:10 +0xf6
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).(github.com/chrislusf/seaweedfs/weed/server.filerHandler)-fm(0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server.go:60 +0x3e
net/http.HandlerFunc.ServeHTTP(0xc82ddf86d0, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
/usr/lib/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc82017eb40, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
/usr/lib/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc852de9f00, 0x7f81e2155190, 0xcdf1bf21a0, 0xc82018c8c0)
/usr/lib/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc921cf6780)
/usr/lib/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
/usr/lib/go/src/net/http/server.go:2137 +0x44e
2017/11/02 01:52:23 http: panic serving 39.36.53.157:46142: leveldb: internal key "\x00\x01d,", len=4: invalid length
goroutine 2456 [running]:
net/http.(*conn).serve.func1(0xc97d935b80)
/usr/lib/go/src/net/http/server.go:1389 +0xc1
panic(0xaf99a0, 0xce2f857050)
/usr/lib/go/src/runtime/panic.go:426 +0x4e9
github.com/syndtr/goleveldb/leveldb.internalKey.assert(0xc8712930ed, 0x4, 0x20)
/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:115 +0x22c
github.com/syndtr/goleveldb/leveldb.internalKey.ukey(0xc8712930ed, 0x4, 0x20, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/key.go:120 +0x46
github.com/syndtr/goleveldb/leveldb.(*iComparer).Compare(0xc852e100a0, 0xc8712930ed, 0x4, 0x20, 0xce2f857010, 0xc, 0xc, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:36 +0x4a
github.com/syndtr/goleveldb/leveldb/table.(*block).seek.func1(0x1, 0x8cce81)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:68 +0x278
sort.Search(0x3, 0xca5cc78bd0, 0xe8)
/usr/lib/go/src/sort/search.go:66 +0x52
github.com/syndtr/goleveldb/leveldb/table.(*block).seek(0xc87231c1c0, 0x7f8204327100, 0xc852e100a0, 0x0, 0x3, 0xce2f857010, 0xc, 0xc, 0xcdcd40, 0x443815, ...)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:69 +0xa9
github.com/syndtr/goleveldb/leveldb/table.(*blockIter).Seek(0xce24069ef0, 0xce2f857010, 0xc, 0xc, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:229 +0x10e
github.com/syndtr/goleveldb/leveldb/table.(*Reader).newBlockIter(0xc8387461a0, 0xc87231c1c0, 0x7f81f57e7168, 0xcb795cc478, 0xc8701be090, 0x1, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:736 +0x1db
github.com/syndtr/goleveldb/leveldb/table.(*Reader).NewIterator(0xc8387461a0, 0xc8701be090, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:804 +0x356
github.com/syndtr/goleveldb/leveldb.(*tOps).newIterator(0xc86b6ad350, 0xc85214bf90, 0xc8701be090, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/table.go:411 +0x1a5
github.com/syndtr/goleveldb/leveldb.(*version).getIterators(0xc844da5cc0, 0xc8701be090, 0x0, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/version.go:253 +0x1a7
github.com/syndtr/goleveldb/leveldb.(*DB).newRawIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0xc8701be090, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:41 +0xdc
github.com/syndtr/goleveldb/leveldb.(*DB).newIterator(0xc8201f4780, 0x0, 0x0, 0x0, 0x0, 0x16571aa28, 0xca5cc79550, 0x0, 0x4ee651)
/root/go/src/github.com/syndtr/goleveldb/leveldb/db_iter.go:79 +0x229
github.com/syndtr/goleveldb/leveldb.(*DB).NewIterator(0xc8201f4780, 0xca5cc79550, 0x0, 0x0, 0x0)
/root/go/src/github.com/syndtr/goleveldb/leveldb/db.go:882 +0x205
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FileListInLevelDb).ListFiles(0xc82009c138, 0xce00000000, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/files_in_leveldb.go:64 +0x48c
github.com/chrislusf/seaweedfs/weed/filer/embedded_filer.(*FilerEmbedded).ListFiles(0xc849337300, 0xce2f856f64, 0x1, 0x0, 0x0, 0x64, 0x0, 0x0, 0x0, 0x0, ...)
/root/go/src/github.com/chrislusf/seaweedfs/weed/filer/embedded_filer/filer_embedded.go:63 +0xf1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).listDirectoryHandler(0xc8201f6000, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:40 +0x6b1
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).GetOrHeadHandler(0xc8201f6000, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0, 0xffffffffffffff01)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers_read.go:50 +0x150
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).filerHandler(0xc8201f6000, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server_handlers.go:10 +0xf6
github.com/chrislusf/seaweedfs/weed/server.(*FilerServer).(github.com/chrislusf/seaweedfs/weed/server.filerHandler)-fm(0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
/root/go/src/github.com/chrislusf/seaweedfs/weed/server/filer_server.go:60 +0x3e
net/http.HandlerFunc.ServeHTTP(0xc82ddf86d0, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
/usr/lib/go/src/net/http/server.go:1618 +0x3a
net/http.(*ServeMux).ServeHTTP(0xc82017eb40, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
/usr/lib/go/src/net/http/server.go:1910 +0x17d
net/http.serverHandler.ServeHTTP(0xc852de9f00, 0x7f81e2155190, 0xcc30caf520, 0xcfef0d87e0)
/usr/lib/go/src/net/http/server.go:2081 +0x19e
net/http.(*conn).serve(0xc97d935b80)
/usr/lib/go/src/net/http/server.go:1472 +0xf2e
created by net/http.(*Server).Serve
/usr/lib/go/src/net/http/server.go:2137 +0x44
So Chris says , the leveldb file seems corrupted.
My question is:
Is there any way to repair those corrupted files, As I have used the above tool, but I made little bit difference only, are there any good tools which can repair those corrupted files, or we are shooting in the dark and it not even possible.
I don't have any knowledge about Golang or leveldb, So pardon me if the question seems silly to all of you. Purpose of posting here is: If you can see the error and suggest about, how to recover that. :)
