问题:
因为我要大批量的洗数据(不要问我为啥不用别的数据库,我就要用mongo!).
由于单线程效率太低,改用多线程同时修改数据,发现mongo并发量太低.
于是跟着网上攻略把服务器上的mongo最大链接数和linux服务器的open files
属性修改了,分别是20000,和655360.,如图
代码上面使用了mongo线程池配置,(这里已经尽可能把各属性配置的很变态!)
代码中mongo的读写部分如图:
大概就是读取30000条数据,开启线程池,目前发现最高线程数500,就是这样,再往上加就报错,也就是我的mongo同时写的数量不能超过500,超过就会报这个错:
Exception in thread "pool-5-thread-908" org.springframework.data.mongodb.UncategorizedMongoDbException: Exception opening socket; nested exception is com.mongodb.MongoSocketOpenException: Exception opening socket
at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:138)
at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2774)
at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:540)
at org.springframework.data.mongodb.core.MongoTemplate.doUpdate(MongoTemplate.java:1571)
at org.springframework.data.mongodb.core.MongoTemplate.doUpdate(MongoTemplate.java:1561)
at org.springframework.data.mongodb.core.MongoTemplate.upsert(MongoTemplate.java:1512)
at org.dgdata.modules.address.controller.AdZoneController$1.run(AdZoneController.java:475)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.mongodb.MongoSocketOpenException: Exception opening socket
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:67)
at com.mongodb.internal.connection.InternalStreamConnection.open(InternalStreamConnection.java:126)
at com.mongodb.internal.connection.UsageTrackingInternalConnection.open(UsageTrackingInternalConnection.java:50)
at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.open(DefaultConnectionPool.java:390)
at com.mongodb.internal.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:106)
at com.mongodb.internal.connection.DefaultConnectionPool.get(DefaultConnectionPool.java:92)
at com.mongodb.internal.connection.DefaultServer.getConnection(DefaultServer.java:85)
at com.mongodb.binding.ClusterBinding$ClusterBindingConnectionSource.getConnection(ClusterBinding.java:114)
at com.mongodb.operation.OperationHelper.withReleasableConnection(OperationHelper.java:424)
at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:191)
at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:67)
at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:193)
at com.mongodb.client.internal.MongoCollectionImpl.executeSingleWriteRequest(MongoCollectionImpl.java:960)
at com.mongodb.client.internal.MongoCollectionImpl.executeUpdate(MongoCollectionImpl.java:951)
at com.mongodb.client.internal.MongoCollectionImpl.updateOne(MongoCollectionImpl.java:613)
at org.springframework.data.mongodb.core.MongoTemplate$8.doInCollection(MongoTemplate.java:1618)
at org.springframework.data.mongodb.core.MongoTemplate$8.doInCollection(MongoTemplate.java:1571)
at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:538)
... 7 more
Caused by: java.net.ConnectException: Connection timed out: connect
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at com.mongodb.internal.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:64)
at com.mongodb.internal.connection.SocketStream.open(SocketStream.java:62)
... 24 more
然后就断开,然后到最后实际修改的数量并没有30000
Closed connection [connectionId{localValue:474, serverValue:33363}] to 192.168.11.165:27017 because there was a socket exception raised on another connection from this pool.
有没有大佬碰到类似问题已经解决的,求指教!
