2 sihu168 sihu168 于 2016.09.07 10:44 提问

Storm 0.10.0 storm写es会出现大量的netty连接失败的Error

WARNING: An exception was thrown by TimerTask.
java.lang.RuntimeException: Giving up to scheduleConnect to Netty-Client-wyzjd02/172.18.2.5:6704 after 41 failed attempts. 221 messages were lost
at backtype.storm.messaging.netty.Client$Connect.run(Client.java:511)
at org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546)
at org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446)
at org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395)
at org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)
2016-09-07 10:16:40.260 b.s.m.n.Client [ERROR] connection attempt 2 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:40.461 b.s.m.n.Client [ERROR] connection attempt 3 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:40.661 b.s.m.n.Client [ERROR] connection attempt 4 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:40.860 b.s.m.n.Client [ERROR] connection attempt 5 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:41.061 b.s.m.n.Client [ERROR] connection attempt 6 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:41.361 b.s.m.n.Client [ERROR] connection attempt 7 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:41.660 b.s.m.n.Client [ERROR] connection attempt 8 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:42.060 b.s.m.n.Client [ERROR] connection attempt 9 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:42.463 b.s.m.n.Client [ERROR] connection attempt 10 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700

2016-09-07 10:16:35.559 b.s.m.n.StormClientHandler [INFO] Connection failed Netty-Client-wyzjd02/172.18.2.5:6704
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.7.0_80]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.7.0_80]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.7.0_80]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.7.0_80]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384) ~[?:1.7.0_80]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [storm-core-0.10.0.jar:0.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_80]
2016-09-07 10:16:35.560 b.s.m.n.Client [ERROR] failed to send 1 messages to Netty-Client-wyzjd02/172.18.2.5:6704: java.nio.channels.ClosedChannelException
2016-09-07 10:16:35.561 b.s.m.n.Client [ERROR] failed to send 1 messages to Netty-Client-wyzjd02/172.18.2.5:6704: java.nio.channels.ClosedChannelException
2016-09-07 10:16:35.561 b.s.m.n.Client [ERROR] failed to send 1 messages to Netty-Client-wyzjd02/172.18.2.5:6704: java.nio.channels.ClosedChannelException
2016-09-07 10:16:35.561 b.s.m.n.Client [ERROR] failed to send 1 messages to Netty-Client-wyzjd02/172.18.2.5:6704: java.nio.channels.ClosedChannelException
2016-09-07 10:16:35.561 b.s.m.n.StormClientHandler [INFO] Connection failed Netty-Client-wyzjd02/172.18.2.5:6704
java.nio.channels.ClosedChannelException
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [storm-core-0.10.0.jar:0.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_80]
2016-09-07 10:16:35.660 b.s.m.n.Client [ERROR] connection attempt 33 to Netty-Client-wyzjd02/172.18.2.5:6704 failed: java.net.ConnectException: Connection refused: wyzjd02/172.18.2.5:6704
2016-09-07 10:16:36.160 b.s.m.n.Client [ERROR] connection attempt 34 to Netty-Client-wyzjd02/172.18.2.5:6704 failed: java.net.ConnectException: Connection refused: wyzjd02/172.18.2.5:6704
2016-09-07 10:16:36.661 b.s.m.n.Client [ERROR] connection attempt 35 to Netty-Client-wyzjd02/172.18.2.5:6704 failed: java.net.ConnectException: Connection refused: wyzjd02/172.18.2.5:6704
2016-09-07 10:16:37.160 b.s.m.n.Client [ERROR] connection attempt 36 to Netty-Client-wyzjd02/172.18.2.5:6704 failed: java.net.ConnectException: Connection refused: wyzjd02/172.18.2.5:6704
2016-09-07 10:16:37.661 b.s.m.n.Client [ERROR] connection attempt 37 to Netty-Client-wyzjd02/172.18.2.5:6704 failed: java.net.ConnectException: Connection refused: wyzjd02/172.18.2.5:6704

1个回答

diqigan
diqigan   2016.11.04 17:51

碰到了类似的问题,各种查解决办法,最后发现是在运行storm时有缺失的jar,在storm UI上应该可以看到异常信息。
再引入所有缺失的jar包之后此错误不再发生,storm topology可以正常运行了。

当时日志如下,在最下边可以看到NoClassDefFoundError
2016-11-04 03:17:04.623 o.a.s.m.n.Client [ERROR] connection attempt 35 to Netty-Client-bigdata2/192.168.1.109:6703 failed: java.net.ConnectException: Connection refused: bigdata2/192.168.1.109:6703
2016-11-04 03:17:05.123 o.a.s.m.n.Client [ERROR] connection attempt 36 to Netty-Client-bigdata2/192.168.1.109:6703 failed: java.net.ConnectException: Connection refused: bigdata2/192.168.1.109:6703
2016-11-04 03:17:05.623 o.a.s.m.n.Client [ERROR] connection attempt 37 to Netty-Client-bigdata2/192.168.1.109:6703 failed: java.net.ConnectException: Connection refused: bigdata2/192.168.1.109:6703
2016-11-04 03:17:06.201 o.a.s.d.worker [INFO] All connections are ready for worker edc4215f-1e59-4ece-b002-d4b800857d9f:6702 with id d1f65733-138c-4aa5-84ac-658a6b69dbfa
2016-11-04 03:17:12.276 o.a.s.d.executor [INFO] Preparing bolt acker:(3)
2016-11-04 03:17:12.276 o.a.s.d.executor [INFO] Opening spout kafka-reader:(6)
2016-11-04 03:17:12.285 o.a.s.d.executor [INFO] Preparing bolt word-counter:(9)
2016-11-04 03:17:12.295 o.a.s.d.executor [INFO] Prepared bolt word-counter:(9)
2016-11-04 03:17:12.299 o.a.s.d.executor [INFO] Prepared bolt __acker:(3)
2016-11-04 03:17:12.306 o.a.s.d.executor [INFO] Preparing bolt __system:(-1)
2016-11-04 03:17:12.312 o.a.s.d.executor [INFO] Prepared bolt __system:(-1)
2016-11-04 03:17:12.410 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2016-11-04 03:17:12.426 o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT
2016-11-04 03:17:12.426 o.a.z.ZooKeeper [INFO] Client environment:host.name=bigdata2
2016-11-04 03:17:12.426 o.a.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_79
2016-11-04 03:17:12.426 o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
2016-11-04 03:17:12.427 o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/java/jdk1.7.0_79/jre
2016-11-04 03:17:12.427 o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/tools/apache-storm-1.0.2/lib/log4j-slf4j-impl-2.1.jar:/tools/apache-storm-1.0.2/lib/objenesis-2.1.jar:/tools/apache-storm-1.0.2/lib/servlet-api-2.5.jar:/tools/apache-storm-1.0.2/lib/storm-core-1.0.2.jar:/tools/apache-storm-1.0.2/lib/minlog-1.3.0.jar:/tools/apache-storm-1.0.2/lib/metrics-core-2.2.0.jar:/tools/apache-storm-1.0.2/lib/curator-framework-2.10.0.jar:/tools/apache-storm-1.0.2/lib/log4j-api-2.1.jar:/tools/apache-storm-1.0.2/lib/disruptor-3.3.2.jar:/tools/apache-storm-1.0.2/lib/storm-rename-hack-1.0.2.jar:/tools/apache-storm-1.0.2/lib/log4j-core-2.1.jar:/tools/apache-storm-1.0.2/lib/jopt-simple-4.9.jar:/tools/apache-storm-1.0.2/lib/curator-client-2.10.0.jar:/tools/apache-storm-1.0.2/lib/snappy-java-1.1.2.6.jar:/tools/apache-storm-1.0.2/lib/reflectasm-1.10.1.jar:/tools/apache-storm-1.0.2/lib/guava-16.0.1.jar:/tools/apache-storm-1.0.2/lib/zkclient-0.9.jar:/tools/apache-storm-1.0.2/lib/scala-library-2.11.8.jar:/tools/apache-storm-1.0.2/lib/storm-kafka-1.0.1.jar:/tools/apache-storm-1.0.2/lib/clojure-1.7.0.jar:/tools/apache-storm-1.0.2/lib/zookeeper-3.4.8.jar:/tools/apache-storm-1.0.2/lib/kafka-clients-0.9.0.1.jar:/tools/apache-storm-1.0.2/lib/asm-5.0.3.jar:/tools/apache-storm-1.0.2/lib/log4j-over-slf4j-1.6.6.jar:/tools/apache-storm-1.0.2/lib/kafka_2.11-0.10.1.0.jar:/tools/apache-storm-1.0.2/lib/kryo-3.0.3.jar:/tools/apache-storm-1.0.2/lib/slf4j-api-1.7.7.jar:/tools/apache-storm-1.0.2/conf:/tmp/storm/workdir/supervisor/stormdist/MyKafkaTopology-5-1478243613/stormjar.jar
2016-11-04 03:17:12.427 o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/tmp/storm/workdir/supervisor/stormdist/MyKafkaTopology-5-1478243613/resources/Linux-amd64:/tmp/storm/workdir/supervisor/stormdist/MyKafkaTopology-5-1478243613/resources:/usr/local/lib:/opt/local/lib:/usr/lib
2016-11-04 03:17:12.427 o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp/storm/workdir/workers/d1f65733-138c-4aa5-84ac-658a6b69dbfa/tmp
2016-11-04 03:17:12.427 o.a.z.ZooKeeper [INFO] Client environment:java.compiler=
2016-11-04 03:17:12.428 o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux
2016-11-04 03:17:12.428 o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64
2016-11-04 03:17:12.428 o.a.z.ZooKeeper [INFO] Client environment:os.version=2.6.32-573.el6.x86_64
2016-11-04 03:17:12.428 o.a.z.ZooKeeper [INFO] Client environment:user.name=root
2016-11-04 03:17:12.428 o.a.z.ZooKeeper [INFO] Client environment:user.home=/root
2016-11-04 03:17:12.429 o.a.z.ZooKeeper [INFO] Client environment:user.dir=/tmp/storm/workdir/workers/d1f65733-138c-4aa5-84ac-658a6b69dbfa
2016-11-04 03:17:12.430 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=192.168.1.108:2181,192.168.1.109:2181,192.168.1.110:2181, sessionTimeout=20000 watcher=org.apache.curator.ConnectionState@728d77f0
2016-11-04 03:17:12.455 o.a.z.ClientCnxn [INFO] Opening socket connection to server 192.168.1.108/192.168.1.108:2181. Will not attempt to authenticate using SASL (unknown error)
2016-11-04 03:17:12.459 o.a.z.ClientCnxn [INFO] Socket connection established to 192.168.1.108/192.168.1.108:2181, initiating session
2016-11-04 03:17:12.468 o.a.z.ClientCnxn [INFO] Session establishment complete on server 192.168.1.108/192.168.1.108:2181, sessionid = 0x1581e7c20f301ac, negotiated timeout = 20000
2016-11-04 03:17:12.472 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
2016-11-04 03:17:12.477 o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2016-11-04 03:17:12.477 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=192.168.1.108:2181,192.168.1.109:2181,192.168.1.110:2181 sessionTimeout=20000 watcher=org.apache.curator.ConnectionState@708b553c
2016-11-04 03:17:12.479 o.a.z.ClientCnxn [INFO] Opening socket connection to server 192.168.1.108/192.168.1.108:2181. Will not attempt to authenticate using SASL (unknown error)
2016-11-04 03:17:12.480 o.a.z.ClientCnxn [INFO] Socket connection established to 192.168.1.108/192.168.1.108:2181, initiating session
2016-11-04 03:17:12.492 o.a.z.ClientCnxn [INFO] Session establishment complete on server 192.168.1.108/192.168.1.108:2181, sessionid = 0x1581e7c20f301ad, negotiated timeout = 20000
2016-11-04 03:17:12.497 o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
2016-11-04 03:17:12.527 o.a.s.util [ERROR] Async loop died!
java.lang.NoClassDefFoundError: org/json/simple/JSONValue
at org.apache.storm.kafka.DynamicBrokersReader.getLeaderFor(DynamicBrokersReader.java:163) ~[storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:87) ~[storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.trident.ZkBrokerReader.(ZkBrokerReader.java:44) ~[storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:58) ~[storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.kafka.KafkaSpout.open(KafkaSpout.java:77) ~[storm-kafka-1.0.1.jar:1.0.1]
at org.apache.storm.daemon.executor$fn
7990$fn__8005.invoke(executor.clj:604) ~[storm-core-1.0.2.jar:1.0.2]
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.0.2.jar:1.0.2]

Csdn user default icon
上传中...
上传图片
插入图片