sihu168 2016-09-07 02:44 采纳率: 0%
浏览 7563

Storm 0.10.0 storm写es会出现大量的netty连接失败的Error

WARNING: An exception was thrown by TimerTask.
java.lang.RuntimeException: Giving up to scheduleConnect to Netty-Client-wyzjd02/172.18.2.5:6704 after 41 failed attempts. 221 messages were lost
at backtype.storm.messaging.netty.Client$Connect.run(Client.java:511)
at org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546)
at org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446)
at org.apache.storm.shade.org.jboss.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395)
at org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at java.lang.Thread.run(Thread.java:745)
2016-09-07 10:16:40.260 b.s.m.n.Client [ERROR] connection attempt 2 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:40.461 b.s.m.n.Client [ERROR] connection attempt 3 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:40.661 b.s.m.n.Client [ERROR] connection attempt 4 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:40.860 b.s.m.n.Client [ERROR] connection attempt 5 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:41.061 b.s.m.n.Client [ERROR] connection attempt 6 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:41.361 b.s.m.n.Client [ERROR] connection attempt 7 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:41.660 b.s.m.n.Client [ERROR] connection attempt 8 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:42.060 b.s.m.n.Client [ERROR] connection attempt 9 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700
2016-09-07 10:16:42.463 b.s.m.n.Client [ERROR] connection attempt 10 to Netty-Client-wyfx03/172.18.2.3:6700 failed: java.net.ConnectException: Connection refused: wyfx03/172.18.2.3:6700

2016-09-07 10:16:35.559 b.s.m.n.StormClientHandler [INFO] Connection failed Netty-Client-wyzjd02/172.18.2.5:6704
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:1.7.0_80]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:1.7.0_80]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[?:1.7.0_80]
at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[?:1.7.0_80]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384) ~[?:1.7.0_80]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:64) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [storm-core-0.10.0.jar:0.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_80]
2016-09-07 10:16:35.560 b.s.m.n.Client [ERROR] failed to send 1 messages to Netty-Client-wyzjd02/172.18.2.5:6704: java.nio.channels.ClosedChannelException
2016-09-07 10:16:35.561 b.s.m.n.Client [ERROR] failed to send 1 messages to Netty-Client-wyzjd02/172.18.2.5:6704: java.nio.channels.ClosedChannelException
2016-09-07 10:16:35.561 b.s.m.n.Client [ERROR] failed to send 1 messages to Netty-Client-wyzjd02/172.18.2.5:6704: java.nio.channels.ClosedChannelException
2016-09-07 10:16:35.561 b.s.m.n.Client [ERROR] failed to send 1 messages to Netty-Client-wyzjd02/172.18.2.5:6704: java.nio.channels.ClosedChannelException
2016-09-07 10:16:35.561 b.s.m.n.StormClientHandler [INFO] Connection failed Netty-Client-wyzjd02/172.18.2.5:6704
java.nio.channels.ClosedChannelException
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.close(AbstractNioWorker.java:373) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:93) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) [storm-core-0.10.0.jar:0.10.0]
at org.apache.storm.shade.org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) [storm-core-0.10.0.jar:0.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [?:1.7.0_80]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [?:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_80]
2016-09-07 10:16:35.660 b.s.m.n.Client [ERROR] connection attempt 33 to Netty-Client-wyzjd02/172.18.2.5:6704 failed: java.net.ConnectException: Connection refused: wyzjd02/172.18.2.5:6704
2016-09-07 10:16:36.160 b.s.m.n.Client [ERROR] connection attempt 34 to Netty-Client-wyzjd02/172.18.2.5:6704 failed: java.net.ConnectException: Connection refused: wyzjd02/172.18.2.5:6704
2016-09-07 10:16:36.661 b.s.m.n.Client [ERROR] connection attempt 35 to Netty-Client-wyzjd02/172.18.2.5:6704 failed: java.net.ConnectException: Connection refused: wyzjd02/172.18.2.5:6704
2016-09-07 10:16:37.160 b.s.m.n.Client [ERROR] connection attempt 36 to Netty-Client-wyzjd02/172.18.2.5:6704 failed: java.net.ConnectException: Connection refused: wyzjd02/172.18.2.5:6704
2016-09-07 10:16:37.661 b.s.m.n.Client [ERROR] connection attempt 37 to Netty-Client-wyzjd02/172.18.2.5:6704 failed: java.net.ConnectException: Connection refused: wyzjd02/172.18.2.5:6704

  • 写回答

1条回答 默认 最新

  • 本人已死 2016-11-04 09:51
    关注

    碰到了类似的问题,各种查解决办法,最后发现是在运行storm时有缺失的jar,在storm UI上应该可以看到异常信息。
    再引入所有缺失的jar包之后此错误不再发生,storm topology可以正常运行了。

    当时日志如下,在最下边可以看到NoClassDefFoundError
    2016-11-04 03:17:04.623 o.a.s.m.n.Client [ERROR] connection attempt 35 to Netty-Client-bigdata2/192.168.1.109:6703 failed: java.net.ConnectException: Connection refused: bigdata2/192.168.1.109:6703
    2016-11-04 03:17:05.123 o.a.s.m.n.Client [ERROR] connection attempt 36 to Netty-Client-bigdata2/192.168.1.109:6703 failed: java.net.ConnectException: Connection refused: bigdata2/192.168.1.109:6703
    2016-11-04 03:17:05.623 o.a.s.m.n.Client [ERROR] connection attempt 37 to Netty-Client-bigdata2/192.168.1.109:6703 failed: java.net.ConnectException: Connection refused: bigdata2/192.168.1.109:6703
    2016-11-04 03:17:06.201 o.a.s.d.worker [INFO] All connections are ready for worker edc4215f-1e59-4ece-b002-d4b800857d9f:6702 with id d1f65733-138c-4aa5-84ac-658a6b69dbfa
    2016-11-04 03:17:12.276 o.a.s.d.executor [INFO] Preparing bolt acker:(3)
    2016-11-04 03:17:12.276 o.a.s.d.executor [INFO] Opening spout kafka-reader:(6)
    2016-11-04 03:17:12.285 o.a.s.d.executor [INFO] Preparing bolt word-counter:(9)
    2016-11-04 03:17:12.295 o.a.s.d.executor [INFO] Prepared bolt word-counter:(9)
    2016-11-04 03:17:12.299 o.a.s.d.executor [INFO] Prepared bolt __acker:(3)
    2016-11-04 03:17:12.306 o.a.s.d.executor [INFO] Preparing bolt __system:(-1)
    2016-11-04 03:17:12.312 o.a.s.d.executor [INFO] Prepared bolt __system:(-1)
    2016-11-04 03:17:12.410 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
    2016-11-04 03:17:12.426 o.a.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT
    2016-11-04 03:17:12.426 o.a.z.ZooKeeper [INFO] Client environment:host.name=bigdata2
    2016-11-04 03:17:12.426 o.a.z.ZooKeeper [INFO] Client environment:java.version=1.7.0_79
    2016-11-04 03:17:12.426 o.a.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation
    2016-11-04 03:17:12.427 o.a.z.ZooKeeper [INFO] Client environment:java.home=/usr/java/jdk1.7.0_79/jre
    2016-11-04 03:17:12.427 o.a.z.ZooKeeper [INFO] Client environment:java.class.path=/tools/apache-storm-1.0.2/lib/log4j-slf4j-impl-2.1.jar:/tools/apache-storm-1.0.2/lib/objenesis-2.1.jar:/tools/apache-storm-1.0.2/lib/servlet-api-2.5.jar:/tools/apache-storm-1.0.2/lib/storm-core-1.0.2.jar:/tools/apache-storm-1.0.2/lib/minlog-1.3.0.jar:/tools/apache-storm-1.0.2/lib/metrics-core-2.2.0.jar:/tools/apache-storm-1.0.2/lib/curator-framework-2.10.0.jar:/tools/apache-storm-1.0.2/lib/log4j-api-2.1.jar:/tools/apache-storm-1.0.2/lib/disruptor-3.3.2.jar:/tools/apache-storm-1.0.2/lib/storm-rename-hack-1.0.2.jar:/tools/apache-storm-1.0.2/lib/log4j-core-2.1.jar:/tools/apache-storm-1.0.2/lib/jopt-simple-4.9.jar:/tools/apache-storm-1.0.2/lib/curator-client-2.10.0.jar:/tools/apache-storm-1.0.2/lib/snappy-java-1.1.2.6.jar:/tools/apache-storm-1.0.2/lib/reflectasm-1.10.1.jar:/tools/apache-storm-1.0.2/lib/guava-16.0.1.jar:/tools/apache-storm-1.0.2/lib/zkclient-0.9.jar:/tools/apache-storm-1.0.2/lib/scala-library-2.11.8.jar:/tools/apache-storm-1.0.2/lib/storm-kafka-1.0.1.jar:/tools/apache-storm-1.0.2/lib/clojure-1.7.0.jar:/tools/apache-storm-1.0.2/lib/zookeeper-3.4.8.jar:/tools/apache-storm-1.0.2/lib/kafka-clients-0.9.0.1.jar:/tools/apache-storm-1.0.2/lib/asm-5.0.3.jar:/tools/apache-storm-1.0.2/lib/log4j-over-slf4j-1.6.6.jar:/tools/apache-storm-1.0.2/lib/kafka_2.11-0.10.1.0.jar:/tools/apache-storm-1.0.2/lib/kryo-3.0.3.jar:/tools/apache-storm-1.0.2/lib/slf4j-api-1.7.7.jar:/tools/apache-storm-1.0.2/conf:/tmp/storm/workdir/supervisor/stormdist/MyKafkaTopology-5-1478243613/stormjar.jar
    2016-11-04 03:17:12.427 o.a.z.ZooKeeper [INFO] Client environment:java.library.path=/tmp/storm/workdir/supervisor/stormdist/MyKafkaTopology-5-1478243613/resources/Linux-amd64:/tmp/storm/workdir/supervisor/stormdist/MyKafkaTopology-5-1478243613/resources:/usr/local/lib:/opt/local/lib:/usr/lib
    2016-11-04 03:17:12.427 o.a.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp/storm/workdir/workers/d1f65733-138c-4aa5-84ac-658a6b69dbfa/tmp
    2016-11-04 03:17:12.427 o.a.z.ZooKeeper [INFO] Client environment:java.compiler=
    2016-11-04 03:17:12.428 o.a.z.ZooKeeper [INFO] Client environment:os.name=Linux
    2016-11-04 03:17:12.428 o.a.z.ZooKeeper [INFO] Client environment:os.arch=amd64
    2016-11-04 03:17:12.428 o.a.z.ZooKeeper [INFO] Client environment:os.version=2.6.32-573.el6.x86_64
    2016-11-04 03:17:12.428 o.a.z.ZooKeeper [INFO] Client environment:user.name=root
    2016-11-04 03:17:12.428 o.a.z.ZooKeeper [INFO] Client environment:user.home=/root
    2016-11-04 03:17:12.429 o.a.z.ZooKeeper [INFO] Client environment:user.dir=/tmp/storm/workdir/workers/d1f65733-138c-4aa5-84ac-658a6b69dbfa
    2016-11-04 03:17:12.430 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=192.168.1.108:2181,192.168.1.109:2181,192.168.1.110:2181, sessionTimeout=20000 watcher=org.apache.curator.ConnectionState@728d77f0
    2016-11-04 03:17:12.455 o.a.z.ClientCnxn [INFO] Opening socket connection to server 192.168.1.108/192.168.1.108:2181. Will not attempt to authenticate using SASL (unknown error)
    2016-11-04 03:17:12.459 o.a.z.ClientCnxn [INFO] Socket connection established to 192.168.1.108/192.168.1.108:2181, initiating session
    2016-11-04 03:17:12.468 o.a.z.ClientCnxn [INFO] Session establishment complete on server 192.168.1.108/192.168.1.108:2181, sessionid = 0x1581e7c20f301ac, negotiated timeout = 20000
    2016-11-04 03:17:12.472 o.a.c.f.i.CuratorFrameworkImpl [INFO] Starting
    2016-11-04 03:17:12.477 o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
    2016-11-04 03:17:12.477 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=192.168.1.108:2181,192.168.1.109:2181,192.168.1.110:2181 sessionTimeout=20000 watcher=org.apache.curator.ConnectionState@708b553c
    2016-11-04 03:17:12.479 o.a.z.ClientCnxn [INFO] Opening socket connection to server 192.168.1.108/192.168.1.108:2181. Will not attempt to authenticate using SASL (unknown error)
    2016-11-04 03:17:12.480 o.a.z.ClientCnxn [INFO] Socket connection established to 192.168.1.108/192.168.1.108:2181, initiating session
    2016-11-04 03:17:12.492 o.a.z.ClientCnxn [INFO] Session establishment complete on server 192.168.1.108/192.168.1.108:2181, sessionid = 0x1581e7c20f301ad, negotiated timeout = 20000
    2016-11-04 03:17:12.497 o.a.c.f.s.ConnectionStateManager [INFO] State change: CONNECTED
    2016-11-04 03:17:12.527 o.a.s.util [ERROR] Async loop died!
    java.lang.NoClassDefFoundError: org/json/simple/JSONValue
    at org.apache.storm.kafka.DynamicBrokersReader.getLeaderFor(DynamicBrokersReader.java:163) ~[storm-kafka-1.0.1.jar:1.0.1]
    at org.apache.storm.kafka.DynamicBrokersReader.getBrokerInfo(DynamicBrokersReader.java:87) ~[storm-kafka-1.0.1.jar:1.0.1]
    at org.apache.storm.kafka.trident.ZkBrokerReader.(ZkBrokerReader.java:44) ~[storm-kafka-1.0.1.jar:1.0.1]
    at org.apache.storm.kafka.KafkaUtils.makeBrokerReader(KafkaUtils.java:58) ~[storm-kafka-1.0.1.jar:1.0.1]
    at org.apache.storm.kafka.KafkaSpout.open(KafkaSpout.java:77) ~[storm-kafka-1.0.1.jar:1.0.1]
    at org.apache.storm.daemon.executor$fn
    7990$fn__8005.invoke(executor.clj:604) ~[storm-core-1.0.2.jar:1.0.2]
    at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) [storm-core-1.0.2.jar:1.0.2]

    评论

报告相同问题?

悬赏问题

  • ¥20 iqoo11 如何下载安装工程模式
  • ¥15 本题的答案是不是有问题
  • ¥15 关于#r语言#的问题:(svydesign)为什么在一个大的数据集中抽取了一个小数据集
  • ¥15 C++使用Gunplot
  • ¥15 这个电路是如何实现路灯控制器的,原理是什么,怎么求解灯亮起后熄灭的时间如图?
  • ¥15 matlab数字图像处理频率域滤波
  • ¥15 在abaqus做了二维正交切削模型,给刀具添加了超声振动条件后输出切削力为什么比普通切削增大这么多
  • ¥15 ELGamal和paillier计算效率谁快?
  • ¥15 蓝桥杯单片机第十三届第一场,整点继电器吸合,5s后断开出现了问题
  • ¥15 file converter 转换格式失败 报错 Error marking filters as finished,如何解决?