问题遇到的现象和发生背景
背景:在vm虚拟的三台主机上运行storm(1.2.4)、kafka(4.0.0)、zookeeper(3.8.4)
问题:启动storm的nimbus、supervisor时,进程会运行一会后会自动停止,甚至有时候会把kafka的进程也弄挂掉
其中配置文件如下:
zookeeper配置:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/module/apache-zookeeper-3.8.4-bin/dataDir
clientPort=2181
server.1=hadoop102:2888:3888
server.2=hadoop103:2888:3888
server.3=hadoop104:2888:3888
storm配置:
storm.zookeeper.servers:
- "hadoop102"
- "hadoop103"
- "hadoop104"
nimbus.seeds: ["hadoop102", "hadoop103"]
storm.local.dir: "/opt/module/apache-storm-1.2.4/dataDir"
supervisor.slots.ports:
- 6700
- 6701
- 6702
- 6703
ui.port: 8081
kafka部分配置:
############################# Server Basics #############################
process.roles=broker,controller
node.id=4
controller.quorum.bootstrap.servers=localhost:9093
controller.quorum.voters=4@hadhoop102:9093,2@hadoop103:9093,3@hadoop104:9093
############################# Socket Server Settings #############################
listeners=PLAINTEXT://hadoop102:9092,CONTROLLER://hadoop102:9093
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://hadoop102:9092,CONTROLLER://hadoop102:9093
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
##nimbus日志报错如下
2025-05-16 15:46:47.655 o.a.s.n.NimbusInfo main [INFO] Nimbus figures out its name to hadoop102
2025-05-16 15:46:47.694 o.a.s.s.o.a.c.u.Compatibility main [INFO] Running in ZooKeeper 3.4.x compatibility mode
2025-05-16 15:46:47.707 o.a.s.s.o.a.c.f.i.CuratorFrameworkImpl main [INFO] Starting
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:zookeeper.version=3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:host.name=hadoop102
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.version=17.0.15
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.vendor=Oracle Corporation
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.home=/opt/module/jdk-17.0.15
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.class.path=/opt/module/apache-storm-1.2.4/*:/opt/module/apache-storm-1.2.4/lib/storm-core-1.2.4.jar:/opt/module/apache-storm-1.2.4/lib/kryo-3.0.3.jar:/opt/module/apache-storm-1.2.4/lib/reflectasm-1.10.1.jar:/opt/module/apache-storm-1.2.4/lib/asm-5.0.3.jar:/opt/module/apache-storm-1.2.4/lib/minlog-1.3.0.jar:/opt/module/apache-storm-1.2.4/lib/objenesis-2.1.jar:/opt/module/apache-storm-1.2.4/lib/clojure-1.7.0.jar:/opt/module/apache-storm-1.2.4/lib/ring-cors-0.1.5.jar:/opt/module/apache-storm-1.2.4/lib/disruptor-3.3.11.jar:/opt/module/apache-storm-1.2.4/lib/log4j-api-2.8.2.jar:/opt/module/apache-storm-1.2.4/lib/log4j-core-2.8.2.jar:/opt/module/apache-storm-1.2.4/lib/log4j-slf4j-impl-2.8.2.jar:/opt/module/apache-storm-1.2.4/lib/slf4j-api-1.7.21.jar:/opt/module/apache-storm-1.2.4/lib/log4j-over-slf4j-1.6.6.jar:/opt/module/apache-storm-1.2.4/lib/metrics-core-3.1.0.jar:/opt/module/apache-storm-1.2.4/lib/metrics-graphite-3.1.0.jar:/opt/module/apache-storm-1.2.4/lib/servlet-api-2.5.jar:/opt/module/apache-storm-1.2.4/lib/storm-rename-hack-1.2.4.jar:/opt/module/apache-storm-1.2.4/extlib/*:/opt/module/apache-storm-1.2.4/extlib-daemon/*:/opt/module/apache-storm-1.2.4/conf
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.io.tmpdir=/tmp
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:java.compiler=<NA>
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.name=Linux
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.arch=amd64
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:os.version=3.10.0-1160.el7.x86_64
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.name=yjc
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.home=/home/yjc
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Client environment:user.dir=/opt/module/apache-storm-1.2.4/bin
2025-05-16 15:46:47.723 o.a.s.s.o.a.z.ZooKeeper main [INFO] Initiating client connection, connectString=hadoop102:2181,hadoop103:2181,hadoop104:2181/storm sessionTimeout=20000 watcher=org.apache.storm.shade.org.apache.curator.ConnectionState@528e5e38
......
......
......
2025-05-16 15:47:10.310 o.a.s.s.o.a.z.ClientCnxn main-SendThread(hadoop103:2181) [WARN] Session 0x0 for server hadoop102/<unresolved>:2181, unexpected error, closing socket connection and attempting reconnect
java.lang.IllegalArgumentException: Unable to canonicalize address hadoop102/<unresolved>:2181 because it's not resolvable
at org.apache.storm.shade.org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:65) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1001) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) ~[storm-core-1.2.4.jar:1.2.4]
2025-05-16 15:47:10.333 o.a.s.s.o.a.z.ClientCnxn main-SendThread(hadoop103:2181) [WARN] Session 0x0 for server hadoop103/<unresolved>:2181, unexpected error, closing socket connection and attempting reconnect
java.lang.IllegalArgumentException: Unable to canonicalize address hadoop103/<unresolved>:2181 because it's not resolvable
at org.apache.storm.shade.org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:65) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1001) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) ~[storm-core-1.2.4.jar:1.2.4]
2025-05-16 15:47:10.410 o.a.s.s.o.a.z.ClientCnxn main-SendThread(hadoop102:2181) [WARN] Session 0x0 for server hadoop104/<unresolved>:2181, unexpected error, closing socket connection and attempting reconnect
java.lang.IllegalArgumentException: Unable to canonicalize address hadoop104/<unresolved>:2181 because it's not resolvable
at org.apache.storm.shade.org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:65) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1001) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) ~[storm-core-1.2.4.jar:1.2.4]
2025-05-16 15:47:10.437 o.a.s.s.o.a.z.ClientCnxn main-SendThread(hadoop103:2181) [WARN] Session 0x0 for server hadoop104/<unresolved>:2181, unexpected error, closing socket connection and attempting reconnect
java.lang.IllegalArgumentException: Unable to canonicalize address hadoop104/<unresolved>:2181 because it's not resolvable
at org.apache.storm.shade.org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:65) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.SaslServerPrincipal.getServerPrincipal(SaslServerPrincipal.java:41) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1001) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060) ~[storm-core-1.2.4.jar:1.2.4]
2025-05-16 15:47:10.438 o.a.s.d.nimbus main [ERROR] Error on initialization of server service-handler
java.lang.RuntimeException: org.apache.storm.shade.org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /storm
at org.apache.storm.util$wrap_in_runtime.invoke(util.clj:54) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.zookeeper$exists_node_QMARK_$fn__975.invoke(zookeeper.clj:111) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:107) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.zookeeper$mkdirs.invoke(zookeeper.clj:127) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.cluster_state.zookeeper_state_factory$_mkState.invoke(zookeeper_state_factory.clj:32) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.cluster_state.zookeeper_state_factory.mkState(Unknown Source) ~[storm-core-1.2.4.jar:1.2.4]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93) ~[clojure-1.7.0.jar:?]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28) ~[clojure-1.7.0.jar:?]
at org.apache.storm.cluster$mk_distributed_cluster_state.doInvoke(cluster.clj:56) ~[storm-core-1.2.4.jar:1.2.4]
at clojure.lang.RestFn.invoke(RestFn.java:559) ~[clojure-1.7.0.jar:?]
at org.apache.storm.cluster$mk_storm_cluster_state.doInvoke(cluster.clj:279) ~[storm-core-1.2.4.jar:1.2.4]
at clojure.lang.RestFn.invoke(RestFn.java:486) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:201) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.daemon.nimbus$fn__9097$exec_fn__1371__auto____9098.invoke(nimbus.clj:2467) ~[storm-core-1.2.4.jar:1.2.4]
at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$fn__9097$service_handler__9130.doInvoke(nimbus.clj:2464) ~[storm-core-1.2.4.jar:1.2.4]
at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2552) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2589) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2612) ~[storm-core-1.2.4.jar:1.2.4]
at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus.main(Unknown Source) ~[storm-core-1.2.4.jar:1.2.4]
Caused by: org.apache.storm.shade.org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /storm
at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:102) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:268) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl$3.call(ExistsBuilderImpl.java:257) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForegroundStandard(ExistsBuilderImpl.java:254) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:247) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:206) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.shade.org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:35) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.zookeeper$exists_node_QMARK_$fn__975.invoke(zookeeper.clj:110) ~[storm-core-1.2.4.jar:1.2.4]
... 27 more
2025-05-16 15:47:10.443 o.a.s.util main [ERROR] Halting process: ("Error on initialization")
java.lang.RuntimeException: ("Error on initialization")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) ~[storm-core-1.2.4.jar:1.2.4]
at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$fn__9097$service_handler__9130.doInvoke(nimbus.clj:2464) ~[storm-core-1.2.4.jar:1.2.4]
at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2552) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2589) ~[storm-core-1.2.4.jar:1.2.4]
at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2612) ~[storm-core-1.2.4.jar:1.2.4]
at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]
at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
at org.apache.storm.daemon.nimbus.main(Unknown Source) ~[storm-core-1.2.4.jar:1.2.4]
希望能得到问题的原因和解决办法,谢谢大家的帮助