问题现象:
启动hbase之后过两三分钟左右就自动崩了
具体描述:
就一台虚拟机,所以都装的单节点,一开始看日志以为是zk的问题,特意调整配置装伪分布式三个节点,结果还是一样崩,日志看来看去就说zk连不上,可是zk服务正常的也没做什么特别限制啊。昨天弄到凌晨都没出结果,蹲巨佬解惑。
操作系统及版本信息:
ubntu22.04.5
hadoop 3.3.0 单节点
zookeeper 3.4.13 单节点/三节点集群
hbase2.4.18 单节点



发现hbase崩了输出的日志如下:
2024-12-02 22:33:40,193 INFO [ReadOnlyZKClient-localhost:2181@0x42beb067-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x10000a5384f0002
2024-12-02 22:33:40,194 INFO [ReadOnlyZKClient-localhost:2181@0x42beb067] zookeeper.ZooKeeper: Session: 0x10000a5384f0002 closed
2024-12-02 22:37:39,622 INFO [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=380.55 MB, usedSize=286.41 KB, freeSize=380.27 MB, max=380.55 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0
2024-12-02 22:37:39,824 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Master not active after 300000ms
at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:221)
at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:177)
at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:407)
at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:243)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:140)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2969)
2024-12-02 22:37:40,901 INFO [main] master.HMaster: STARTING service HMaster
2024-12-02 22:37:40,902 INFO [main] util.VersionInfo: HBase 2.4.18
2024-12-02 22:37:40,902 INFO [main] util.VersionInfo: Source code repository git://82c3649f26c3/home/zhangduo/hbase-rm/output/hbase revision=a1767f4d76859c0068720a6c1e5cb78282ebfe1e
2024-12-02 22:37:40,903 INFO [main] util.VersionInfo: Compiled by zhangduo on Tue May 21 08:45:43 UTC 2024
2024-12-02 22:37:40,903 INFO [main] util.VersionInfo: From source with checksum 6f22009564e4e2c12ba04bfc2a643af18f467b1f68387968534a2b5d285d389e9d1990a1e353d060ecd689d630ca0b094b808a4d45cf5599ae0551e779cdc0bb
2024-12-02 22:37:41,059 INFO [main] zookeeper.RecoverableZooKeeper: Process identifier=clean znode for master connecting to ZooKeeper ensemble=localhost:2181
2024-12-02 22:37:41,066 INFO [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.8.4-9316c2a7a97e1666d8f4593f34dd6fc36ecc436c, built on 2024-02-12 22:16 UTC
2024-12-02 22:37:41,066 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=ubuntu-virtual-machine
2024-12-02 22:37:41,067 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_421
2024-12-02 22:37:41,067 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2024-12-02 22:37:41,067 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/jdk1.8.0_421/jre
2024-12-02 22:37:41,067 INFO [main] zookeeper.ZooKeeper: ne-pluginstorage-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-services-core-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-services-api-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-registry-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-router-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.3.0.jar
2024-12-02 22:37:41,067 INFO [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop/lib/native
2024-12-02 22:37:41,067 INFO [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2024-12-02 22:37:41,067 INFO [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2024-12-02 22:37:41,068 INFO [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2024-12-02 22:37:41,068 INFO [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2024-12-02 22:37:41,068 INFO [main] zookeeper.ZooKeeper: Client environment:os.version=6.8.0-48-generic
2024-12-02 22:37:41,069 INFO [main] zookeeper.ZooKeeper: Client environment:user.name=hadoop
2024-12-02 22:37:41,069 INFO [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
2024-12-02 22:37:41,069 INFO [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop
2024-12-02 22:37:41,069 INFO [main] zookeeper.ZooKeeper: Client environment:os.memory.free=42MB
2024-12-02 22:37:41,069 INFO [main] zookeeper.ZooKeeper: Client environment:os.memory.max=860MB
2024-12-02 22:37:41,070 INFO [main] zookeeper.ZooKeeper: Client environment:os.memory.total=75MB
2024-12-02 22:37:41,072 INFO [main] zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@15043a2f
2024-12-02 22:37:41,078 INFO [main] common.X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2024-12-02 22:37:41,089 INFO [main] zookeeper.ClientCnxnSocket: jute.maxbuffer value is 1048575 Bytes
2024-12-02 22:37:41,096 INFO [main] zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled=false
2024-12-02 22:37:41,104 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181.
2024-12-02 22:37:41,104 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: SASL config status: Will not attempt to authenticate using SASL (unknown error)
2024-12-02 22:37:41,112 WARN [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: 连接被拒绝
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
2024-12-02 22:37:42,221 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181.
2024-12-02 22:37:42,222 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: SASL config status: Will not attempt to authenticate using SASL (unknown error)
2024-12-02 22:37:42,222 WARN [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: 连接被拒绝
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
2024-12-02 22:37:42,324 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 1 attempts
2024-12-02 22:37:42,325 WARN [main] hbase.ZNodeClearer: ZooKeeper exception deleting znode
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/ubuntu-virtual-machine,16000,1733149957822
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1614)
at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:162)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:927)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:920)
at org.apache.hadoop.hbase.ZNodeClearer.clear(ZNodeClearer.java:201)
at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:156)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:140)
at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2969)
2024-12-02 22:37:43,325 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181.
2024-12-02 22:37:43,325 INFO [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: SASL config status: Will not attempt to authenticate using SASL (unknown error)
2024-12-02 22:37:43,428 INFO [main] zookeeper.ZooKeeper: Session: 0x0 closed
2024-12-02 22:37:43,428 INFO [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x0