珈蓝冰雨 2024-12-03 12:49 采纳率: 33.3%
浏览 18

hbase单节点故障帖

问题现象:
启动hbase之后过两三分钟左右就自动崩了

具体描述:
就一台虚拟机,所以都装的单节点,一开始看日志以为是zk的问题,特意调整配置装伪分布式三个节点,结果还是一样崩,日志看来看去就说zk连不上,可是zk服务正常的也没做什么特别限制啊。昨天弄到凌晨都没出结果,蹲巨佬解惑。

操作系统及版本信息:
ubntu22.04.5
hadoop 3.3.0 单节点
zookeeper 3.4.13 单节点/三节点集群
hbase2.4.18 单节点

img

img

img

发现hbase崩了输出的日志如下:


2024-12-02 22:33:40,193 INFO  [ReadOnlyZKClient-localhost:2181@0x42beb067-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x10000a5384f0002
2024-12-02 22:33:40,194 INFO  [ReadOnlyZKClient-localhost:2181@0x42beb067] zookeeper.ZooKeeper: Session: 0x10000a5384f0002 closed
2024-12-02 22:37:39,622 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=380.55 MB, usedSize=286.41 KB, freeSize=380.27 MB, max=380.55 MB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=29, evicted=0, evictedPerRun=0.0
2024-12-02 22:37:39,824 ERROR [main] master.HMasterCommandLine: Master exiting
java.lang.RuntimeException: Master not active after 300000ms
        at org.apache.hadoop.hbase.util.JVMClusterUtil.waitForEvent(JVMClusterUtil.java:221)
        at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:177)
        at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:407)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:243)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:145)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:140)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2969)
2024-12-02 22:37:40,901 INFO  [main] master.HMaster: STARTING service HMaster
2024-12-02 22:37:40,902 INFO  [main] util.VersionInfo: HBase 2.4.18
2024-12-02 22:37:40,902 INFO  [main] util.VersionInfo: Source code repository git://82c3649f26c3/home/zhangduo/hbase-rm/output/hbase revision=a1767f4d76859c0068720a6c1e5cb78282ebfe1e
2024-12-02 22:37:40,903 INFO  [main] util.VersionInfo: Compiled by zhangduo on Tue May 21 08:45:43 UTC 2024
2024-12-02 22:37:40,903 INFO  [main] util.VersionInfo: From source with checksum 6f22009564e4e2c12ba04bfc2a643af18f467b1f68387968534a2b5d285d389e9d1990a1e353d060ecd689d630ca0b094b808a4d45cf5599ae0551e779cdc0bb
2024-12-02 22:37:41,059 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=clean znode for master connecting to ZooKeeper ensemble=localhost:2181
2024-12-02 22:37:41,066 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.8.4-9316c2a7a97e1666d8f4593f34dd6fc36ecc436c, built on 2024-02-12 22:16 UTC
2024-12-02 22:37:41,066 INFO  [main] zookeeper.ZooKeeper: Client environment:host.name=ubuntu-virtual-machine
2024-12-02 22:37:41,067 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_421
2024-12-02 22:37:41,067 INFO  [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2024-12-02 22:37:41,067 INFO  [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/lib/jvm/jdk1.8.0_421/jre
2024-12-02 22:37:41,067 INFO  [main] zookeeper.ZooKeeper: ne-pluginstorage-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-services-core-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-services-api-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-registry-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-router-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.3.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.3.0.jar
2024-12-02 22:37:41,067 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop/lib/native
2024-12-02 22:37:41,067 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2024-12-02 22:37:41,067 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2024-12-02 22:37:41,068 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2024-12-02 22:37:41,068 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2024-12-02 22:37:41,068 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=6.8.0-48-generic
2024-12-02 22:37:41,069 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=hadoop
2024-12-02 22:37:41,069 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/home/hadoop
2024-12-02 22:37:41,069 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/home/hadoop
2024-12-02 22:37:41,069 INFO  [main] zookeeper.ZooKeeper: Client environment:os.memory.free=42MB
2024-12-02 22:37:41,069 INFO  [main] zookeeper.ZooKeeper: Client environment:os.memory.max=860MB
2024-12-02 22:37:41,070 INFO  [main] zookeeper.ZooKeeper: Client environment:os.memory.total=75MB
2024-12-02 22:37:41,072 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@15043a2f
2024-12-02 22:37:41,078 INFO  [main] common.X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2024-12-02 22:37:41,089 INFO  [main] zookeeper.ClientCnxnSocket: jute.maxbuffer value is 1048575 Bytes
2024-12-02 22:37:41,096 INFO  [main] zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled=false
2024-12-02 22:37:41,104 INFO  [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181.
2024-12-02 22:37:41,104 INFO  [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: SASL config status: Will not attempt to authenticate using SASL (unknown error)
2024-12-02 22:37:41,112 WARN  [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: 连接被拒绝
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
2024-12-02 22:37:42,221 INFO  [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181.
2024-12-02 22:37:42,222 INFO  [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: SASL config status: Will not attempt to authenticate using SASL (unknown error)
2024-12-02 22:37:42,222 WARN  [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Session 0x0 for server localhost/127.0.0.1:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException.
java.net.ConnectException: 连接被拒绝
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:715)
        at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:344)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
2024-12-02 22:37:42,324 ERROR [main] zookeeper.RecoverableZooKeeper: ZooKeeper delete failed after 1 attempts
2024-12-02 22:37:42,325 WARN  [main] hbase.ZNodeClearer: ZooKeeper exception deleting znode
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/rs/ubuntu-virtual-machine,16000,1733149957822
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1614)
        at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:162)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:927)
        at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNodeFailSilent(ZKUtil.java:920)
        at org.apache.hadoop.hbase.ZNodeClearer.clear(ZNodeClearer.java:201)
        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:156)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:140)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2969)
2024-12-02 22:37:43,325 INFO  [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181.
2024-12-02 22:37:43,325 INFO  [main-SendThread(localhost:2181)] zookeeper.ClientCnxn: SASL config status: Will not attempt to authenticate using SASL (unknown error)
2024-12-02 22:37:43,428 INFO  [main] zookeeper.ZooKeeper: Session: 0x0 closed
2024-12-02 22:37:43,428 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x0

  • 写回答

1条回答 默认 最新

  • 此木Y 2024-12-06 18:26
    关注

    ERROR [main] master.HMasterCommandLine: Master exiting 看这个是说Master已经存在了,可以JPS看一下进程,或者去zkCil.sh,用ls \ 看一下有没有HBASE

    评论

报告相同问题?

问题事件

  • 修改了问题 12月3日
  • 创建了问题 12月3日