qq_42759137 2020-03-22 18:33 采纳率: 33.3%
浏览 463

kafka数据上传hbase的问题

我使用的环境是hdp的伪分布集群
我的项目是flume采集数据发送到kafka的各个topic当中 再由jar文件使得从kafka当中获取数据 发送到hbase做持久化 然后因为数据量颇大 每次传个半个小时的数据 regionserver就挂掉了
项目是肯定没问题的 因为目前在学习阶段 别人是可以执行且不报错的

问题如下所示

java.io.FileNotFoundException: File /tmp/hbase-root/hbase/lib does not exist
    at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431) ~[hadoop-common-2.7.3.jar!/:na]
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517) ~[hadoop-common-2.7.3.jar!/:na]
    at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557) ~[hadoop-common-2.7.3.jar!/:na]
    at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:674) ~[hadoop-common-2.7.3.jar!/:na]
    at org.apache.hadoop.hbase.util.DynamicClassLoader.loadNewJars(DynamicClassLoader.java:178) [hbase-common-1.1.2.jar!/:1.1.2]
    at org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:142) [hbase-common-1.1.2.jar!/:1.1.2]
    at java.lang.Class.forName0(Native Method) [na:1.8.0_161]
    at java.lang.Class.forName(Class.java:348) [na:1.8.0_161]
    at org.apache.hadoop.hbase.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1543) [hbase-client-1.1.2.jar!/:1.1.2]
    at org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:120) [hbase-client-1.1.2.jar!/:1.1.2]
    at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:134) [hbase-client-1.1.2.jar!/:1.1.2]
    at org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:54) [hbase-client-1.1.2.jar!/:1.1.2]
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200) [hbase-client-1.1.2.jar!/:1.1.2]
    at org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:708) [hbase-client-1.1.2.jar!/:1.1.2]

他突然开始寻找 File /tmp/hbase-root/hbase/lib does not exist 这个路径的文件 我的项目中并没有从这个路径下寻找文件
我前往到这个路径 路径是空的 就是根本没有这个路径
然后我前往hbase的log中查看
hbase来了一套组合拳 

2020-03-21 19:29:49,789 ERROR [Thread-19] util.PolicyRefresher: PolicyRefresher(serviceName=Sandbox_hbase): failed to refresh policies. Will continue to use last known version of policies (6)
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503)
    at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:135)
    at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
    at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
    at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
    ... 8 more

然后就是读取超时

com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503)
    at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:135)
    at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
    at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
    at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)

然后就是最匪夷所思的异常

2020-03-21 19:33:36,252 ERROR [Thread-19] util.PolicyRefresher: PolicyRefresher(serviceName=Sandbox_hbase): failed to refresh policies. Will continue to use last known version of policies (6)
com.sun.jersey.api.client.ClientHandlerException: java.net.SocketTimeoutException: Read timed out
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149)
    at com.sun.jersey.api.client.Client.handle(Client.java:648)
    at com.sun.jersey.api.client.WebResource.handle(WebResource.java:670)
    at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
    at com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:503)
    at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:135)
    at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264)
    at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202)
    at org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:171)
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:240)
    at com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:147)
    ... 8 more

求大佬解答

  • 写回答

1条回答 默认 最新

  • 憧憬blog 2023-03-15 02:21
    关注

    根据你提供的问题，可能存在以下几个可能的原因：

    1. HBase配置问题：错误信息中显示File /tmp/hbase-root/hbase/lib does not exist，看起来是在加载HBase相关的类库时出现了问题。检查HBase配置文件hbase-site.xml中配置的hbase.tmp.dir属性是否存在，并且是否配置正确。

    2. 系统资源不足：你说每次传送半个小时的数据，就导致了regionserver挂掉。这可能是由于系统资源不足导致的。你可以检查HBase的日志，查看是否有OOM（Out Of Memory）的错误，或者查看系统资源的使用情况（比如CPU、内存、硬盘空间等）。

    3. 网络问题：你的日志中出现了"java.net.SocketTimeoutException: Read timed out"的错误，这意味着HBase客户端无法正常连接HBase服务器。确保HBase服务器正常运行，并尝试使用其他工具测试客户端是否可以连接HBase服务器。

    总之，以上的建议仅供参考，由于具体情况不得而知，若以上解决方案无效，请提供更多详细信息，以方便更好的解决问题。

    评论

报告相同问题？

悬赏问题

  • ¥15 DIFY API Endpoint 问题。
  • ¥20 sub地址DHCP问题
  • ¥15 delta降尺度计算的一些细节，有偿
  • ¥15 Arduino红外遥控代码有问题
  • ¥15 数值计算离散正交多项式
  • ¥30 数值计算均差系数编程
  • ¥15 redis-full-check比较 两个集群的数据出错
  • ¥15 Matlab编程问题
  • ¥15 训练的多模态特征融合模型准确度很低怎么办
  • ¥15 kylin启动报错log4j类冲突