hive on spark 打开文件过多错误

刚开始没事，执行hive sql 没一会就会出现错误，这时候重启hivesever2就好了，过一段时间有出现。log如下：


764913 2022-05-20 03:29:38,807 Log4j2-TF-2-AsyncLogger[AsyncContext@5b464ce8]-1 ERROR Unable to invoke factory method in class org.apache.hadoop.hive.ql.l
       og.HushableRandomAccessFileAppender for element HushableMutableRandomAccess: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.lo
       g4j.core.appender.RandomAccessFileManager$RandomAccessFileManagerFactory@63fd56c1] unable to create manager for [/tmp/andy/operation_logs/7566fc99-
       684d-4773-b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b-ab86-6fb214fe6908] with data [org.apache.logging.log4j.core.appender.RandomAcces
       sFileManager$FactoryData@277ebeec] java.lang.reflect.InvocationTargetException
764914     at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
764915     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
764916     at java.lang.reflect.Method.invoke(Method.java:498)
764917     at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
764918     at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
764919     at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
764920     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:271)
764921     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
764922     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
764923     at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
764924     at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
764925     at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
764926     at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
764927     at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
764928     at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
764929     at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
764930     at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
764931     at org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:380)
764932     at org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:152)
764933     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:45)
764934     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:29)
764935     at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
764936     at java.lang.Thread.run(Thread.java:748)
764937 Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.RandomAccessFileManager$RandomAccessFileManagerF
       actory@63fd56c1] unable to create manager for [/tmp/andy/operation_logs/7566fc99-684d-4773-b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b
       -ab86-6fb214fe6908] with data [org.apache.logging.log4j.core.appender.RandomAccessFileManager$FactoryData@277ebeec]
764938     at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
764939     at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:115)
764940     at org.apache.logging.log4j.core.appender.RandomAccessFileManager.getFileManager(RandomAccessFileManager.java:74)
764941     at org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender.createAppender(HushableRandomAccessFileAppender.java:178)
764942     ... 23 more

 2022-05-20 03:29:38,809 Log4j2-TF-2-AsyncLogger[AsyncContext@5b464ce8]-1 ERROR RandomAccessFileManager (/tmp/andy/operation_logs/7566fc99-684d-4773
       -b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b-ab86-6fb214fe6908) java.io.FileNotFoundException: /tmp/andy/operation_logs/7566fc99-684d-
       4773-b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b-ab86-6fb214fe6908 (打开的文件过多) java.io.FileNotFoundException: /tmp/andy/operation
       _logs/7566fc99-684d-4773-b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b-ab86-6fb214fe6908 (打开的文件过多)
764945     at java.io.RandomAccessFile.open0(Native Method)
764946     at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
764947     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
764948     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:124)
764949     at org.apache.logging.log4j.core.appender.RandomAccessFileManager$RandomAccessFileManagerFactory.createManager(RandomAccessFileManager.java:199
       )
764950     at org.apache.logging.log4j.core.appender.RandomAccessFileManager$RandomAccessFileManagerFactory.createManager(RandomAccessFileManager.java:177
       )
764951     at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
764952     at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:115)
764953     at org.apache.logging.log4j.core.appender.RandomAccessFileManager.getFileManager(RandomAccessFileManager.java:74)
764954     at org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender.createAppender(HushableRandomAccessFileAppender.java:178)
764955     at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
764956     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
764957     at java.lang.reflect.Method.invoke(Method.java:498)
764958     at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
764959     at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
764960     at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
764961     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:271)
764962     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
764963     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
764964     at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
764965     at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
764966     at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
764967     at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
764968     at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
764969     at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
764970     at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
764971     at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
764972     at org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:380)
764973     at org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:152)
764974     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:45)
764975     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:29)
764976     at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
764977     at java.lang.Thread.run(Thread.java:748)

764979 2022-05-20 03:29:38,809 Log4j2-TF-2-AsyncLogger[AsyncContext@5b464ce8]-1 ERROR Unable to invoke factory method in class org.apache.hadoop.hive.ql.l
       og.HushableRandomAccessFileAppender for element HushableMutableRandomAccess: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.lo
       g4j.core.appender.RandomAccessFileManager$RandomAccessFileManagerFactory@63fd56c1] unable to create manager for [/tmp/andy/operation_logs/7566fc99-
       684d-4773-b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b-ab86-6fb214fe6908] with data [org.apache.logging.log4j.core.appender.RandomAcces
       sFileManager$FactoryData@5e6d472a] java.lang.reflect.InvocationTargetException
764980     at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
764981     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
764982     at java.lang.reflect.Method.invoke(Method.java:498)
764983     at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
764984     at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
764985     at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
764986     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:271)
764987     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
764988     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
764989     at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
764990     at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
764991     at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
764992     at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
764993     at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
764994     at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
764995     at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
764996     at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
764997     at org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:380)
764998     at org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:152)
764999     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:45)
765000     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:29)
765001     at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
765002     at java.lang.Thread.run(Thread.java:748)

写回答
好问题 0 提建议
追加酬金
关注问题
分享
邀请回答
编辑收藏删除
收藏举报

3条回答默认最新

关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
呔小怪兽休走 2022-05-25 16:37
关注
你用的beeline方式连接的吧 beeline连接是有限的当连接超出的时候就会出问题你尝试一下直接连接hivecli 应该会好一点

本回答被题主选为最佳回答 , 对您是否有帮助呢?

解决 1
无用
评论打赏
分享
举报

评论

按下Enter换行，Ctrl+Enter发表内容

查看更多回答(2条)

报告相同问题？

关注问题

hive on spark 打开文件过多错误 hadoop hive spark
2022-05-19 19:51

回答 3 已采纳你用的beeline方式连接的吧 beeline连接是有限的当连接超出的时候就会出问题你尝试一下直接连接hivecli 应该会好一点
配置hive on spark测试，创建表，insert测试效果报错，如何解决？ hadoop hive spark 有问必答
2022-04-21 01:37

回答 2 已采纳你在b站视频下面看看评论区，评论区下面的前几个有大佬提供了解决思路。我当时这里报错了，我用了评论区给的方法成功了。如果评论区的方法没有解决，说一个最不好听的方法，你把hive和spark重新装一遍。我
hive on spark运行sql报错 hive spark 有问必答
2021-09-28 16:58

回答 2 已采纳该问题由hive与spark两者版本不兼容导致的，具体的解释，在官网是有说明的：Hive on Spark is only tested with a specific version of Spar
Hive on Spark调优（大数据技术8）
2023-05-20 09:45

菜鸟Octopus的博客在Hive中，无论其计算引擎是什么，所有的计算任务都可分为Map阶段和Reduce阶段。默认情况下，Hive会根据Reduce端输入数据的大小，估算一个Reduce并行度。但是在某些情况下，其估计值不一定是最合适的，故需要人为...
Hive on Spark配置问题 hive spark
2022-12-08 11:36

回答 3 已采纳 hive-seit.xml文件里面必须要添加设置执行引擎为spark
大数据框架中的hadoop和hive spark presto tez是什么关系 hadoop hive spark
2022-12-24 10:29

回答 1 已采纳 Hadoop是一个分布式计算框架，可以在大数据集上运行分布式应用程序。它由许多组件组成，包括HDFS（分布式文件系统）和MapReduce（分布式计算引擎）。Hive是一个基于Hadoop的数据仓库系
hive中无法查看spark创建的表 hadoop hive spark 有问必答
2021-06-25 12:55

回答 1 已采纳你切到对应的数据库吗，xshell只是远程工具。
大数据Spark纯净版安装包，用于快速集成Hive on Spark
2024-03-26 15:01

Spark纯净版安装包是一个用于安装Apache Spark的软件包，该软件包提供了Spark的基本功能和组件，但不包含任何额外的扩展或依赖项。纯净版安装包旨在提供一个轻量级、简单易用的Spark安装选项，适用于用户希望快速...
Spark 读取 Hive 数据报错 NoSuchMethodError : org.apache.spark.sql.catalyst.catalog.SessionCatalog hive spark 大数据
2022-12-15 15:54

回答 1 已采纳 22/12/15 15:32:44 INFO SparkContext: Invoking stop() from shutdown hook集群资源不足，且动态资源分配申请的executors、内存
Hive导入$$分隔符文件 hive 大数据数据仓库
2022-07-26 16:15

回答 1 已采纳不能用row format delimited fields terminated by，改用 ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.
spark与hive版本问题? 本机正常运行,服务器上报错. hive java spark
2021-07-30 17:14

回答 1 已采纳可参考：https://blog.csdn.net/zt15732625878/article/details/85245731看客户端的 hive-jdbc的version 和hive Server
Hive on Spark调优（大数据技术3）
2023-05-17 07:30

菜鸟Octopus的博客在将NodeManager的总内存平均分配给每个Executor，最后再将单个Executor的内存按照大约10:1的比例分配到spark.executor.memory和spark.executor.memoryOverhead。动态分配可根据一个Spark应用的工作负载，动态的调整...
hive创建外部表导入csv数据格式出错 hadoop hive 大数据
2022-06-06 17:18

回答 2 已采纳在window用excel把csv改成\t 分割,建表分割符改成\t
大数据最佳实践-hive on spark
2022-11-27 19:22

WakeUpCcc的博客目录 Hive on Spark与SparkSQLSpark 内存配置spark动态分配Hive Hive on Spark与SparkSQL Hive是Hadoop中的标准SQL引擎，也是最古老的引擎之一。H...
Hive on Spark调优（大数据技术6）
2023-05-19 23:30

菜鸟Octopus的博客相关文章： Hive on Spark调优（大数据技术1） Hive on Spark调优（大数据技术2） Hive on Spark调优（大数据技术3） Hive on Spark调优（大数据技术4） Hive on Spark调优（大数据技术5） Hive on Spark调优...
没有解决我的问题, 去提问

问题事件

关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
系统已结题 6月2日
关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
已采纳回答 5月25日
关注

码龄粉丝数原力等级 --

被采纳

被点赞

采纳率
创建了问题 5月19日

悬赏问题

¥15 Attention is all you need 的代码运行
¥15 一个服务器已经有一个系统了如果用usb再装一个系统，原来的系统会被覆盖掉吗
¥15 使用esm_msa1_t12_100M_UR50S蛋白质语言模型进行零样本预测时，终端显示出了sequence handled的进度条，但是并不出结果就自动终止回到命令提示行了是怎么回事：
¥15 前置放大电路与功率放大电路相连放大倍数出现问题
¥30 关于<main>标签页面跳转的问题
¥80 部署运行web自动化项目
¥15 腾讯云如何建立同一个项目中物模型之间的联系
¥30 VMware 云桌面水印如何添加
¥15 用ns3仿真出5G核心网网元
¥15 matlab答疑关于海上风电的爬坡事件检测

hive on spark 打开文件过多错误

3条回答 默认 最新

问题事件

悬赏问题

3条回答默认最新