andyonlines 2022-05-19 19:51 采纳率: 50%
浏览 104
已结题

hive on spark 打开文件过多错误

刚开始没事,执行hive sql 没一会就会出现错误,这时候重启hivesever2就好了,过一段时间有出现。log如下:


764913 2022-05-20 03:29:38,807 Log4j2-TF-2-AsyncLogger[AsyncContext@5b464ce8]-1 ERROR Unable to invoke factory method in class org.apache.hadoop.hive.ql.l
       og.HushableRandomAccessFileAppender for element HushableMutableRandomAccess: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.lo
       g4j.core.appender.RandomAccessFileManager$RandomAccessFileManagerFactory@63fd56c1] unable to create manager for [/tmp/andy/operation_logs/7566fc99-
       684d-4773-b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b-ab86-6fb214fe6908] with data [org.apache.logging.log4j.core.appender.RandomAcces
       sFileManager$FactoryData@277ebeec] java.lang.reflect.InvocationTargetException
764914     at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
764915     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
764916     at java.lang.reflect.Method.invoke(Method.java:498)
764917     at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
764918     at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
764919     at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
764920     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:271)
764921     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
764922     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
764923     at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
764924     at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
764925     at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
764926     at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
764927     at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
764928     at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
764929     at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
764930     at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
764931     at org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:380)
764932     at org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:152)
764933     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:45)
764934     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:29)
764935     at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
764936     at java.lang.Thread.run(Thread.java:748)
764937 Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.RandomAccessFileManager$RandomAccessFileManagerF
       actory@63fd56c1] unable to create manager for [/tmp/andy/operation_logs/7566fc99-684d-4773-b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b
       -ab86-6fb214fe6908] with data [org.apache.logging.log4j.core.appender.RandomAccessFileManager$FactoryData@277ebeec]
764938     at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:115)
764939     at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:115)
764940     at org.apache.logging.log4j.core.appender.RandomAccessFileManager.getFileManager(RandomAccessFileManager.java:74)
764941     at org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender.createAppender(HushableRandomAccessFileAppender.java:178)
764942     ... 23 more

 2022-05-20 03:29:38,809 Log4j2-TF-2-AsyncLogger[AsyncContext@5b464ce8]-1 ERROR RandomAccessFileManager (/tmp/andy/operation_logs/7566fc99-684d-4773
       -b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b-ab86-6fb214fe6908) java.io.FileNotFoundException: /tmp/andy/operation_logs/7566fc99-684d-
       4773-b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b-ab86-6fb214fe6908 (打开的文件过多) java.io.FileNotFoundException: /tmp/andy/operation
       _logs/7566fc99-684d-4773-b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b-ab86-6fb214fe6908 (打开的文件过多)
764945     at java.io.RandomAccessFile.open0(Native Method)
764946     at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
764947     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
764948     at java.io.RandomAccessFile.<init>(RandomAccessFile.java:124)
764949     at org.apache.logging.log4j.core.appender.RandomAccessFileManager$RandomAccessFileManagerFactory.createManager(RandomAccessFileManager.java:199
       )
764950     at org.apache.logging.log4j.core.appender.RandomAccessFileManager$RandomAccessFileManagerFactory.createManager(RandomAccessFileManager.java:177
       )
764951     at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:113)
764952     at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:115)
764953     at org.apache.logging.log4j.core.appender.RandomAccessFileManager.getFileManager(RandomAccessFileManager.java:74)
764954     at org.apache.hadoop.hive.ql.log.HushableRandomAccessFileAppender.createAppender(HushableRandomAccessFileAppender.java:178)
764955     at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
764956     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
764957     at java.lang.reflect.Method.invoke(Method.java:498)
764958     at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
764959     at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
764960     at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
764961     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:271)
764962     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
764963     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
764964     at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
764965     at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
764966     at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
764967     at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
764968     at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
764969     at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
764970     at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
764971     at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
764972     at org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:380)
764973     at org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:152)
764974     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:45)
764975     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:29)
764976     at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
764977     at java.lang.Thread.run(Thread.java:748)

764979 2022-05-20 03:29:38,809 Log4j2-TF-2-AsyncLogger[AsyncContext@5b464ce8]-1 ERROR Unable to invoke factory method in class org.apache.hadoop.hive.ql.l
       og.HushableRandomAccessFileAppender for element HushableMutableRandomAccess: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.lo
       g4j.core.appender.RandomAccessFileManager$RandomAccessFileManagerFactory@63fd56c1] unable to create manager for [/tmp/andy/operation_logs/7566fc99-
       684d-4773-b87a-4462f9b86538/andy_20220520032938_eadaa0c2-7010-400b-ab86-6fb214fe6908] with data [org.apache.logging.log4j.core.appender.RandomAcces
       sFileManager$FactoryData@5e6d472a] java.lang.reflect.InvocationTargetException
764980     at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
764981     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
764982     at java.lang.reflect.Method.invoke(Method.java:498)
764983     at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
764984     at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:958)
764985     at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:898)
764986     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.createAppender(RoutingAppender.java:271)
764987     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.getControl(RoutingAppender.java:255)
764988     at org.apache.logging.log4j.core.appender.routing.RoutingAppender.append(RoutingAppender.java:225)
764989     at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
764990     at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
764991     at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
764992     at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
764993     at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:448)
764994     at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:433)
764995     at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:417)
764996     at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:79)
764997     at org.apache.logging.log4j.core.async.AsyncLogger.actualAsyncLog(AsyncLogger.java:380)
764998     at org.apache.logging.log4j.core.async.RingBufferLogEvent.execute(RingBufferLogEvent.java:152)
764999     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:45)
765000     at org.apache.logging.log4j.core.async.RingBufferLogEventHandler.onEvent(RingBufferLogEventHandler.java:29)
765001     at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:129)
765002     at java.lang.Thread.run(Thread.java:748)
  • 写回答

3条回答 默认 最新

  • 呔 小怪兽休走 2022-05-25 16:37
    关注

    你用的beeline方式连接的吧 beeline连接是有限的当连接超出的时候就会出问题 你尝试一下 直接连接hivecli 应该会好一点

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(2条)

报告相同问题?

问题事件

  • 系统已结题 6月2日
  • 已采纳回答 5月25日
  • 创建了问题 5月19日

悬赏问题

  • ¥100 华为ensp只要2-9实验运行结果能做的来
  • ¥15 我这个代码哪里有问题 acm 平台上显示错误 90%,我自己运行好像没什么问题
  • ¥50 C#编程中使用printDocument类实现文字排版打印问题
  • ¥15 找会编程的帅哥美女 可以用MATLAB里面的simulink编程,用Keil5编也可以。
  • ¥15 已知隐函数其中一个变量τ的具体值,求另一个变量
  • ¥15 r语言Hurst指数
  • ¥15 Acrn IVSHMEM doorbell问题
  • ¥15 yolov5中的val测试集训练时数量变小问题
  • ¥15 MPLS/VPN实验中MPLS的配置问题
  • ¥15 materialstudio氢键计算问题