关于hadoop_hive的问题 10C

本人接触hadoop与hive不长时间 因为是接手别人的工作所以有点吃力。最近遇到一个问题,不知道应该怎样解决,希望有大神可以解答一下。

问题是这样的:我执行一个脚本从hive里面读取数据然后写入一个csv文件里,hiveQL语句其实也就是从一个表中读取数据,加上一些字段得时间段参数之类的。
但是在跑的时候时不时会出现问题 日志如下:

Task with the most failures(4):

Task ID:
task_1513574350768_3535_m_000655

URL:

http://hadoopnode102:8088/taskdetails.jsp?jobid=job_1513574350768_3535&tipid=task_1513574350768_3535_m_000655

Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (数据在这里就先不铺出来了)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (数据在这里就先不铺出来了)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:503)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:176)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.net.SocketTimeoutException: 75000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/132.96.186.7:58295 remote=/132.96.186.9:50010]
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:723)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at org.apache.hadoop.hive.ql.exec.FilterOperator.processOp(FilterOperator.java:120)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:815)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:95)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:157)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:493)
... 9 more
Caused by: java.net.SocketTimeoutException: 75000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/132.96.186.7:58295 remote=/132.96.186.9:50010]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2201)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1439)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)

Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

但是跑多几次就可以跑出来, 所以我觉得应该不是数据问题,是配置上没有做好
而且这个查询只有map也没有reduce,不知道该如何排查。 希望有大神知道如何解决指导指导。谢谢~

3个回答

ExecMapper.java:185

qq_16133099
qq_16133099 看了 但是有点不太懂~ try { if (this.mo.getDone()) { done = true; } else { this.mo.process((Writable)value); } } catch (Throwable e) { this.abort = true; if ((e instanceof OutOfMemoryError)) { throw ((OutOfMemoryError)e); } l4j.fatal(StringUtils.stringifyException(e)); throw new RuntimeException(e); }
大约 2 年之前 回复

感觉hive安装的有点问题

qq_16133099
qq_16133099 但是这个已经用了很久了 。。。而且数据量是有点大
大约 2 年之前 回复

好像内存不够用了,任务并行多也可能内存不够用。运行任务时监控下集群内存

Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
立即提问