weixin_54445252 2022-10-30 21:26 采纳率: 0%
浏览 45

MapReduce,运行的错误

问题遇到的现象和发生背景

MapReduce运行的问题

用代码块功能插入代码，请勿粘贴截图

[wwh@hadoop102 opt]$ hadoop jar phone1.jar com.p1.PhoneSubmitter /phone_data.txt /a1
2022-10-30 21:04:53,039 INFO client.RMProxy: Connecting to ResourceManager at hadoop103/192.168.10.103:8032
2022-10-30 21:04:54,286 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
2022-10-30 21:04:54,328 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/wwh/.staging/job_1667121230615_0001
2022-10-30 21:04:54,589 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-10-30 21:04:54,826 INFO input.FileInputFormat: Total input files to process : 1
2022-10-30 21:04:55,015 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-10-30 21:04:55,067 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-10-30 21:04:55,101 INFO mapreduce.JobSubmitter: number of splits:1
2022-10-30 21:04:55,375 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2022-10-30 21:04:55,434 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1667121230615_0001
2022-10-30 21:04:55,434 INFO mapreduce.JobSubmitter: Executing with tokens: []
2022-10-30 21:04:55,671 INFO conf.Configuration: resource-types.xml not found
2022-10-30 21:04:55,685 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2022-10-30 21:04:56,504 INFO impl.YarnClientImpl: Submitted application application_1667121230615_0001
2022-10-30 21:04:56,571 INFO mapreduce.Job: The url to track the job: http://hadoop103:8088/proxy/application_1667121230615_0001/
2022-10-30 21:04:56,572 INFO mapreduce.Job: Running job: job_1667121230615_0001
2022-10-30 21:05:11,072 INFO mapreduce.Job: Job job_1667121230615_0001 running in uber mode : false
2022-10-30 21:05:11,074 INFO mapreduce.Job: map 0% reduce 0%
2022-10-30 21:05:18,293 INFO mapreduce.Job: Task Id : attempt_1667121230615_0001_m_000000_0, Status : FAILED
[2022-10-30 21:05:17.255]Container [pid=45231,containerID=container_1667121230615_0001_01_000002] is running 255060480B beyond the 'VIRTUAL' memory limit. Current usage: 76.8 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1667121230615_0001_01_000002 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 45248 45231 45231 45231 (java) 164 158 2500128768 19561 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/wwh/appcache/application_1667121230615_0001/container_1667121230615_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1667121230615_0001/container_1667121230615_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.103 40380 attempt_1667121230615_0001_m_000000_0 2
|- 45231 45230 45231 45231 (bash) 0 0 9789440 110 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/wwh/appcache/application_1667121230615_0001/container_1667121230615_0001_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1667121230615_0001/container_1667121230615_0001_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.103 40380 attempt_1667121230615_0001_m_000000_0 2 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1667121230615_0001/container_1667121230615_0001_01_000002/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1667121230615_0001/container_1667121230615_0001_01_000002/stderr

[2022-10-30 21:05:17.534]Container killed on request. Exit code is 143
[2022-10-30 21:05:17.556]Container exited with a non-zero exit code 143.

2022-10-30 21:05:30,445 INFO mapreduce.Job: Task Id : attempt_1667121230615_0001_m_000000_1, Status : FAILED
[2022-10-30 21:05:28.505]Container [pid=45244,containerID=container_1667121230615_0001_01_000003] is running 261368320B beyond the 'VIRTUAL' memory limit. Current usage: 147.5 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1667121230615_0001_01_000003 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 45244 45243 45244 45244 (bash) 0 1 9789440 98 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/wwh/appcache/application_1667121230615_0001/container_1667121230615_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1667121230615_0001/container_1667121230615_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.103 40380 attempt_1667121230615_0001_m_000000_1 3 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1667121230615_0001/container_1667121230615_0001_01_000003/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1667121230615_0001/container_1667121230615_0001_01_000003/stderr
|- 45254 45244 45244 45244 (java) 189 181 2506436608 37659 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/wwh/appcache/application_1667121230615_0001/container_1667121230615_0001_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1667121230615_0001/container_1667121230615_0001_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.103 40380 attempt_1667121230615_0001_m_000000_1 3

[2022-10-30 21:05:29.167]Container killed on request. Exit code is 143
[2022-10-30 21:05:29.325]Container exited with a non-zero exit code 143.

2022-10-30 21:05:36,512 INFO mapreduce.Job: Task Id : attempt_1667121230615_0001_m_000000_2, Status : FAILED
Error: java.lang.ClassNotFoundException: com.flow.Flow
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.p1.PhoneMapper.map(PhoneMapper.java:25)
at com.p1.PhoneMapper.map(PhoneMapper.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2022-10-30 21:05:42,596 INFO mapreduce.Job: map 100% reduce 100%
2022-10-30 21:05:42,610 INFO mapreduce.Job: Job job_1667121230615_0001 failed with state FAILED due to: Task failed task_1667121230615_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

2022-10-30 21:05:42,699 INFO mapreduce.Job: Counters: 13
Job Counters
Failed map tasks=4
Killed reduce tasks=5
Launched map tasks=4
Other local map tasks=3
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=21828
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=21828
Total vcore-milliseconds taken by all map tasks=21828
Total megabyte-milliseconds taken by all map tasks=22351872
Map-Reduce Framework
CPU time spent (ms)=0
Physical memory (bytes) snapshot=0
Virtual memory (bytes) snapshot=0

运行结果及报错内容

Error: java.lang.ClassNotFoundException: com.flow.Flow
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.p1.PhoneMapper.map(PhoneMapper.java:25)
at com.p1.PhoneMapper.map(PhoneMapper.java:1)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

  • 写回答

1条回答 默认 最新

  • AllenGd 大数据领域优质创作者 2022-10-31 16:17
    关注

    [2022-10-30 21:05:17.255]Container [pid=45231,containerID=container_1667121230615_0001_01_000002] is running 255060480B beyond the 'VIRTUAL' memory limit. Current usage: 76.8 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
    从这里来看是机器上运行的container尝试用更多的内存，但是被Nodemanager给中断了，所以需要配置所需内存大小
    解决方法：
    1、在/xxx/hadoop/下的mapred-site.xml中添加如下配置：（其中的数字大小可以根据自己电脑的内存大小情况更改）

    <property>
　　<name>mapreduce.map.memory.mb</name>
　　<value>1536</value>
</property>
<property>
　　<name>mapreduce.map.java.opts</name>
　　<value>-Xmx1024M</value>
</property>
<property>
　　<name>mapreduce.reduce.memory.mb</name>
　　<value>2048</value>
</property>
<property>
　　<name>mapreduce.reduce.java.opts</name>
　　<value>-Xmx2048M</value>
</property>

    2、重启yarn进程即可。
    在hadoop目录下的sbin中：
    stop-yarn.sh
    start-yarn.sh

    评论 编辑记录

报告相同问题？

问题事件

  • 创建了问题 10月30日

悬赏问题

  • ¥15 安卓adb backup备份应用数据失败
  • ¥15 eclipse运行项目时遇到的问题
  • ¥15 关于#c##的问题：最近需要用CAT工具Trados进行一些开发
  • ¥15 南大pa1 小游戏没有界面，并且报了如下错误，尝试过换显卡驱动，但是好像不行
  • ¥15 没有证书，nginx怎么反向代理到只能接受https的公网网站
  • ¥50 成都蓉城足球俱乐部小程序抢票
  • ¥15 yolov7训练自己的数据集
  • ¥15 esp8266与51单片机连接问题(标签-单片机|关键词-串口)（相关搜索：51单片机|单片机|测试代码）
  • ¥15 电力市场出清matlab yalmip kkt 双层优化问题
  • ¥30 ros小车路径规划实现不了，如何解决？(操作系统-ubuntu)