weixin_43941983 2021-07-04 00:52 采纳率: 0%
浏览 286

hadoop集群 wordcount 没有输出

菜鸟初学遇到问题，执行以下操作
1、[aaa123@hadoop102 hadoop-3.1.3]$ hadoop fs -mkdir /wcinput
2、[aaa123@hadoop102 hadoop-3.1.3]$ hadoop fs -put wcinput/word.txt /wcinput
3、[aaa123@hadoop102 hadoop-3.1.3]$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.3.jar wordcount /wcinput /wcoutput
然而在集群目录/wcoutput下没有输出信息是什么问题？

2021-07-04 00:28:45,338 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/yubao/.staging/job_1625322038998_0010
2021-07-04 00:28:45,556 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2021-07-04 00:28:46,022 INFO input.FileInputFormat: Total input files to process : 1
2021-07-04 00:28:46,190 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2021-07-04 00:28:46,271 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2021-07-04 00:28:46,325 INFO mapreduce.JobSubmitter: number of splits:1
2021-07-04 00:28:46,604 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2021-07-04 00:28:46,678 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1625322038998_0010
2021-07-04 00:28:46,678 INFO mapreduce.JobSubmitter: Executing with tokens: []
2021-07-04 00:28:47,069 INFO conf.Configuration: resource-types.xml not found
2021-07-04 00:28:47,069 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2021-07-04 00:28:47,203 INFO impl.YarnClientImpl: Submitted application application_1625322038998_0010
2021-07-04 00:28:47,322 INFO mapreduce.Job: The url to track the job: http://hadoop103:8088/proxy/application_1625322038998_0010/
2021-07-04 00:28:47,323 INFO mapreduce.Job: Running job: job_1625322038998_0010
2021-07-04 00:28:58,920 INFO mapreduce.Job: Job job_1625322038998_0010 running in uber mode : false
2021-07-04 00:28:58,923 INFO mapreduce.Job:  map 0% reduce 0%
2021-07-04 00:29:04,025 INFO mapreduce.Job: Task Id : attempt_1625322038998_0010_m_000000_0, Status : FAILED
[2021-07-04 00:29:02.055]Container [pid=6489,containerID=container_1625322038998_0010_01_000002] is running 294070784B beyond the 'VIRTUAL' memory limit. Current usage: 110.9 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1625322038998_0010_01_000002 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 6489 6488 6489 6489 (bash) 0 1 9789440 289 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/yubao/appcache/application_1625322038998_0010/container_1625322038998_0010_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.102 41939 attempt_1625322038998_0010_m_000000_0 2 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000002/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000002/stderr  
    |- 6499 6489 6489 6489 (java) 499 34 2539139072 28104 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/yubao/appcache/application_1625322038998_0010/container_1625322038998_0010_01_000002/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000002 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.102 41939 attempt_1625322038998_0010_m_000000_0 2 

[2021-07-04 00:29:02.182]Container killed on request. Exit code is 143
[2021-07-04 00:29:02.240]Container exited with a non-zero exit code 143. 

2021-07-04 00:29:11,246 INFO mapreduce.Job: Task Id : attempt_1625322038998_0010_m_000000_1, Status : FAILED
[2021-07-04 00:29:10.917]Container [pid=9607,containerID=container_1625322038998_0010_01_000003] is running 305662464B beyond the 'VIRTUAL' memory limit. Current usage: 167.8 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1625322038998_0010_01_000003 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 9607 9606 9607 9607 (bash) 0 1 9789440 288 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/yubao/appcache/application_1625322038998_0010/container_1625322038998_0010_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.102 41939 attempt_1625322038998_0010_m_000000_1 3 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000003/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000003/stderr  
    |- 9618 9607 9607 9607 (java) 640 328 2550730752 42673 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/yubao/appcache/application_1625322038998_0010/container_1625322038998_0010_01_000003/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000003 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.102 41939 attempt_1625322038998_0010_m_000000_1 3 

[2021-07-04 00:29:11.000]Container killed on request. Exit code is 143
[2021-07-04 00:29:11.044]Container exited with a non-zero exit code 143. 

2021-07-04 00:29:17,325 INFO mapreduce.Job: Task Id : attempt_1625322038998_0010_m_000000_2, Status : FAILED
[2021-07-04 00:29:16.767]Container [pid=6677,containerID=container_1625322038998_0010_01_000004] is running 295569920B beyond the 'VIRTUAL' memory limit. Current usage: 114.9 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container.
Dump of the process-tree for container_1625322038998_0010_01_000004 :
    |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
    |- 6687 6677 6677 6677 (java) 516 39 2540638208 29117 /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/yubao/appcache/application_1625322038998_0010/container_1625322038998_0010_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.102 41939 attempt_1625322038998_0010_m_000000_2 4 
    |- 6677 6676 6677 6677 (bash) 0 1 9789440 289 /bin/bash -c /opt/module/jdk1.8.0_212/bin/java -Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN   -Xmx820m -Djava.io.tmpdir=/opt/module/hadoop-3.1.3/data/nm-local-dir/usercache/yubao/appcache/application_1625322038998_0010/container_1625322038998_0010_01_000004/tmp -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000004 -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Dhadoop.root.logfile=syslog org.apache.hadoop.mapred.YarnChild 192.168.10.102 41939 attempt_1625322038998_0010_m_000000_2 4 1>/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000004/stdout 2>/opt/module/hadoop-3.1.3/logs/userlogs/application_1625322038998_0010/container_1625322038998_0010_01_000004/stderr  

[2021-07-04 00:29:16.899]Container killed on request. Exit code is 143
[2021-07-04 00:29:16.917]Container exited with a non-zero exit code 143. 

2021-07-04 00:29:24,392 INFO mapreduce.Job:  map 100% reduce 100%
2021-07-04 00:29:25,416 INFO mapreduce.Job: Job job_1625322038998_0010 failed with state FAILED due to: Task failed task_1625322038998_0010_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

2021-07-04 00:29:25,549 INFO mapreduce.Job: Counters: 13
    Job Counters 
        Failed map tasks=4
        Killed reduce tasks=1
        Launched map tasks=4
        Other local map tasks=3
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=17739
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=17739
        Total vcore-milliseconds taken by all map tasks=17739
        Total megabyte-milliseconds taken by all map tasks=18164736
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
  • 写回答

3条回答 默认 最新

  • weixin_43941983 2021-08-22 10:15
    关注

    关键点是日志的这条 110.9 MB of 1 GB physical memory used; 2.4 GB of 2.1 GB virtual memory used. Killing container. yarn虚拟运行内存不够。
    解决方法：增加默认大小

    <property>
       <name>yarn.nodemanager.vmem-pmem-ratio</name>
       <value>5</value>
</property>
    评论

报告相同问题？

悬赏问题

  • ¥15 python怎么在已有视频文件后添加新帧
  • ¥20 虚幻UE引擎如何让多个同一个蓝图的NPC执行一样的动画,
  • ¥15 fluent里模拟降膜反应的UDF编写
  • ¥15 MYSQL 多表拼接link
  • ¥15 关于某款2.13寸墨水屏的问题
  • ¥15 obsidian的中文层级自动编号
  • ¥15 同一个网口一个电脑连接有网，另一个电脑连接没网
  • ¥15 神经网络模型一直不能上GPU
  • ¥15 pyqt怎么把滑块和输入框相互绑定，求解决！
  • ¥20 wpf datagrid单元闪烁效果失灵