surplus-G 2022-04-29 15:47 采纳率: 33.3%
浏览 410

用hadoop自带的jar包计算圆周率pi报错

想用hadoop中的jar包计算pi值,测试下能不能正常使用,结果一直报错。在网上查阅了其他人的解决办法,修改yarn.xml文件和mapred.xml文件的配置,但都没有用。
报错如下:

Starting Job
22/04/29 15:39:50 INFO client.RMProxy: Connecting to ResourceManager at node1/192.168.88.151:8032
22/04/29 15:39:51 INFO input.FileInputFormat: Total input paths to process : 10
22/04/29 15:39:51 INFO mapreduce.JobSubmitter: number of splits:10
22/04/29 15:39:52 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1651215132796_0004
22/04/29 15:39:52 INFO impl.YarnClientImpl: Submitted application application_1651215132796_0004
22/04/29 15:39:52 INFO mapreduce.Job: The url to track the job: http://node1:8088/proxy/application_1651215132796_0004/
22/04/29 15:39:52 INFO mapreduce.Job: Running job: job_1651215132796_0004
22/04/29 15:39:55 INFO mapreduce.Job: Job job_1651215132796_0004 running in uber mode : false
22/04/29 15:39:55 INFO mapreduce.Job:  map 0% reduce 0%
22/04/29 15:39:55 INFO mapreduce.Job: Job job_1651215132796_0004 failed with state FAILED due to: Application application_1651215132796_0004 failed 2 times due to AM Container for appattempt_1651215132796_0004_000002 exited with  exitCode: 1
For more detailed output, check application tracking page:http://node1:8088/cluster/app/application_1651215132796_0004Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1651215132796_0004_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
        at org.apache.hadoop.util.Shell.run(Shell.java:482)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)


Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
22/04/29 15:39:55 INFO mapreduce.Job: Counters: 0
Job Finished in 5.517 seconds
java.io.FileNotFoundException: File does not exist: hdfs://node1:8020/user/root/QuasiMonteCarlo_1651217985507_1172643346/out/reduce-out
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
        at org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1820)
        at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1843)
        at org.apache.hadoop.examples.QuasiMonteCarlo.estimatePi(QuasiMonteCarlo.java:314)
        at org.apache.hadoop.examples.QuasiMonteCarlo.run(QuasiMonteCarlo.java:355)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.examples.QuasiMonteCarlo.main(QuasiMonteCarlo.java:363)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

希望大家帮我看看到底是哪里出了错,非常感谢!

  • 写回答

2条回答 默认 最新

  • 浩豪 2022-04-29 21:43
    关注

    问题出在HDFS的配置上,你的HDFS并未配置好,可以检查HDFS的相关配置。看看DataNode是否正常启动,HDFS状态是否正常?

    HDFS未配置好,所以才导致程序无法创建和打开输出文件,所以报错。

    评论

报告相同问题?

问题事件

  • 创建了问题 4月29日

悬赏问题

  • ¥15 请教:如何用postman调用本地虚拟机区块链接上的合约?
  • ¥15 为什么使用javacv转封装rtsp为rtmp时出现如下问题:[h264 @ 000000004faf7500]no frame?
  • ¥15 乘性高斯噪声在深度学习网络中的应用
  • ¥15 运筹学排序问题中的在线排序
  • ¥15 关于docker部署flink集成hadoop的yarn,请教个问题 flink启动yarn-session.sh连不上hadoop,这个整了好几天一直不行,求帮忙看一下怎么解决
  • ¥15 深度学习根据CNN网络模型,搭建BP模型并训练MNIST数据集
  • ¥15 C++ 头文件/宏冲突问题解决
  • ¥15 用comsol模拟大气湍流通过底部加热(温度不同)的腔体
  • ¥50 安卓adb backup备份子用户应用数据失败
  • ¥20 有人能用聚类分析帮我分析一下文本内容嘛