yzdlkxkl 2023-03-22 00:06 采纳率: 88.9%
浏览 15
已结题

hadoop运行jar包的那些事

提交ja包 放在hadoop上运行报空指针异常太恐怖了,我实在是不会了


[root@master hadoop]# hadoop jar /export/software/clean.jar com.position.clean.CleanMain /JobData/20230321 /JobData/output
2023-03-21 08:28:25,761 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.11.129:8032
0 [main] INFO org.apache.hadoop.yarn.client.RMProxy  - Connecting to ResourceManager at master/192.168.11.129:8032
2023-03-21 08:28:26,630 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1679409515809_0003
869 [main] INFO org.apache.hadoop.mapreduce.JobResourceUploader  - Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1679409515809_0003
2023-03-21 08:28:26,802 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1041 [Thread-8] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient  - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-21 08:28:27,140 INFO input.FileInputFormat: Total input files to process : 30
1379 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat  - Total input files to process : 30
2023-03-21 08:28:27,175 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 4170
1414 [main] INFO org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat  - DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 4170
2023-03-21 08:28:27,207 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1446 [Thread-11] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient  - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-21 08:28:27,253 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1492 [Thread-13] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient  - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-21 08:28:27,284 INFO mapreduce.JobSubmitter: number of splits:1
1523 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter  - number of splits:1
2023-03-21 08:28:27,463 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1702 [Thread-15] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient  - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-21 08:28:27,529 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1679409515809_0003
1768 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter  - Submitting tokens for job: job_1679409515809_0003
2023-03-21 08:28:27,529 INFO mapreduce.JobSubmitter: Executing with tokens: []
1768 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter  - Executing with tokens: []
2023-03-21 08:28:27,858 INFO conf.Configuration: resource-types.xml not found
2097 [main] INFO org.apache.hadoop.conf.Configuration  - resource-types.xml not found
2023-03-21 08:28:27,858 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2097 [main] INFO org.apache.hadoop.yarn.util.resource.ResourceUtils  - Unable to find 'resource-types.xml'.
2023-03-21 08:28:27,972 INFO impl.YarnClientImpl: Submitted application application_1679409515809_0003
2211 [main] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl  - Submitted application application_1679409515809_0003
2023-03-21 08:28:28,032 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1679409515809_0003/
2271 [main] INFO org.apache.hadoop.mapreduce.Job  - The url to track the job: http://master:8088/proxy/application_1679409515809_0003/
2023-03-21 08:28:28,033 INFO mapreduce.Job: Running job: job_1679409515809_0003
2272 [main] INFO org.apache.hadoop.mapreduce.Job  - Running job: job_1679409515809_0003
2023-03-21 08:28:35,381 INFO mapreduce.Job: Job job_1679409515809_0003 running in uber mode : false
9620 [main] INFO org.apache.hadoop.mapreduce.Job  - Job job_1679409515809_0003 running in uber mode : false
2023-03-21 08:28:35,384 INFO mapreduce.Job:  map 0% reduce 0%
9623 [main] INFO org.apache.hadoop.mapreduce.Job  -  map 0% reduce 0%
2023-03-21 08:28:43,539 INFO mapreduce.Job: Task Id : attempt_1679409515809_0003_m_000000_0, Status : FAILED
17778 [main] INFO org.apache.hadoop.mapreduce.Job  - Task Id : attempt_1679409515809_0003_m_000000_0, Status : FAILED
Error: java.lang.NullPointerException
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2023-03-21 08:28:49,725 INFO mapreduce.Job: Task Id : attempt_1679409515809_0003_m_000000_1, Status : FAILED
23964 [main] INFO org.apache.hadoop.mapreduce.Job  - Task Id : attempt_1679409515809_0003_m_000000_1, Status : FAILED
Error: java.lang.NullPointerException
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2023-03-21 08:28:54,842 INFO mapreduce.Job: Task Id : attempt_1679409515809_0003_m_000000_2, Status : FAILED
29081 [main] INFO org.apache.hadoop.mapreduce.Job  - Task Id : attempt_1679409515809_0003_m_000000_2, Status : FAILED
Error: java.lang.NullPointerException
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2023-03-21 08:29:00,919 INFO mapreduce.Job:  map 100% reduce 100%
35158 [main] INFO org.apache.hadoop.mapreduce.Job  -  map 100% reduce 100%
2023-03-21 08:29:00,939 INFO mapreduce.Job: Job job_1679409515809_0003 failed with state FAILED due to: Task failed task_1679409515809_0003_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

35178 [main] INFO org.apache.hadoop.mapreduce.Job  - Job job_1679409515809_0003 failed with state FAILED due to: Task failed task_1679409515809_0003_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

2023-03-21 08:29:01,069 INFO mapreduce.Job: Counters: 9
    Job Counters 
        Failed map tasks=4
        Killed reduce tasks=1
        Launched map tasks=4
        Other local map tasks=4
        Total time spent by all maps in occupied slots (ms)=34568
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=17284
        Total vcore-milliseconds taken by all map tasks=17284
        Total megabyte-milliseconds taken by all map tasks=35397632
35308 [main] INFO org.apache.hadoop.mapreduce.Job  - Counters: 9
    Job Counters 
        Failed map tasks=4
        Killed reduce tasks=1
        Launched map tasks=4
        Other local map tasks=4
        Total time spent by all maps in occupied slots (ms)=34568
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=17284
        Total vcore-milliseconds taken by all map tasks=17284
        Total megabyte-milliseconds taken by all map tasks=35397632

  • 写回答

2条回答 默认 最新

  • AllenGd 大数据领域优质创作者 2023-03-22 09:29
    关注

    1.、程序执行出现Error: java.lang.NullPointerException
    空指针异常,确保java程序的正确。变量什么的使用前先实例化声明,不要有数组越界之类的现象。检查程序。
    2、执行自己的程序的时候,(各种)报错,请确保一下情况:
    前提都是你的程序是正确通过编译的
    集群模式下,请把要处理的数据写到HDFS里,并且确保HDFS路径正确
    指定执行的jar包的入口类名(我不知道为什么有时候不指定也是可以运行的)
    正确的写法类似:hadoop jar myCount.jar myCount input output
    3.ssh无法正常通信的问题
    4.程序编译问题,各种包没有的情况,请确保你把hadoop目录下 和hadoop/lib目录下的jar包都有引入。

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论
查看更多回答(1条)

报告相同问题?

问题事件

  • 系统已结题 3月30日
  • 已采纳回答 3月22日
  • 创建了问题 3月22日

悬赏问题

  • ¥65 LineageOs-21.0系统编译问题
  • ¥30 关于#c++#的问题,请各位专家解答!
  • ¥15 App的会员连续扣费
  • ¥15 不同数据类型的特征融合应该怎么做
  • ¥15 用proteus软件设计一个基于8086微处理器的简易温度计
  • ¥15 用联想小新14Pro
  • ¥15 multisim中关于74ls192n和DSWPK开关仿真图分析(减法计数器)
  • ¥15 w3wp,exe 中发生未处理的 Microsoft ,NETFramework 异常。
  • ¥20 C51单片机程序及仿真(加减器)
  • ¥15 AQWA | 水动力分析 二阶波浪力