hadoop运行jar包的那些事

Question

提交ja包 放在hadoop上运行报空指针异常太恐怖了，我实在是不会了

[root@master hadoop]# hadoop jar /export/software/clean.jar com.position.clean.CleanMain /JobData/20230321 /JobData/output
2023-03-21 08:28:25,761 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.11.129:8032
0 [main] INFO org.apache.hadoop.yarn.client.RMProxy  - Connecting to ResourceManager at master/192.168.11.129:8032
2023-03-21 08:28:26,630 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1679409515809_0003
869 [main] INFO org.apache.hadoop.mapreduce.JobResourceUploader  - Disabling Erasure Coding for path: /tmp/hadoop-yarn/staging/root/.staging/job_1679409515809_0003
2023-03-21 08:28:26,802 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1041 [Thread-8] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient  - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-21 08:28:27,140 INFO input.FileInputFormat: Total input files to process : 30
1379 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat  - Total input files to process : 30
2023-03-21 08:28:27,175 INFO input.CombineFileInputFormat: DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 4170
1414 [main] INFO org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat  - DEBUG: Terminated node allocation with : CompletedNodes: 3, size left: 4170
2023-03-21 08:28:27,207 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1446 [Thread-11] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient  - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-21 08:28:27,253 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1492 [Thread-13] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient  - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-21 08:28:27,284 INFO mapreduce.JobSubmitter: number of splits:1
1523 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter  - number of splits:1
2023-03-21 08:28:27,463 INFO sasl.SaslDataTransferClient: SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
1702 [Thread-15] INFO org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient  - SASL encryption trust check: localHostTrusted = false, remoteHostTrusted = false
2023-03-21 08:28:27,529 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1679409515809_0003
1768 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter  - Submitting tokens for job: job_1679409515809_0003
2023-03-21 08:28:27,529 INFO mapreduce.JobSubmitter: Executing with tokens: []
1768 [main] INFO org.apache.hadoop.mapreduce.JobSubmitter  - Executing with tokens: []
2023-03-21 08:28:27,858 INFO conf.Configuration: resource-types.xml not found
2097 [main] INFO org.apache.hadoop.conf.Configuration  - resource-types.xml not found
2023-03-21 08:28:27,858 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
2097 [main] INFO org.apache.hadoop.yarn.util.resource.ResourceUtils  - Unable to find 'resource-types.xml'.
2023-03-21 08:28:27,972 INFO impl.YarnClientImpl: Submitted application application_1679409515809_0003
2211 [main] INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl  - Submitted application application_1679409515809_0003
2023-03-21 08:28:28,032 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1679409515809_0003/
2271 [main] INFO org.apache.hadoop.mapreduce.Job  - The url to track the job: http://master:8088/proxy/application_1679409515809_0003/
2023-03-21 08:28:28,033 INFO mapreduce.Job: Running job: job_1679409515809_0003
2272 [main] INFO org.apache.hadoop.mapreduce.Job  - Running job: job_1679409515809_0003
2023-03-21 08:28:35,381 INFO mapreduce.Job: Job job_1679409515809_0003 running in uber mode : false
9620 [main] INFO org.apache.hadoop.mapreduce.Job  - Job job_1679409515809_0003 running in uber mode : false
2023-03-21 08:28:35,384 INFO mapreduce.Job:  map 0% reduce 0%
9623 [main] INFO org.apache.hadoop.mapreduce.Job  -  map 0% reduce 0%
2023-03-21 08:28:43,539 INFO mapreduce.Job: Task Id : attempt_1679409515809_0003_m_000000_0, Status : FAILED
17778 [main] INFO org.apache.hadoop.mapreduce.Job  - Task Id : attempt_1679409515809_0003_m_000000_0, Status : FAILED
Error: java.lang.NullPointerException
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2023-03-21 08:28:49,725 INFO mapreduce.Job: Task Id : attempt_1679409515809_0003_m_000000_1, Status : FAILED
23964 [main] INFO org.apache.hadoop.mapreduce.Job  - Task Id : attempt_1679409515809_0003_m_000000_1, Status : FAILED
Error: java.lang.NullPointerException
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2023-03-21 08:28:54,842 INFO mapreduce.Job: Task Id : attempt_1679409515809_0003_m_000000_2, Status : FAILED
29081 [main] INFO org.apache.hadoop.mapreduce.Job  - Task Id : attempt_1679409515809_0003_m_000000_2, Status : FAILED
Error: java.lang.NullPointerException
    at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:560)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:798)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)

2023-03-21 08:29:00,919 INFO mapreduce.Job:  map 100% reduce 100%
35158 [main] INFO org.apache.hadoop.mapreduce.Job  -  map 100% reduce 100%
2023-03-21 08:29:00,939 INFO mapreduce.Job: Job job_1679409515809_0003 failed with state FAILED due to: Task failed task_1679409515809_0003_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

35178 [main] INFO org.apache.hadoop.mapreduce.Job  - Job job_1679409515809_0003 failed with state FAILED due to: Task failed task_1679409515809_0003_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0 killedMaps:0 killedReduces: 0

2023-03-21 08:29:01,069 INFO mapreduce.Job: Counters: 9
    Job Counters 
        Failed map tasks=4
        Killed reduce tasks=1
        Launched map tasks=4
        Other local map tasks=4
        Total time spent by all maps in occupied slots (ms)=34568
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=17284
        Total vcore-milliseconds taken by all map tasks=17284
        Total megabyte-milliseconds taken by all map tasks=35397632
35308 [main] INFO org.apache.hadoop.mapreduce.Job  - Counters: 9
    Job Counters 
        Failed map tasks=4
        Killed reduce tasks=1
        Launched map tasks=4
        Other local map tasks=4
        Total time spent by all maps in occupied slots (ms)=34568
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=17284
        Total vcore-milliseconds taken by all map tasks=17284
        Total megabyte-milliseconds taken by all map tasks=35397632

Answer 1

1.、程序执行出现Error: java.lang.NullPointerException
空指针异常，确保java程序的正确。变量什么的使用前先实例化声明，不要有数组越界之类的现象。检查程序。
2、执行自己的程序的时候，（各种）报错，请确保一下情况：
前提都是你的程序是正确通过编译的
集群模式下，请把要处理的数据写到HDFS里，并且确保HDFS路径正确
指定执行的jar包的入口类名（我不知道为什么有时候不指定也是可以运行的）
正确的写法类似：hadoop jar myCount.jar myCount input output
3.ssh无法正常通信的问题
4.程序编译问题，各种包没有的情况，请确保你把hadoop目录下 和hadoop/lib目录下的jar包都有引入。