hadoop中wordcount使用IKsegmenter分词器后本地编译运行通过，打成jar包后报IKsegmenter类无法找到

Question

使用eclipse在本地上编译运行的时候没有问题，output能够获得正常结果。
但是使用hadoop运行时就会报错，之前没有使用过ecipse，在网上找了半天，export里面的选项来回试了一下也没成功..以下是报错信息：

19/11/19 20:57:39 INFO client.RMProxy: Connecting to ResourceManager at bigdata-senior01.chybinmy.com/192.168.100.10:8032
19/11/19 20:57:40 INFO input.FileInputFormat: Total input files to process : 1
19/11/19 20:57:40 INFO mapreduce.JobSubmitter: number of splits:1
19/11/19 20:57:41 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled
19/11/19 20:57:41 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1574165004078_0005
19/11/19 20:57:41 INFO conf.Configuration: resource-types.xml not found
19/11/19 20:57:41 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
19/11/19 20:57:41 INFO resource.ResourceUtils: Adding resource type - name = memory-mb, units = Mi, type = COUNTABLE
19/11/19 20:57:41 INFO resource.ResourceUtils: Adding resource type - name = vcores, units = , type = COUNTABLE
19/11/19 20:57:42 INFO impl.YarnClientImpl: Submitted application application_1574165004078_0005
19/11/19 20:57:42 INFO mapreduce.Job: The url to track the job: http://bigdata-senior01.chybinmy.com:8088/proxy/application_1574165004078_0005/
19/11/19 20:57:42 INFO mapreduce.Job: Running job: job_1574165004078_0005
19/11/19 20:57:49 INFO mapreduce.Job: Job job_1574165004078_0005 running in uber mode : false
19/11/19 20:57:49 INFO mapreduce.Job:  map 0% reduce 0%
19/11/19 20:57:53 INFO mapreduce.Job: Task Id : attempt_1574165004078_0005_m_000000_0, Status : FAILED
Error: java.lang.ClassNotFoundException: org.wltea.analyzer.core.IKSegmenter
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:36)
    at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)

19/11/19 20:57:58 INFO mapreduce.Job: Task Id : attempt_1574165004078_0005_m_000000_1, Status : FAILED
Error: java.lang.ClassNotFoundException: org.wltea.analyzer.core.IKSegmenter
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:36)
    at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)

19/11/19 20:58:03 INFO mapreduce.Job: Task Id : attempt_1574165004078_0005_m_000000_2, Status : FAILED
Error: java.lang.ClassNotFoundException: org.wltea.analyzer.core.IKSegmenter
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:36)
    at com.example.test.ChineseWordCount$TokenizerMapper.map(ChineseWordCount.java:1)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:177)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:171)

19/11/19 20:58:11 INFO mapreduce.Job:  map 100% reduce 100%
19/11/19 20:58:12 INFO mapreduce.Job: Job job_1574165004078_0005 failed with state FAILED due to: Task failed task_1574165004078_0005_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

19/11/19 20:58:12 INFO mapreduce.Job: Counters: 13
    Job Counters 
        Failed map tasks=4
        Killed reduce tasks=1
        Launched map tasks=4
        Other local map tasks=3
        Data-local map tasks=1
        Total time spent by all maps in occupied slots (ms)=12897
        Total time spent by all reduces in occupied slots (ms)=0
        Total time spent by all map tasks (ms)=12897
        Total vcore-milliseconds taken by all map tasks=12897
        Total megabyte-milliseconds taken by all map tasks=13206528
    Map-Reduce Framework
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
[hadoop@bigdata-senior01 hadoop-2.10.0]$ bin/hdfs dfs -ls /
Found 4 items
drwxr-xr-x   - hadoop supergroup          0 2019-11-19 20:13 /input
drwxr-xr-x   - hadoop supergroup          0 2019-11-19 20:54 /output
drwxr-xr-x   - hadoop supergroup          0 2019-11-19 20:58 /output2
drwx------   - hadoop supergroup          0 2019-11-19 20:16 /tmp

Answer 1

这个错误 java.lang.ClassNotFoundException: org.wltea.analyzer.core.IKSegmenter 表明 Hadoop 无法找到 org.wltea.analyzer.core.IKSegmenter 这个类。这可能是由于缺少相应的依赖库或者没有将依赖库打包到您的 Jar 文件中导致的。

为了解决这个问题，您需要确保在 Hadoop 的类路径中包含了 org.wltea.analyzer.core.IKSegmenter 类所在的依赖库。您可以尝试将相关的 jar 文件添加到 Hadoop 的 classpath 中或者将其包含在您的项目的依赖中，并重新提交作业。如果您使用 Maven 进行构建项目，可以将相关的依赖添加到 pom.xml 文件中。

另外，请确保您的项目中包含了所有需要的依赖，并且在提交作业时将它们一起打包到 Jar 文件中。