job.waitForCompletion(true) 报空指针错误

在本地运行一个简单的 mapreduce 在提交job是报空指针异常
是什么情况 本机是 mac电脑

图片说明

Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
其他相关推荐
hadoop 设置输出文件格式,job.setOutputFormatClass(SequenceFileOutputFormat.class); 报错

hadoop 2.9.2 版本 设置输出输出文件格式时,可以 job.setOutputFormatClass(TextOutputFormat.class); // 默认的输出组件 但是,job.setOutputFormatClass(SequenceFileOutputFormat.class); 这样时,会报错。看其他人的博客,可以这样设置。 错误提示: The method setOutputFormatClass(Class<? extends OutputFormat>) in the type Job is not applicable for the arguments (Class<SequenceFileOutputFormat>) ``` package cn.edu360.mr.indexSequance; import java.io.File; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.mapred.SequenceFileOutputFormat; import org.apache.hadoop.mapreduce.InputSplit; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.FileSplit; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; /** * 构建倒排索引,产生结果; hello a.txt->3 b.txt->4 * * 分两步, * 一 * map: 产生K: 单词+文档 V:次数(1) * reduce: 产生 K:单词+文档 V:总次数 * *二 * map: 产生 K:单词 V:文档 在文档中出现的次数 * reduce : 输出 单词 文档, 文档中出现的次数 * * * @author Administrator * */ public class IndexStep1 { /** * 分割单词, * 输出 单词+文档 次数 1 * @author Administrator * */ public static class IndexMapper1 extends Mapper<LongWritable, Text, Text, IntWritable> { @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { // 获取maptask所处理的数据任务上下文信息: 文件路径, 偏移量范围 // 若访问数据为数据库,则为: 库名, 表名, 行范围 InputSplit inputSplit = context.getInputSplit(); // InputSplit为抽象类 FileSplit fileSplit = (FileSplit) inputSplit; // 强转inputSplit为FileSplit类型, FileSplit为InputSplit的实现类,针对文件 String filename = fileSplit.getPath().getName(); // 获取文件名 String line = value.toString(); String[] words = line.split(" "); for (String word : words) { word = format(word); if (word.length() >2) { context.write(new Text(word + "-" + filename), new IntWritable(1)); } } } } /** * 统计文档+单词 * * 输出: K:文档+单词 V: 单词在该文档中出现的总次数 * @author Administrator * */ public static class IndexReduce1 extends Reducer<Text, IntWritable, Text, IntWritable> { @Override protected void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int count = 0; for (IntWritable value : values) { count = count + value.get(); } context.write(key, new IntWritable(count)); } } public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { Configuration configuration = new Configuration(); JobConf jobConf = new JobConf(configuration); Job job = Job.getInstance(configuration); job.setMapperClass(IndexMapper1.class); job.setReducerClass(IndexReduce1.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(IntWritable.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); File file = new File("d:\\testOut"); if (file.exists()) { deleteDir(file); } job.setOutputFormatClass(TextOutputFormat.class); // 默认的输出组件 job.setOutputFormatClass(SequenceFileOutputFormat.class); FileInputFormat.setInputPaths(job, new Path("d:\\test1")); FileOutputFormat.setOutputPath(job, new Path("d:\\testOut")); job.waitForCompletion(true); } // 使用正则表达式去除标点符号,大写转小写 public static String format(String s) { String str = s.replaceAll("\\pP|\\pS", ""); return str.toLowerCase(); // 大写转小写 } /** * 递归删除文件夹及文件下的所有文件 * @param dir * @return */ private static boolean deleteDir(File dir) { if (dir.isDirectory()) { String[] children = dir.list(); // 递归删除目录中的子目录下 for (int i=0; i<children.length; i++) { boolean success = deleteDir(new File(dir, children[i])); if (!success) { return false; } } } // 目录此时为空,可以删除 return dir.delete(); } } ```

运行mapredurce出现Method threw 'java.lang.IllegalStateException' exception. Cannot evaluate org.apache.hadoop.mapreduce.Job.toString()

执行下述代码后在,创建job后会有上述异常,但是可以执行到最后,但是job没有提交上去执行,在历史里也看不到有执行记录求帮助新手o(╥﹏╥)o。 package MapReducer; import com.sun.org.apache.bcel.internal.generic.RETURN; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; import java.io.File; import java.io.IOException; import java.net.URI; import java.net.URISyntaxException; import java.util.StringTokenizer; /** * @Describe MapReducer第一个读取文档并计数 * @Author zhanglei * @Date 2019/11/18 22:53 **/ public class WordCountApp extends Configured implements Tool { public int run(String[] strings) throws Exception { String input_path="hdfs://192.168.91.130:8020/data/wc.txt"; String output_path="hdfs://192.168.91.130:8020/data/outputwc"; Configuration configuration = getConf(); final FileSystem fileSystem = FileSystem.get(new URI(input_path),configuration); if(fileSystem.exists(new Path(output_path))){ fileSystem.delete(new Path(output_path),true); } Job job = Job.getInstance(configuration,"WordCountApp"); job.setJarByClass(WordCountApp.class); job.setMapperClass(WordCountMapper.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setReducerClass(WordCountReducer.class); job.setInputFormatClass(TextInputFormat.class); Path inpath = new Path(input_path); FileInputFormat.addInputPath(job,inpath); job.setOutputFormatClass(TextOutputFormat.class); Path outpath = new Path(output_path); FileOutputFormat.setOutputPath(job,outpath); return job.waitForCompletion(true) ? 0:1; } //继承 public static class WordCountMapper extends Mapper<Object,Text,Text,IntWritable>{ private final static IntWritable one= new IntWritable(1); private Text word = new Text(); public void map(Object key,Text value,Context context) throws IOException, InterruptedException { Text t = value; StringTokenizer itr = new StringTokenizer(value.toString()); while(itr.hasMoreTokens()){ word.set(itr.nextToken()); context.write(word,one); } } } public static class WordCountReducer extends Reducer<Object,Text,Text,IntWritable>{ private final static IntWritable res= new IntWritable(1); public void reduce(Text key,Iterable<IntWritable> values,Context context) throws IOException, InterruptedException { int sum = 0; for(IntWritable val:values){ sum+=val.get(); } res.set(sum); context.write(key,res); } } public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new WordCountApp(), args); System.exit(exitCode); } }

用eclipse提交任务时卡在running job,但用命令提交时正常,有没有遇到这种问题啊?

在eclipse中dubug一直显示下面的: ``` ``` ``` ``` ``` [DEBUG][2019/03/07 17:57:10646][org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1171)] IPC Client (511717113) connection to bigdata01/192.168.50.131:8032 from root got value #58 [DEBUG][2019/03/07 17:57:10646][org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:248)] Call: getApplicationReport took 3ms [DEBUG][2019/03/07 17:57:11648][org.apache.hadoop.security.UserGroupInformation.logPrivilegedAction(UserGroupInformation.java:1863)] PrivilegedAction as:root (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320) [DEBUG][2019/03/07 17:57:11649][org.apache.hadoop.ipc.Client$Connection$3.run(Client.java:1117)] IPC Client (511717113) connection to bigdata01/192.168.50.131:8032 from root sending #59 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport [DEBUG][2019/03/07 17:57:11652][org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1171)] IPC Client (511717113) connection to bigdata01/192.168.50.131:8032 from root got value #59 [DEBUG][2019/03/07 17:57:11653][org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:248)] Call: getApplicationReport took 4ms [DEBUG][2019/03/07 17:57:11654][org.apache.hadoop.security.UserGroupInformation.logPrivilegedAction(UserGroupInformation.java:1863)] PrivilegedAction as:root (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:320) [DEBUG][2019/03/07 17:57:11655][org.apache.hadoop.ipc.Client$Connection$3.run(Client.java:1117)] IPC Client (511717113) connection to bigdata01/192.168.50.131:8032 from root sending #60 org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport [DEBUG][2019/03/07 17:57:11657][org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1171)] IPC Client (511717113) connection to bigdata01/192.168.50.131:8032 from root got value #60 ``` ` 在web界面一直显示:![图片说明](https://img-ask.csdn.net/upload/201903/07/1551952935_380051.jpg) 在yarn-root-resourcemanager-hadoop01.log 中的信息: ![图片说明](https://img-ask.csdn.net/upload/201903/08/1552007975_434091.png)

org.apache.hadoop.mapred.LocalJobRunner这个类在那个包里?

我在用sqoop1的javaapi操作,但是一执行命令就会报这个错,hadoop集群并不在运行程序的机器上,我是缺少这个类么,我翻了一般依赖里面确实没有 ``` Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.mapred.LocalJobRunner.<init>(Lorg/apache/hadoop/conf/Configuration;)V at org.apache.hadoop.mapred.LocalClientProtocolProvider.create(LocalClientProtocolProvider.java:42) at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:95) at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82) at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75) at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1260) at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1256) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866) at org.apache.hadoop.mapreduce.Job.connect(Job.java:1255) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1284) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at org.apache.sqoop.mapreduce.ExportJobBase.doSubmitJob(ExportJobBase.java:322) at org.apache.sqoop.mapreduce.ExportJobBase.runJob(ExportJobBase.java:299) at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:440) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at com.mshuoke.datagw.impl.sqoop.SqoopTest.main(SqoopTest.java:52) 09:55:47.069 [Thread-4] DEBUG org.apache.hadoop.util.ShutdownHookManager - ShutdownHookManger complete shutdown. ```

Hadoop自定义分组和多ReductTask出现异常

我现在有三个节点 程序在windows下编写,并将Job提交到了集群的Yarn上去执行,出现异常.但是在Linux下使用Hadoop jar 执行是可以的.之前在执行WordCount和其他小程序时候, 并没有出错,我认为错误原因在于这个ReductTask.请大牛指导一下.万分感谢.. ``` 2015-12-04 15:33:43,100 INFO [main] client.RMProxy (RMProxy.java:createRMProxy(92)) - Connecting to ResourceManager at hadoop01/10.5.110.250:8032 2015-12-04 15:33:43,458 WARN [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(150)) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2015-12-04 15:33:43,478 WARN [main] mapreduce.JobSubmitter (JobSubmitter.java:copyAndConfigureFiles(259)) - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2015-12-04 15:33:43,525 INFO [main] input.FileInputFormat (FileInputFormat.java:listStatus(280)) - Total input paths to process : 1 2015-12-04 15:33:43,573 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:submitJobInternal(396)) - number of splits:1 2015-12-04 15:33:43,655 INFO [main] mapreduce.JobSubmitter (JobSubmitter.java:printTokens(479)) - Submitting tokens for job: job_1449213919153_0002 2015-12-04 15:33:43,744 INFO [main] mapred.YARNRunner (YARNRunner.java:createApplicationSubmissionContext(369)) - Job jar is not present. Not adding any jar to the list of resources. 2015-12-04 15:33:43,778 INFO [main] impl.YarnClientImpl (YarnClientImpl.java:submitApplication(204)) - Submitted application application_1449213919153_0002 2015-12-04 15:33:43,807 INFO [main] mapreduce.Job (Job.java:submit(1289)) - The url to track the job: http://hadoop01:8088/proxy/application_1449213919153_0002/ 2015-12-04 15:33:43,808 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1334)) - Running job: job_1449213919153_0002 2015-12-04 15:33:46,823 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1355)) - Job job_1449213919153_0002 running in uber mode : false 2015-12-04 15:33:46,825 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1362)) - map 0% reduce 0% 2015-12-04 15:33:46,833 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1375)) - Job job_1449213919153_0002 failed with state FAILED due to: Application application_1449213919153_0002 failed 2 times due to AM Container for appattempt_1449213919153_0002_000002 exited with exitCode: -1000 due to: File file:/tmp/hadoop-yarn/staging/lixiwei/.staging/job_1449213919153_0002/job.splitmetainfo does not exist .Failing this attempt.. Failing the application. 2015-12-04 15:33:46,861 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - Counters: 0 ``` 程序如下: ``` public class FlowSumArea { public static class FlowSumAreaMapper extends Mapper<LongWritable, Text, Text, FlowBean> { @Override protected void map(LongWritable key, Text value, Mapper<LongWritable, Text, Text, FlowBean>.Context context) throws IOException, InterruptedException { String line = value.toString(); String[] fields = StringUtils.split(line, "\t"); String phoneNo = fields[1]; long upFlow = Long.parseLong(fields[7]); long downFLow = Long.parseLong(fields[8]); context.write(new Text(phoneNo), new FlowBean(phoneNo, upFlow, downFLow)); } } public static class FlowSumAreaReducer extends Reducer<Text, FlowBean, Text, FlowBean> { @Override protected void reduce(Text key, Iterable<FlowBean> values, Reducer<Text, FlowBean, Text, FlowBean>.Context context) throws IOException, InterruptedException { long upFlowCounter = 0; long downFlowCounter = 0; for (FlowBean bean : values) { upFlowCounter += bean.getUpFlow(); downFlowCounter += bean.getDownFlow(); } context.write(key, new FlowBean(key.toString(), upFlowCounter, downFlowCounter)); } } public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { // 1.获取配置文件 Configuration conf = new Configuration(); // 2.设置Job Job job = Job.getInstance(); job.setJarByClass(FlowSumArea.class); job.setMapperClass(FlowSumAreaMapper.class); job.setReducerClass(FlowSumAreaReducer.class); job.setPartitionerClass(AreaPartitioner.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(FlowBean.class); // 设置Reduce的任务并发数,应该跟分组的数量保持一致 job.setNumReduceTasks(6); // 3.设置输入输出路径 FileInputFormat.setInputPaths(job, new Path("C:\\Users\\51195\\Desktop\\flow\\flowarea\\srcdata")); FileOutputFormat.setOutputPath(job, new Path("C:\\Users\\51195\\Desktop\\flow\\flowarea\\outputdata6")); // FileInputFormat.setInputPaths(job, new Path(args[0])); // FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true)?0:1); } } ``` 这个是分组程序 ``` public class AreaPartitioner<KEY, VALUE> extends Partitioner<KEY, VALUE>{ private static HashMap<String,Integer> areaMap = new HashMap<>(); static{ areaMap.put("135", 0); areaMap.put("136", 1); areaMap.put("137", 2); areaMap.put("138", 3); areaMap.put("139", 4); } @Override public int getPartition(KEY key, VALUE value, int numPartitions) { //从key中拿到手机号,查询手机归属地字典,不同的省份返回不同的组号 int areaCoder = areaMap.get(key.toString().substring(0, 3))==null?5:areaMap.get(key.toString().substring(0, 3)); return areaCoder; } } ```

有大神知道这个job的异常吗

![图片说明](https://img-ask.csdn.net/upload/201907/10/1562752753_837230.jpg) 已知为类转换异常,但是具体的异常未知,发现强转的类是在一个jar包下

在Linux hadoop环境中运行sh脚本,报异常java.io.IOException: No FileSystem for scheme: E

jar包是从eclipse中导出的,代码没有问题,在Windows下可以正确运行。 在Linux下用脚本运行,出现问题。 脚本内容: ![图片说明](https://img-ask.csdn.net/upload/201812/20/1545295208_386974.png) 运行后报错: ![图片说明](https://img-ask.csdn.net/upload/201812/20/1545295282_304754.jpg) jar包main方法: ``` public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); conf.set("fs.hdfs.impl", "org.apache.hadoop.hdfs.DistributedFileSystem"); Job job = Job.getInstance(conf); job.setJarByClass(AppLogDataClean.class); job.setMapperClass(AppLogDataCleanMapper.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(NullWritable.class); job.setNumReduceTasks(0); // 避免生成默认的part-m-00000等文件,因为,数据已经交给MultipleOutputs输出了 LazyOutputFormat.setOutputFormatClass(job, TextOutputFormat.class); FileInputFormat.setInputPaths(job, new Path("E:/educ/infile/20170816new")); FileOutputFormat.setOutputPath(job, new Path("E:/educ/outfile/LogTest2/clean")); boolean res = job.waitForCompletion(true); System.exit(res ? 0 : 1); } ``` 配置文件应该也都没问题,之前所有业务都能正常操作,所以现在应该怎么解决? 求助大神

求救高手,Mapreduce导入数据到Hadoop报ClassNotFoundException

最近在用Mapreduce连Hadoop,出现各类问题。请高手答疑。 环境: hadoop:2.7.0 Hbase:1.0.1.1 刚开始的时候报:HBaseConfiguration 找不到,百度之,说将 hbase的lib下的jar复制到hadoop的lib下 复制之,无果,找各类参考资料修改hadoop参数,都还是报异常。 最后无奈,只能修改 hadoop-env.sh,将 hbase 的lib加入到 classpath下。 ![图片说明](https://img-ask.csdn.net/upload/201509/14/1442242731_955123.jpg) 最后终于不报这个异常。 可是接着更加无解的事情发生了。 ![图片说明](https://img-ask.csdn.net/upload/201509/14/1442242933_762859.jpg) 报出各类我自己定义的类找不到。 网上找遍了所有的贴,没找到答案。 ``` public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); conf = HBaseConfiguration.create(conf); if (args.length != 6) { System.err.println("Usage: MroFormat <in-mro> <in-xdr> <sample tbname> <event tbname>"); System.exit(2); } //makeConfig(conf, args); String inpath1 = args[0]; String inpath2 = args[1]; Job job = Job.getInstance(conf,"MyTest"); job.setNumReduceTasks(40); job.setJarByClass(Main.class); job.setReducerClass(ReduceDeal.MroFormatReducer.class); //job.setReducerClass(ReduceDeal.TestReducer.class); job.setSortComparatorClass(MapDeal.SortKeyComparator.class); job.setPartitionerClass(MapDeal.CellIDPartitioner.class); job.setGroupingComparatorClass(MapDeal.SortKeyGroupComparator.class); job.setMapOutputKeyClass(CellTimeKeyPare.class); job.setMapOutputValueClass(Text.class); MultipleInputs.addInputPath(job, new Path(inpath1), KeyValueTextInputFormat.class, MapDeal.MroMapper.class); MultipleInputs.addInputPath(job, new Path(inpath2), TextInputFormat.class, MapDeal.XdrMapper.class); job.setOutputFormatClass(MultiTableOutputFormat.class); //job.setOutputFormatClass(NullOutputFormat.class); //LOG.info(job.getPartitionerClass().getName()); //TableMapReduceUtil.addDependencyJars(job); //TableMapReduceUtil.addDependencyJars(job.getConfiguration()); //TableMapReduceUtil.initTableReducerJob("tab1", ReduceDeal.MroFormatReducer.class, job); System.exit(job.waitForCompletion(true) ? 0 : 1); } ``` 在本机的win7环境下,代码能跑过,但打包放到服务器就报错,找不着类。去掉了“conf = HBaseConfiguration.create(conf);”这段代码后,下面的 处理类就可以找到。请大牛们帮忙看看,非常感谢

hbase mapreduce 报错 java.lang.NullPointerException

http://bbs.csdn.net/topics/390865764 这篇文章出错相似,求教大牛们 2017-09-15 23:19:15 [WARN]-[] Your hostname, admin-PC resolves to a loopback/non-reachable address: fe80:0:0:0:0:5efe:c0a8:164%23, but we couldn't find any external IP address! 2017-09-15 23:19:15 [INFO]-[org.apache.hadoop.conf.Configuration.deprecation] session.id is deprecated. Instead, use dfs.metrics.session-id 2017-09-15 23:19:15 [INFO]-[org.apache.hadoop.metrics.jvm.JvmMetrics] Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread "main" java.lang.NullPointerException at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010) at org.apache.hadoop.util.Shell.runCommand(Shell.java:487) at org.apache.hadoop.util.Shell.run(Shell.java:460) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:720) at org.apache.hadoop.util.Shell.execCommand(Shell.java:813) at org.apache.hadoop.util.Shell.execCommand(Shell.java:796) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:656) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:444) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:308) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:147) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325) at TestOnlyMapper.main(TestOnlyMapper.java:35) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) ----------------------分割线 代码------------------------------------------------- import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.Cell; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.client.Result; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.io.ImmutableBytesWritable; import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; import org.apache.hadoop.hbase.mapreduce.TableMapper; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.output.NullOutputFormat; import java.io.IOException; /** * Created by admin on 2017/9/15. */ public class TestOnlyMapper { public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { Configuration conf = HBaseConfiguration.create(); conf.set("hbase.rootdir","hdfs://hadoop.master:8020/hdfs/hbase"); conf.set("hbase.zookeeper.quorum","hadoop.master,hadoop.slave11,hadoop.slave12"); conf.set("hbase.zookeeper.property.clientPort","2181"); Job job= new Job(conf,"test"); job.setJarByClass(TestOnlyMapper.class); Scan scan = new Scan(); job.setMapSpeculativeExecution(false); job.setReduceSpeculativeExecution(false); TableMapReduceUtil.initTableMapperJob("test11",scan,OMapper.class,null,null,job); job.setOutputFormatClass(NullOutputFormat.class); job.waitForCompletion(true); } } class OMapper extends TableMapper<Text, LongWritable> { @Override protected void map(ImmutableBytesWritable key, Result value, Context context) throws IOException, InterruptedException { for(Cell cell:value.listCells()) { System.out.println("---------------------"); System.out.println("cell.getQualifier()= "+cell.getQualifier().toString()); System.out.println("---------------------"); } } }

hadoop 报java.lang.InstantiationException

package mapreduce; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.partition.HashPartitioner; public class WordCountApp { static final String INPUT_PATH = "hdfs://chaoren:9000/hello"; static final String OUTPUT_PATH = "hdfs://chaoren:9000/hello_statics"; public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, WordCountApp.class.getSimpleName()); //1。1输入的目录在哪里 FileInputFormat.setInputPaths(job, INPUT_PATH); //指定对输入的数据进行格式化处理 job.setInputFormatClass(TextInputFormat.class); //1.2指定自定义的mapper类 job.setMapperClass(MyMapper.class); //指定map输出的<key,value>类型 job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(LongWritable.class); //1.3 分区 job.setPartitionerClass(HashPartitioner.class); job.setNumReduceTasks(1); //1.4 TODO 排序分组 //1.5 TODO <可选>规约 //2.2指定自定义的Reducer类 job.setReducerClass(MyReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(LongWritable.class); //2.2指定输出路径 FileOutputFormat.setOutputPath(job, new Path(OUTPUT_PATH)); //2.3指定输出的格式化类 job.setOutputFormatClass(FileOutputFormat.class); //把作业提交jobTracker运行 job.waitForCompletion(true); } /** * KEYIN 即k1 表示每一行的起始位置<偏移量> * VALUEIN 即v1 表示每一行的文本内容 * KEYOUT 即k1 表示每一行中的单词 * VALUEOUT 即v1 表示每一行中,每个单词出现的次数 * @author Administrator * */ static class MyMapper extends Mapper<LongWritable, Text, Text, LongWritable> { @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String[] splited = value.toString().split("\t"); for(String word : splited) { context.write(new Text(word), new LongWritable(1)); } } } /** * KEYIN 即k2 表示每一行中每个单词 * VALUEIN 即v2 表示每一行中每个单词出现的次数 * KEYOUT 即k3 表示整个文件中的不同单词 * VALUEOUT 即v3 表示整个文件中的不同单词的出现总数 * @author Administrator * */ static class MyReducer extends Reducer<Text,LongWritable, Text, LongWritable>{ protected void reduce(Text k2, Iterable<LongWritable> v2s, Context context) throws IOException, InterruptedException { long sum = 0l; for(LongWritable v2 : v2s){ sum+=v2.get(); } context.write(k2, new LongWritable(sum)); } } }

Hadoop2.x ,一直报无法初始化对象,这个是什么原因啊

15/07/25 03:54:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 15/07/25 03:54:31 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 15/07/25 03:54:32 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 15/07/25 03:54:32 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/root/.staging/job_1437805442648_0002 Exception in thread "main" java.lang.RuntimeException: java.lang.InstantiationException at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:131) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:594) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314) at com.baizhi.myhadoop.TestCombineFileInputFormat.main(TestCombineFileInputFormat.java:66) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: java.lang.InstantiationException at sun.reflect.InstantiationExceptionConstructorAccessorImpl.newInstance(InstantiationExceptionConstructorAccessorImpl.java:48) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:129) ... 17 more

eclipse在类中找不到 main 方法, 请将 main 方法定义为: public static void main

在类 LazyMapReduce.LazyMapReduce 中找不到 main 方法, 请将 main 方法定义为: public static void main(String[] args) 否则 JavaFX 应用程序类必须扩展javafx.application.Application 求大神解答一下问题出在哪里了 package LazyMapReduce; import java.io.IOException; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapred.TextOutputFormat; import org.apache.hadoop.mapred.lib.HashPartitioner; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.util.GenericOptionsParser; import java.io.IOException; import java.util.HashMap; import java.util.Map; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Partitioner; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class LazyMapReduce { public static void main(String args[]) throws IOException,InterruptedException,ClassNotFoundException { Configuration conf = new Configuration(); String[] iargs=new String[]{"hdfs://master:9000/Lazy_input","hdfs://master:9000/Lazy_input/output"}; String[] otherArgs = new GenericOptionsParser(conf,iargs).getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("usage:intertedindex <in> <out>"); System.exit(2); } Job job = new Job(conf, "LazyMapReduce"); job.setJarByClass(LazyMapReduce.class); job.setInputFormatClass(TextInputFormat.class); job.setMapperClass(Mapper.class); job.setMapOutputKeyClass(LongWritable.class); job.setMapOutputValueClass(Text.class); job.setJarByClass(HashPartitioner.class); job.setReducerClass(Reducer.class); job.setOutputKeyClass(LongWritable.class); job.setOutputValueClass(Text.class); job.setOutputKeyClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); }}

hadoop的第一练习,wordcount出错了。。。不知道这是怎么回事求指教。。

有没有hadoop的前辈帮看看怎么回事,编译啊打包啊,都没错,就在最后用hadoop jar只想的时候报了错,我也看不太懂是哪里出的问题。源代码是在网上找的。 ![图片说明](https://img-ask.csdn.net/upload/201602/18/1455788004_97286.png) ![图片说明](https://img-ask.csdn.net/upload/201602/18/1455788026_686440.png) ``` import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; public class WordCount { public static class WordCountMap extends Mapper<LongWritable, Text, Text, IntWritable> { private final IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString(); StringTokenizer token = new StringTokenizer(line); while (token.hasMoreTokens()) { word.set(token.nextToken()); context.write(word, one); } } } public static class WordCountReduce extends Reducer<Text, IntWritable, Text, IntWritable> { public void reduce(Text key, Iterable<IntWritable> values, Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } context.write(key, new IntWritable(sum)); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf); job.setJarByClass(WordCount.class); job.setJobName("wordcount"); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setMapperClass(WordCountMap.class); job.setReducerClass(WordCountReduce.class); job.setInputFormatClass(TextInputFormat.class); job.setOutputFormatClass(TextOutputFormat.class); FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); job.waitForCompletion(true); } } ```

hadoop使用yarn运行jar 报java.lang.ClassNotFoundException 找不到类 (找不到的不是主类)

1、写了一个数据分析的程序,用idea打成jar包,依赖jar都打进去了 ![图片说明](https://img-ask.csdn.net/upload/201911/03/1572779664_439750.png) 已经设置了 job.setJarByClass(CountDurationRunner.class); 2、开启hadoop zookeeper 和hbase集群 3、yarn运行jar : $ /opt/module/hadoop-2.7.2/bin/yarn jar ct_analysis.jar runner.CountDurationRunner 报错截图:![图片说明](https://img-ask.csdn.net/upload/201911/03/1572779908_781957.png) CountDurationRunner类代码: ``` package runner; import kv.key.ComDimension; //就是这里第一个就没找到 import kv.value.CountDurationValue; import mapper.CountDurationMapper; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.HBaseConfiguration; import org.apache.hadoop.hbase.TableName; import org.apache.hadoop.hbase.client.Admin; import org.apache.hadoop.hbase.client.Connection; import org.apache.hadoop.hbase.client.ConnectionFactory; import org.apache.hadoop.hbase.client.Scan; import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; import outputformat.MysqlOutputFormat; import reducer.CountDurationReducer; import java.io.IOException; public class CountDurationRunner implements Tool { private Configuration conf = null; @Override public void setConf(Configuration conf) { this.conf = HBaseConfiguration.create(conf); } @Override public Configuration getConf() { return this.conf; } @Override public int run(String[] args) throws Exception { //得到conf Configuration conf = this.getConf(); //实例化job Job job = Job.getInstance(conf); job.setJarByClass(CountDurationRunner.class); //组装Mapper InputFormat initHbaseInputConfig(job); //组装Reducer outputFormat initHbaseOutputConfig(job); return job.waitForCompletion(true) ? 0 : 1; } private void initHbaseOutputConfig(Job job) { Connection connection = null; Admin admin = null; String tableName = "ns_ct:calllog"; try { connection = ConnectionFactory.createConnection(job.getConfiguration()); admin = connection.getAdmin(); if(!admin.tableExists(TableName.valueOf(tableName))) throw new RuntimeException("没有找到目标表"); Scan scan = new Scan(); //初始化Mapper TableMapReduceUtil.initTableMapperJob( tableName, scan, CountDurationMapper.class, ComDimension.class, Text.class, job, true); }catch (IOException e){ e.printStackTrace(); }finally { try { if(admin!=null) admin.close(); if(connection!=null) connection.close(); } catch (IOException e) { e.printStackTrace(); } } } private void initHbaseInputConfig(Job job) { job.setReducerClass(CountDurationReducer.class); job.setOutputKeyClass(ComDimension.class); job.setOutputValueClass(CountDurationValue.class); job.setOutputFormatClass(MysqlOutputFormat.class); } public static void main(String[] args) { try { int status = ToolRunner.run(new CountDurationRunner(), args); System.exit(status); } catch (Exception e) { e.printStackTrace(); } } } 这问题困扰很久了,有人说classPath不对,不知道如何修改,求助! ```

mapreduce计算的数据导入mysql出错,导入到本地都ok,哪位路过的大佬能帮忙看下

![报错信息](https://img-ask.csdn.net/upload/201909/29/1569747157_413167.png) Driver代码如下 package com.sky.cmcc.offlineComputeMR; import com.sky.cmcc.pojo.MFee; import com.sky.cmcc.pojo.RFee; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.NullWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.db.DBConfiguration; import org.apache.hadoop.mapreduce.lib.db.DBOutputFormat; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import java.io.IOException; /** * Classname CmccDriver * Date 2019/9/29 11:08 * Created by Teddys * Description 负责加载配置,启动MR,写入数据到mysql */ public class CmccDriver { // 定义msyql的四项 private static String DriverClass="com.mysql.jdbc.Driver"; private static String url="jdbc:mysql://localhost:3306/bot?characterEncoding=UTF-8"; private static String username="root"; private static String password="123456"; public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { //1 加载配置文件和设定Job Configuration conf = new Configuration(); Job job = Job.getInstance(conf); //连接mysql DBConfiguration.configureDB(conf,DriverClass,url,username,password); //2 设置Job的加载路径 job.setJarByClass(CmccDriver.class); //3 指定job的mapper和reducer的类 job.setMapperClass(CmccMapper.class); job.setReducerClass(CmccReducer.class); //4 设置mqpper和最后的输出类型 job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(MFee.class); job.setOutputKeyClass(RFee.class); job.setOutputValueClass(NullWritable.class); //5 设置输入和输出的路径 FileInputFormat.setInputPaths(job,new Path(args[0])); //注意:输出路径为mysql DBOutputFormat.setOutput(job,"cmcc0513", "day","chargefee","shouldfee","orderCount","chargePayTime","chargeSuccessCount"); //6 提交任务,执行程序 boolean b = job.waitForCompletion(true); System.exit(b ? 0 : 1); System.out.println("b===="+b); } }

java.lang.ClassCastException

public class ipSort { public static class Map extends Mapper<LongWritable, IntWritable, IntWritable, Text>{ //将输入文件转换成<ipNum,ipAdd>的形式 private final static IntWritable ipNum = new IntWritable(); private Text ipAdd = new Text(); public void map(LongWritable key, IntWritable value, Context context) throws IOException, InterruptedException{ //把每一行转成字符串 String line = value.toString(); // 分割每一行 StringTokenizer token = new StringTokenizer(line); //solve every line while(token.hasMoreElements()){ //divided by blank StringTokenizer tokenLine = new StringTokenizer(token.nextToken()); ipAdd.set(token.nextToken().trim()); ipNum.set(Integer.valueOf(token.nextToken().trim())); context.write(ipNum,new Text(ipAdd)); } } } public static class Reduce extends Reducer<IntWritable, Text, Text, IntWritable>{ //把Map阶段的输出结果颠倒; private Text result = new Text(); public void reduce(IntWritable key,Iterable<Text> values, Context context) throws IOException, InterruptedException{ for(Text val : values){ result.set(val.toString()); context.write(new Text(result),key); } } } public static class IntKeyDescComparator extends WritableComparator{ protected IntKeyDescComparator(){ super(IntWritable.class,true); } public int compare(WritableComparable a, WritableComparable b){ return super.compare(a, b); } } public static void main(String args[]) throws IOException, ClassNotFoundException, InterruptedException{ System.setProperty("hadoop.home.dir", "C:\\Users\\lenovo\\Desktop\\hadoop-2.6.0\\hadoop-2.6.0"); Configuration conf = new Configuration(); conf.set("mapred.job.tracker", "192.168.142.138"); Job job = new Job(conf,"ipSort"); job.setJarByClass(ipSort.class); job.setSortComparatorClass(IntKeyDescComparator.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setOutputKeyClass(IntWritable.class); job.setOutputValueClass(Text.class); FileInputFormat.addInputPath(job, new Path("hdfs://10.170.54.193:9000/input")); FileOutputFormat.setOutputPath(job, new Path("hdfs://10.170.54.193:9000/output")); System.exit(job.waitForCompletion(true)?0:1); } 运行时出现问题Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.io.IntWritable,但是找不到哪里类型转换错误了

mapreduce编程出现错误:Job running in uber mode : false?

eclipse编写mapreduce程序,实现统计多个目录下所有文件的文件行数总数。 代码如下: package ZhangBo; import java.io.IOException; //import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; //import org.apache.hadoop.mapreduce.lib.input.MultipleInputs; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; //import org.apache.hadoop.io.LongWritable; //import org.apache.hadoop.util.GenericOptionsParser; //import org.junit.Test; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; //import org.apache.log4j.MDC; //import org.apache.log4j.Logger; //import firstnight.testlog4jj; public class WordCount { //////log static Log logger = LogFactory.getLog(WordCount.class); public static class TokenizerMapper extends Mapper<LongWritable, Text, Text, IntWritable>{ //一行的起始位置 private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context ) throws IOException, InterruptedException { //进行分割 logger.debug("This is debug message."); logger.error("This is error message."); logger.info("ni hao "); context.write(word,one); } } //对输入的key value值进行处理 转换成新的 key value值 进行一系列框架自带的处理 排序 合并 分句等等 public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; //循环遍历 IntWritable for (IntWritable val : values) { sum += val.get();//get 转 int } result.set(sum); context.write(key, result); } } private static FileSystem hdfs; private static String args1; private static String args2; private static String args3; Configuration conf1 = new Configuration(); private static void showDir(FileStatus fs) throws Exception { Path path = fs.getPath(); if (fs.isDir()) { args2 = path.toString(); if (args2 != args3) args1 = args1 +","+args2; System.out.println("目录:" + path); FileStatus[] f = hdfs.listStatus(path); if (f.length > 0) { for (FileStatus file:f) { showDir(file); args3 = file.getPath().toString(); } } } else { System.out.println("文件:" + path); } System.out.println("目录之和: "+args1); } public void test() throws Exception { hdfs = FileSystem.get(conf1); FileStatus[] fs = hdfs.listStatus(new Path("hdfs://C15/zbo")); args1 = "hdfs://C15/zbo"; if (fs.length > 0) { for (FileStatus f : fs) { showDir(f); } } else { System.out.println("没有什么好遍历的...."); } } public static void main(String[] args) throws Exception { new WordCount().test(); Configuration conf = new Configuration(); ////////////////////// Job job = new Job(conf,"word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); FileInputFormat.addInputPaths(job,args1); FileOutputFormat.setOutputPath(job, new Path("outbo")); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); System.exit(job.waitForCompletion(true)?0:1); } } 在目录hdfs://C15/zbo下创建目录和文件 在eclipse上打jar包ho.jar 执行程序 :hadoop jar ZhangBo.WordCount 执行结果如下所示: [root@c15m1 /]# hadoop jar ho.jar ZhangBo.WordCount 文件:hdfs://C15/zbo/file1.txt 目录之和: hdfs://C15/zbo 文件:hdfs://C15/zbo/file2.txt 目录之和: hdfs://C15/zbo 目录:hdfs://C15/zbo/zbo 文件:hdfs://C15/zbo/zbo/file1.txt 目录之和: hdfs://C15/zbo,hdfs://C15/zbo/zbo 文件:hdfs://C15/zbo/zbo/file2.txt 目录之和: hdfs://C15/zbo,hdfs://C15/zbo/zbo 目录:hdfs://C15/zbo/zbo/zbo 文件:hdfs://C15/zbo/zbo/zbo/file1.txt 目录之和: hdfs://C15/zbo,hdfs://C15/zbo/zbo,hdfs://C15/zbo/zbo/zbo 文件:hdfs://C15/zbo/zbo/zbo/file2.txt 目录之和: hdfs://C15/zbo,hdfs://C15/zbo/zbo,hdfs://C15/zbo/zbo/zbo 目录之和: hdfs://C15/zbo,hdfs://C15/zbo/zbo,hdfs://C15/zbo/zbo/zbo 目录之和: hdfs://C15/zbo,hdfs://C15/zbo/zbo,hdfs://C15/zbo/zbo/zbo 15/11/17 08:55:37 INFO client.RMProxy: Connecting to ResourceManager at c15m1.ecld.com/132.121.94.213:8050 15/11/17 08:55:38 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 72820725 for hdfs on ha-hdfs:C15 15/11/17 08:55:38 INFO security.TokenCache: Got dt for hdfs://C15; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:C15, Ident: (HDFS_DELEGATION_TOKEN token 72820725 for hdfs) 15/11/17 08:55:38 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 15/11/17 08:55:38 INFO input.FileInputFormat: Total input paths to process : 8 15/11/17 08:55:38 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 15/11/17 08:55:38 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev dbd51f0fb61f5347228a7a23fe0765ac1242fcdf] 15/11/17 08:55:47 INFO mapreduce.JobSubmitter: number of splits:8 15/11/17 08:55:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1440641224751_187144 15/11/17 08:55:48 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:C15, Ident: (HDFS_DELEGATION_TOKEN token 72820725 for hdfs) 15/11/17 08:55:48 INFO impl.YarnClientImpl: Submitted application application_1440641224751_187144 15/11/17 08:55:49 INFO mapreduce.Job: The url to track the job: http://c15m1.ecld.com:8088/proxy/application_1440641224751_187144/ 15/11/17 08:55:49 INFO mapreduce.Job: Running job: job_1440641224751_187144 15/11/17 08:55:51 INFO mapreduce.Job: Job job_1440641224751_187144 running in uber mode : false 15/11/17 08:55:51 INFO mapreduce.Job: map 0% reduce 0% 15/11/17 08:55:51 INFO mapreduce.Job: Job job_1440641224751_187144 failed with state FAILED due to: Application application_1440641224751_187144 failed 2 times due to AM Container for appattempt_1440641224751_187144_000002 exited with exitCode: -1000 due to: Application application_1440641224751_187144 initialization failed (exitCode=255) with output: Requested user hdfs is banned .Failing this attempt.. Failing the application. 15/11/17 08:55:51 INFO mapreduce.Job: Counters: 0 问题:running in uber mode : false是什么问题?

Exception in thread "main" java.lang.NoClassDefFoundError:

这是我的java代码,用于Mapreduce打包,至于Linux里。 import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import java.io.IOException; public class WordCountAPP { public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { if(args.length == 0){ args = new String[]{"/wordcount.txt","/wordcount-result"}; } Configuration conf=new Configuration(); Job job=Job.getInstance(conf,WordCountMap.class.getName()); //打成jar包运行 job.setJarByClass(WordCountMap.class); //数据来自哪里 FileInputFormat.setInputPaths(job,args[0]); //使用哪个mapper处理输入的数据 job.setMapperClass(WordCountMap.class); //map输出的数据类型是什么 job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(LongWritable.class); //使用哪个reducer处理输出的数据 job.setReducerClass(WordCountReduce.class); //reduce输出数据的类型是什么 job.setOutputKeyClass(Text.class); job.setOutputValueClass(LongWritable.class); //数据输出到哪里 FileOutputFormat.setOutputPath(job,new Path(args[1])); //交给yarn去执行,直到执行结束后才退出本程序 job.waitForCompletion(true); } public static class WordCountMap extends Mapper<LongWritable,Text,Text,LongWritable> { @Override public void map(LongWritable key, Text value, Mapper<LongWritable,Text,Text,LongWritable>.Context context) throws IOException, InterruptedException { String line=value.toString(); String[] splited=line.split(" "); for(String word: splited){ context.write(new Text(word),new LongWritable(1)); } } } public static class WordCountReduce extends Reducer<Text,LongWritable,Text,LongWritable> { @Override public void reduce(Text key, Iterable<LongWritable> values, Reducer<Text,LongWritable,Text,LongWritable>.Context context) throws IOException, InterruptedException { long count=0L; for (LongWritable v:values){ count+=v.get(); } LongWritable v2=new LongWritable(count); context.write(key,v2); } } } 这是运行后报的错 Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration at WordCountAPP.main(WordCountAPP.java:17) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 1 more

统计phone_data 数据,代码没有问题但是报错

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Exception in thread "main" java.lang.NullPointerException at java.lang.ProcessBuilder.start(Unknown Source) at org.apache.hadoop.util.Shell.runCommand(Shell.java:483) at org.apache.hadoop.util.Shell.run(Shell.java:456) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) at org.apache.hadoop.util.Shell.execCommand(Shell.java:815) at org.apache.hadoop.util.Shell.execCommand(Shell.java:798) at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:731) at org.apache.hadoop.fs.RawLocalFileSystem.mkOneDirWithMode(RawLocalFileSystem.java:489) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirsWithOptionalPermission(RawLocalFileSystem.java:529) at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:507) at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:305) at org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:133) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:144) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Unknown Source) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at com.atguigu.mapreduce.flowsum.FlowDriver.main(FlowDriver.java:40)

大学四年自学走来,这些私藏的实用工具/学习网站我贡献出来了

大学四年,看课本是不可能一直看课本的了,对于学习,特别是自学,善于搜索网上的一些资源来辅助,还是非常有必要的,下面我就把这几年私藏的各种资源,网站贡献出来给你们。主要有:电子书搜索、实用工具、在线视频学习网站、非视频学习网站、软件下载、面试/求职必备网站。 注意:文中提到的所有资源,文末我都给你整理好了,你们只管拿去,如果觉得不错,转发、分享就是最大的支持了。 一、电子书搜索 对于大部分程序员...

在中国程序员是青春饭吗?

今年,我也32了 ,为了不给大家误导,咨询了猎头、圈内好友,以及年过35岁的几位老程序员……舍了老脸去揭人家伤疤……希望能给大家以帮助,记得帮我点赞哦。 目录: 你以为的人生 一次又一次的伤害 猎头界的真相 如何应对互联网行业的「中年危机」 一、你以为的人生 刚入行时,拿着傲人的工资,想着好好干,以为我们的人生是这样的: 等真到了那一天,你会发现,你的人生很可能是这样的: ...

springboot+jwt实现token登陆权限认证

一 前言 此篇文章的内容也是学习不久,终于到周末有时间码一篇文章分享知识追寻者的粉丝们,学完本篇文章,读者将对token类的登陆认证流程有个全面的了解,可以动态搭建自己的登陆认证过程;对小项目而已是个轻量级的认证机制,符合开发需求;更多精彩原创内容关注公主号知识追寻者,读者的肯定,就是对作者的创作的最大支持; 二 jwt实现登陆认证流程 用户使用账号和面发出post请求 服务器接受到请求后使用私...

技术大佬:我去,你写的 switch 语句也太老土了吧

昨天早上通过远程的方式 review 了两名新来同事的代码,大部分代码都写得很漂亮,严谨的同时注释也很到位,这令我非常满意。但当我看到他们当中有一个人写的 switch 语句时,还是忍不住破口大骂:“我擦,小王,你丫写的 switch 语句也太老土了吧!” 来看看小王写的代码吧,看完不要骂我装逼啊。 private static String createPlayer(PlayerTypes p...

女程序员,为什么比男程序员少???

昨天看到一档综艺节目,讨论了两个话题:(1)中国学生的数学成绩,平均下来看,会比国外好?为什么?(2)男生的数学成绩,平均下来看,会比女生好?为什么?同时,我又联想到了一个技术圈经常讨...

总结了 150 余个神奇网站,你不来瞅瞅吗?

原博客再更新,可能就没了,之后将持续更新本篇博客。

副业收入是我做程序媛的3倍,工作外的B面人生是怎样的?

提到“程序员”,多数人脑海里首先想到的大约是:为人木讷、薪水超高、工作枯燥…… 然而,当离开工作岗位,撕去层层标签,脱下“程序员”这身外套,有的人生动又有趣,马上展现出了完全不同的A/B面人生! 不论是简单的爱好,还是正经的副业,他们都干得同样出色。偶尔,还能和程序员的特质结合,产生奇妙的“化学反应”。 @Charlotte:平日素颜示人,周末美妆博主 大家都以为程序媛也个个不修边幅,但我们也许...

MySQL数据库面试题(2020最新版)

文章目录数据库基础知识为什么要使用数据库什么是SQL?什么是MySQL?数据库三大范式是什么mysql有关权限的表都有哪几个MySQL的binlog有有几种录入格式?分别有什么区别?数据类型mysql有哪些数据类型引擎MySQL存储引擎MyISAM与InnoDB区别MyISAM索引与InnoDB索引的区别?InnoDB引擎的4大特性存储引擎选择索引什么是索引?索引有哪些优缺点?索引使用场景(重点)...

如果你是老板,你会不会踢了这样的员工?

有个好朋友ZS,是技术总监,昨天问我:“有一个老下属,跟了我很多年,做事勤勤恳恳,主动性也很好。但随着公司的发展,他的进步速度,跟不上团队的步伐了,有点...

我入职阿里后,才知道原来简历这么写

私下里,有不少读者问我:“二哥,如何才能写出一份专业的技术简历呢?我总感觉自己写的简历太烂了,所以投了无数份,都石沉大海了。”说实话,我自己好多年没有写过简历了,但我认识的一个同行,他在阿里,给我说了一些他当年写简历的方法论,我感觉太牛逼了,实在是忍不住,就分享了出来,希望能够帮助到你。 01、简历的本质 作为简历的撰写者,你必须要搞清楚一点,简历的本质是什么,它就是为了来销售你的价值主张的。往深...

程序员写出这样的代码,能不挨骂吗?

当你换槽填坑时,面对一个新的环境。能够快速熟练,上手实现业务需求是关键。但是,哪些因素会影响你快速上手呢?是原有代码写的不够好?还是注释写的不够好?昨夜...

外包程序员的幸福生活

今天给你们讲述一个外包程序员的幸福生活。男主是Z哥,不是在外包公司上班的那种,是一名自由职业者,接外包项目自己干。接下来讲的都是真人真事。 先给大家介绍一下男主,Z哥,老程序员,是我十多年前的老同事,技术大牛,当过CTO,也创过业。因为我俩都爱好喝酒、踢球,再加上住的距离不算远,所以一直也断断续续的联系着,我对Z哥的状况也有大概了解。 Z哥几年前创业失败,后来他开始干起了外包,利用自己的技术能...

优雅的替换if-else语句

场景 日常开发,if-else语句写的不少吧??当逻辑分支非常多的时候,if-else套了一层又一层,虽然业务功能倒是实现了,但是看起来是真的很不优雅,尤其是对于我这种有强迫症的程序"猿",看到这么多if-else,脑袋瓜子就嗡嗡的,总想着解锁新姿势:干掉过多的if-else!!!本文将介绍三板斧手段: 优先判断条件,条件不满足的,逻辑及时中断返回; 采用策略模式+工厂模式; 结合注解,锦...

离职半年了,老东家又发 offer,回不回?

有小伙伴问松哥这个问题,他在上海某公司,在离职了几个月后,前公司的领导联系到他,希望他能够返聘回去,他很纠结要不要回去? 俗话说好马不吃回头草,但是这个小伙伴既然感到纠结了,我觉得至少说明了两个问题:1.曾经的公司还不错;2.现在的日子也不是很如意。否则应该就不会纠结了。 老实说,松哥之前也有过类似的经历,今天就来和小伙伴们聊聊回头草到底吃不吃。 首先一个基本观点,就是离职了也没必要和老东家弄的苦...

2020阿里全球数学大赛:3万名高手、4道题、2天2夜未交卷

阿里巴巴全球数学竞赛( Alibaba Global Mathematics Competition)由马云发起,由中国科学技术协会、阿里巴巴基金会、阿里巴巴达摩院共同举办。大赛不设报名门槛,全世界爱好数学的人都可参与,不论是否出身数学专业、是否投身数学研究。 2020年阿里巴巴达摩院邀请北京大学、剑桥大学、浙江大学等高校的顶尖数学教师组建了出题组。中科院院士、美国艺术与科学院院士、北京国际数学...

为什么你不想学习?只想玩?人是如何一步一步废掉的

不知道是不是只有我这样子,还是你们也有过类似的经历。 上学的时候总有很多光辉历史,学年名列前茅,或者单科目大佬,但是虽然慢慢地长大了,你开始懈怠了,开始废掉了。。。 什么?你说不知道具体的情况是怎么样的? 我来告诉你: 你常常潜意识里或者心理觉得,自己真正的生活或者奋斗还没有开始。总是幻想着自己还拥有大把时间,还有无限的可能,自己还能逆风翻盘,只不是自己还没开始罢了,自己以后肯定会变得特别厉害...

男生更看重女生的身材脸蛋,还是思想?

往往,我们看不进去大段大段的逻辑。深刻的哲理,往往短而精悍,一阵见血。问:产品经理挺漂亮的,有点心动,但不知道合不合得来。男生更看重女生的身材脸蛋,还是...

为什么程序员做外包会被瞧不起?

二哥,有个事想询问下您的意见,您觉得应届生值得去外包吗?公司虽然挺大的,中xx,但待遇感觉挺低,马上要报到,挺纠结的。

当HR压你价,说你只值7K,你该怎么回答?

当HR压你价,说你只值7K时,你可以流畅地回答,记住,是流畅,不能犹豫。 礼貌地说:“7K是吗?了解了。嗯~其实我对贵司的面试官印象很好。只不过,现在我的手头上已经有一份11K的offer。来面试,主要也是自己对贵司挺有兴趣的,所以过来看看……”(未完) 这段话主要是陪HR互诈的同时,从公司兴趣,公司职员印象上,都给予对方正面的肯定,既能提升HR的好感度,又能让谈判气氛融洽,为后面的发挥留足空间。...

面试:第十六章:Java中级开发

HashMap底层实现原理,红黑树,B+树,B树的结构原理 Spring的AOP和IOC是什么?它们常见的使用场景有哪些?Spring事务,事务的属性,传播行为,数据库隔离级别 Spring和SpringMVC,MyBatis以及SpringBoot的注解分别有哪些?SpringMVC的工作原理,SpringBoot框架的优点,MyBatis框架的优点 SpringCould组件有哪些,他们...

早上躺尸,晚上干活:硅谷科技公司这么流行迟到?

硅谷科技公司上班时间OPEN早已不是什么新鲜事,早九晚五是常态,但有很多企业由于不打卡,员工们10点、11点才“姗姗来迟”的情况也屡见不鲜。 这种灵活的考勤制度为人羡慕,甚至近年来,国内某些互联网企业也纷纷效仿。不过,硅谷普遍弹性的上班制度是怎么由来的呢?这种“流行性迟到”真的有那么轻松、悠哉吗? 《动态规划专题班》 课程试听内容: 动态规划的解题要领 动态规划三大类 求最值/计数/可行性 常...

面试阿里p7,被按在地上摩擦,鬼知道我经历了什么?

面试阿里p7被问到的问题(当时我只知道第一个):@Conditional是做什么的?@Conditional多个条件是什么逻辑关系?条件判断在什么时候执...

终于懂了TCP和UDP协议区别

终于懂了TCP和UDP协议区别

Python爬虫,高清美图我全都要(彼岸桌面壁纸)

爬取彼岸桌面网站较为简单,用到了requests、lxml、Beautiful Soup4

无代码时代来临,程序员如何保住饭碗?

编程语言层出不穷,从最初的机器语言到如今2500种以上的高级语言,程序员们大呼“学到头秃”。程序员一边面临编程语言不断推陈出新,一边面临由于许多代码已存在,程序员编写新应用程序时存在重复“搬砖”的现象。 无代码/低代码编程应运而生。无代码/低代码是一种创建应用的方法,它可以让开发者使用最少的编码知识来快速开发应用程序。开发者通过图形界面中,可视化建模来组装和配置应用程序。这样一来,开发者直...

面试了一个 31 岁程序员,让我有所触动,30岁以上的程序员该何去何从?

最近面试了一个31岁8年经验的程序猿,让我有点感慨,大龄程序猿该何去何从。

大三实习生,字节跳动面经分享,已拿Offer

说实话,自己的算法,我一个不会,太难了吧

程序员垃圾简历长什么样?

已经连续五年参加大厂校招、社招的技术面试工作,简历看的不下于万份 这篇文章会用实例告诉你,什么是差的程序员简历! 疫情快要结束了,各个公司也都开始春招了,作为即将红遍大江南北的新晋UP主,那当然要为小伙伴们做点事(手动狗头)。 就在公众号里公开征简历,义务帮大家看,并一一点评。《启舰:春招在即,义务帮大家看看简历吧》 一石激起千层浪,三天收到两百多封简历。 花光了两个星期的所有空闲时...

Java岗开发3年,公司临时抽查算法,离职后这几题我记一辈子

前几天我们公司做了一件蠢事,非常非常愚蠢的事情。我原以为从学校出来之后,除了找工作有测试外,不会有任何与考试有关的事儿。 但是,天有不测风云,公司技术总监、人事总监两位大佬突然降临到我们事业线,叫上我老大,给我们组织了一场别开生面的“考试”。 那是一个风和日丽的下午,我翘着二郎腿,左手端着一杯卡布奇诺,右手抓着我的罗技鼠标,滚动着轮轴,穿梭在头条热点之间。 “淡黄的长裙~蓬松的头发...

大胆预测下未来5年的Web开发

在2019年的ReactiveConf 上,《Elm in Action》的作者Richard Feldman对未来5年Web开发的发展做了预测,很有意思,分享给大家。如果你有机会从头...

立即提问
相关内容推荐