【新手】Hadoop MapReduce 执行中Map没有输出

hadoop - hadoop2.6 分布式 - 简单实例学习 - 统计某年的最高温度和按年份将温度从高到底排序 - 原明卓 - 博客频道 - CSDN.NET http://blog.csdn.net/lablenet/article/details/50608197#java

我按照这篇博客做的,运行结果见图。

16/10/19 05:27:51 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
16/10/19 05:27:52 INFO input.FileInputFormat: Total input paths to process : 1
16/10/19 05:27:52 INFO util.NativeCodeLoader: Loaded the native-hadoop library
16/10/19 05:27:52 WARN snappy.LoadSnappy: Snappy native library not loaded
16/10/19 05:27:54 INFO mapred.JobClient: Running job: job_201610190234_0013
16/10/19 05:27:55 INFO mapred.JobClient: map 0% reduce 0%
16/10/19 05:28:24 INFO mapred.JobClient: map 100% reduce 0%
16/10/19 05:28:41 INFO mapred.JobClient: map 100% reduce 20%
16/10/19 05:28:42 INFO mapred.JobClient: map 100% reduce 40%
16/10/19 05:28:50 INFO mapred.JobClient: map 100% reduce 46%
16/10/19 05:28:51 INFO mapred.JobClient: map 100% reduce 60%
16/10/19 05:29:01 INFO mapred.JobClient: map 100% reduce 100%
16/10/19 05:29:01 INFO mapred.JobClient: Job complete: job_201610190234_0013
16/10/19 05:29:01 INFO mapred.JobClient: Counters: 28
16/10/19 05:29:01 INFO mapred.JobClient: Job Counters
16/10/19 05:29:01 INFO mapred.JobClient: Launched reduce tasks=6
16/10/19 05:29:01 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=26528
16/10/19 05:29:01 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
16/10/19 05:29:01 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
16/10/19 05:29:01 INFO mapred.JobClient: Launched map tasks=1
16/10/19 05:29:01 INFO mapred.JobClient: Data-local map tasks=1
16/10/19 05:29:01 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=107381
16/10/19 05:29:01 INFO mapred.JobClient: File Output Format Counters
16/10/19 05:29:01 INFO mapred.JobClient: Bytes Written=0
16/10/19 05:29:01 INFO mapred.JobClient: FileSystemCounters
16/10/19 05:29:01 INFO mapred.JobClient: FILE_BYTES_READ=30
16/10/19 05:29:01 INFO mapred.JobClient: HDFS_BYTES_READ=1393
16/10/19 05:29:01 INFO mapred.JobClient: FILE_BYTES_WRITTEN=354256
16/10/19 05:29:01 INFO mapred.JobClient: File Input Format Counters
16/10/19 05:29:01 INFO mapred.JobClient: Bytes Read=1283
16/10/19 05:29:01 INFO mapred.JobClient: Map-Reduce Framework
16/10/19 05:29:01 INFO mapred.JobClient: Map output materialized bytes=30
16/10/19 05:29:01 INFO mapred.JobClient: Map input records=46
16/10/19 05:29:01 INFO mapred.JobClient: Reduce shuffle bytes=30
16/10/19 05:29:01 INFO mapred.JobClient: Spilled Records=0
16/10/19 05:29:01 INFO mapred.JobClient: Map output bytes=0
16/10/19 05:29:01 INFO mapred.JobClient: CPU time spent (ms)=16910
16/10/19 05:29:01 INFO mapred.JobClient: Total committed heap usage (bytes)=195301376
16/10/19 05:29:01 INFO mapred.JobClient: Combine input records=0
16/10/19 05:29:01 INFO mapred.JobClient: SPLIT_RAW_BYTES=110
16/10/19 05:29:01 INFO mapred.JobClient: Reduce input records=0
16/10/19 05:29:01 INFO mapred.JobClient: Reduce input groups=0
16/10/19 05:29:01 INFO mapred.JobClient: Combine output records=0
16/10/19 05:29:01 INFO mapred.JobClient: Physical memory (bytes) snapshot=331567104
16/10/19 05:29:01 INFO mapred.JobClient: Reduce output records=0
16/10/19 05:29:01 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2264113152

16/10/19 05:29:01 INFO mapred.JobClient: Map output records=0

yyyy-MM-dd HH:mm:ss\t温度
example:1995-10-10 10:10:10 6.54
这是数据源格式,我把
RunJob中的
int year=c.get(1);

String hot=ss[1].substring(0,ss[1].lastIndexOf("°C"));

KeyPari keyPari=new KeyPari();

keyPari.setYear(year);

中的°C改成了\n。


代码和博文的一样,只删掉了MAP里面的IF判断和修改了输入输出路径。求前辈们指教一下为什么会这样,深表感激。

2个回答

yyyy-MM-dd HH:mm:ss\t温度
example:1995-10-10 10:10:10 6.54
这是数据源格式,我把
RunJob中的
int year=c.get(1);

String hot=ss[1].substring(0,ss[1].lastIndexOf("°C"));

KeyPari keyPari=new KeyPari();

keyPari.setYear(year);

中的°C改成了\n。

Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
其他相关推荐
QuartzJob 调用 hadoop mapreduce 报错
Error: java.io.IOException: com.mysql.jdbc.Driver at org.apache.hadoop.mapreduce.lib.db.DBOutputFormat.getRecordWriter(DBOutputFormat.java:185) at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Hadoop mapreduce传值问题
最近mapreduce编写遇到了问题。在step4中,reduce可以同时收到从map中传来的A和B两组数据。但是在step5中的reudce却无法同时收到A、B两组数据,出现了有A没B,有B没A的现象,即A和B无法在同一次循环中出现。 step5,我几乎是从step4复制过来的,很奇怪他们的执行步骤为什么不一样。 step4 ``` import java.io.IOException; import java.util.HashMap; import java.util.Iterator; import java.util.Map; import java.util.regex.Pattern; import org.apache.commons.net.telnet.EchoOptionHandler; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.FileSplit; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat.LogWriter; //同现矩阵和用户偏好矩阵相乘 public class Step4 { public static boolean run(Configuration con, Map<String, String>map) { try { FileSystem fs = FileSystem.get(con); Job job = Job.getInstance(); job.setJobName("step4"); job.setJarByClass(App.class); job.setMapperClass(Step4_Mapper.class); job.setReducerClass(Step4_Reducer.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(Text.class); FileInputFormat.setInputPaths(job, new Path[] { new Path(map.get("Step4Input1")), new Path(map.get("Step4Input2")) }); Path outpath = new Path(map.get("Step4Output")); if(fs.exists(outpath)){ fs.delete(outpath,true); } FileOutputFormat.setOutputPath(job, outpath); boolean f = job.waitForCompletion(true); return f; }catch(Exception e) { e.printStackTrace(); } return false; } static class Step4_Mapper extends Mapper<LongWritable, Text, Text, Text>{ private String flag; //每次map时都会先判断一次 @Override protected void setup(Context context )throws IOException,InterruptedException{ FileSplit split = (FileSplit) context.getInputSplit(); flag = split.getPath().getParent().getName(); System.out.print(flag+ "*************************"); } @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{ String[] tokens = Pattern.compile("[\t,]").split(value.toString()); //物品共现矩阵 if(flag.equals("step3")) { // i2:i3 1 // i2:i2 2 String[] v1 = tokens[0].split(":"); String itemID1 = v1[0]; String itemID2 = v1[1]; String num = tokens[1]; Text k = new Text(itemID1); Text v = new Text("A:"+itemID2+","+num); //A:i2,1 context.write(k,v); }else if(flag.equals("step2")) {//用户评价矩阵 // u2 i1:2,i3:4 String userID = tokens[0]; for(int i=1;i<tokens.length;i++) { String[] vector = tokens[i].split(":"); String itemID = vector[0]; //物品ID String pref = vector[1];//评分 Text k = new Text(itemID); Text v = new Text("B:"+userID+","+pref); context.write(k, v); } } } } static class Step4_Reducer extends Reducer<Text, Text, Text, Text>{ @Override protected void reduce(Text key, Iterable<Text>values, Context context) throws IOException,InterruptedException{ //A为同现矩阵,B为用户偏好矩阵 //某一个物品k,针对它和其他所有物品的同现次数v,都在mapA集合中 // Text k = new Text(itemID1); //Text v = new Text("A:"+itemID2+","+num); //A:i2,1 // context.write(k,v); //和该物品(key中的itemID)同现的其他物品的同现集合 //其他物品ID为map的key,同现数字为值 Map<String, Integer> mapA = new HashMap<String,Integer>(); //该物品(key中的itemID),所有用户的推荐权重分数 Map<String, Integer>mapB = new HashMap<String,Integer>(); for(Text line:values) { String val = line.toString(); if(val.startsWith("A:")) { String[] kv = Pattern.compile("[\t,]").split(val.substring(2)); try { mapA.put(kv[0], Integer.parseInt(kv[1])); }catch(Exception e) { e.printStackTrace(); } }else if(val.startsWith("B:")) { String[] kv = Pattern.compile("[\t,]").split(val.substring(2)); try { mapB.put(kv[0], Integer.parseInt(kv[1])); }catch(Exception e) { e.printStackTrace(); } } } double result = 0; Iterator<String>iter = mapA.keySet().iterator(); while(iter.hasNext()) { String mapk = iter.next(); //itemID int num =mapA.get(mapk).intValue(); // 获取同现值 Iterator<String>iterb = mapB.keySet().iterator(); while(iterb.hasNext()) { String mapkb = iterb.next(); int pref = mapB.get(mapkb).intValue(); result = num*pref; Text k = new Text(mapkb); Text v = new Text(mapk+ "," + result); context.write(k, v); } } } } } ``` step5 ``` import java.io.IOException; import java.util.HashMap; import java.util.Iterator; import java.util.Map; import java.util.regex.Pattern; import org.apache.commons.net.telnet.EchoOptionHandler; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.Mapper.Context; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.FileSplit; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.yarn.logaggregation.AggregatedLogFormat.LogWriter; //获得结果矩阵 public class Step5 { public static boolean run(Configuration con, Map<String, String>map) { try { FileSystem fs = FileSystem.get(con); Job job = Job.getInstance(); job.setJobName("step5"); job.setJarByClass(App.class); job.setMapperClass(Step5_Mapper.class); job.setReducerClass(Step5_Reducer.class); job.setMapOutputKeyClass(Text.class); job.setMapOutputValueClass(Text.class); FileInputFormat.setInputPaths(job, new Path[] { new Path(map.get("Step5Input1")), new Path(map.get("Step5Input2")) }); Path outpath = new Path(map.get("Step5Output")); if(fs.exists(outpath)){ fs.delete(outpath,true); } FileOutputFormat.setOutputPath(job, outpath); boolean f = job.waitForCompletion(true); return f; }catch(Exception e) { e.printStackTrace(); } return false; } static class Step5_Mapper extends Mapper<LongWritable, Text, Text, Text>{ private String flag; //每次map时都会先判断一次 @Override protected void setup(Context context )throws IOException,InterruptedException{ FileSplit split = (FileSplit) context.getInputSplit(); flag = split.getPath().getParent().getName(); System.out.print(flag+ "*************************"); } @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException{ String[] tokens = Pattern.compile("[\t,]").split(value.toString()); if(flag.equals("step4")) { // i2:i3 1 // i2:i2 2 Text k = new Text(tokens[0]); Text v = new Text("A:"+tokens[1]+","+tokens[2]); context.write(k, v); }else if(flag.equals("step2")) {//用户评价矩阵 // u2 i1:2,i3:4 String userID = tokens[0]; for(int i=1;i<tokens.length;i++) { String[] vector = tokens[i].split(":"); String itemID = vector[0]; //物品ID String pref = vector[1];//评分 Text k = new Text(itemID); Text v = new Text("B:"+userID+","+pref); context.write(k, v); } } } } //本reduce 负责累加结果 static class Step5_Reducer extends Reducer<Text, Text, Text, Text>{ protected void reduce(Text key, Iterable<Text>values, Context context) throws IOException,InterruptedException{ //其他物品ID为map的key,同现数字为值 Map<String, Double> mapA = new HashMap<String,Double>(); //该物品(key中的itemID),所有用户的推荐权重分数 Map<String, Integer>mapB = new HashMap<String,Integer>(); for(Text line : values) { String val = line.toString(); if(val.startsWith("A:")) { String[] kv = Pattern.compile("[\t,]").split(val.substring(2)); String tokens = kv[1]; String itemID = kv[0];//物品id Double score = Double.parseDouble(tokens); //相乘结果 //相加计算 if(mapA.containsKey(itemID)) { mapA.put(itemID, mapA.get(itemID)+score); }else { mapA.put(itemID, score); } }else if(val.startsWith("B:")) { String[] kv = Pattern.compile("[\t,]").split(val.substring(2)); try { mapB.put(kv[0], Integer.parseInt(kv[1])); }catch(Exception e) { e.printStackTrace(); } } } Iterator<String> iter = mapA.keySet().iterator(); while(iter.hasNext()) { String itemID = iter.next(); double score = mapA.get(itemID); Text v = new Text(itemID+","+score); Iterator<String>iterb = mapB.keySet().iterator(); while(iterb.hasNext()) { String mapkb = iterb.next(); Text k = new Text(mapkb); if(k.equals(key)) { continue; }else { context.write(key, v); } } } } } } ``` step4和step5配置 ![图片说明](https://img-ask.csdn.net/upload/201804/25/1524617462_994374.png) step4,在for循环中同时出现A和B ![step4,在for循环中同时出现A和B](https://img-ask.csdn.net/upload/201804/25/1524616391_511813.png) step5中,A和B无法出现在同一次循环 ![有A没B,此时mapB是无法点击开的](https://img-ask.csdn.net/upload/201804/25/1524616746_557066.png) 直接跳出了for循环进入下面的while循环,此时没有mapB,while无法正常进行 ![跳出了for循环](https://img-ask.csdn.net/upload/201804/25/1524616866_908151.png) 进行了多次step5后,输出完所有mapA之后,在下一次step5才进入mapB,此时轮到mapA是空的,而只有mapB ![mapA是空的,只有mapB](https://img-ask.csdn.net/upload/201804/25/1524617121_817431.png)
在eclipse运行hadoop mapreduce例子报错
在终端运行hadoop带的例子正常,hadoop节点正常,错误如下 17/09/05 20:20:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/09/05 20:20:16 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 17/09/05 20:20:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread "main" java.net.ConnectException: Call From master/192.168.1.110 to localhost:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1479) at org.apache.hadoop.ipc.Client.call(Client.java:1412) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:707) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1785) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1068) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at mapreduce.Temperature.main(Temperature.java:202) Caused by: java.net.ConnectException: 拒绝连接 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528) at org.apache.hadoop.ipc.Client.call(Client.java:1451) ... 28 more
eclipse运行hadoop mapreduce程序如下错误
2017-09-06 15:48:42,677 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner (LocalJobRunner.java:statusUpdate(591)) - 2017-09-06 15:48:42,686 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1460)) - Starting flush of map output 2017-09-06 15:48:42,686 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1482)) - Spilling map output 2017-09-06 15:48:42,686 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1483)) - bufstart = 0; bufend = 108; bufvoid = 104857600 2017-09-06 15:48:42,686 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:flush(1485)) - kvstart = 26214396(104857584); kvend = 26214352(104857408); length = 45/6553600 2017-09-06 15:48:42,733 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask (MapTask.java:sortAndSpill(1667)) - Finished spill 0 2017-09-06 15:48:42,743 INFO [LocalJobRunner Map Task Executor #0] mapred.Task (Task.java:done(1038)) - Task:attempt_local1469942249_0001_m_000000_0 is done. And is in the process of committing 2017-09-06 15:48:42,751 INFO [Thread-19] mapred.LocalJobRunner (LocalJobRunner.java:runTasks(456)) - map task executor complete. 2017-09-06 15:48:42,783 WARN [Thread-19] mapred.LocalJobRunner (LocalJobRunner.java:run(560)) - job_local1469942249_0001 java.lang.Exception: java.lang.NoSuchMethodError: org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getRssMemorySize()J at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522) Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.yarn.util.ResourceCalculatorProcessTree.getRssMemorySize()J at org.apache.hadoop.mapred.Task.updateResourceCounters(Task.java:872) at org.apache.hadoop.mapred.Task.updateCounters(Task.java:1021) at org.apache.hadoop.mapred.Task.done(Task.java:1040) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:345) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2017-09-06 15:48:43,333 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1360)) - Job job_local1469942249_0001 running in uber mode : false 2017-09-06 15:48:43,335 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1367)) - map 0% reduce 0% 2017-09-06 15:48:43,337 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1380)) - Job job_local1469942249_0001 failed with state FAILED due to: NA 2017-09-06 15:48:43,352 INFO [main] mapreduce.Job (Job.java:monitorAndPrintJob(1385)) - Counters: 10 Map-Reduce Framework Map input records=12 Map output records=12 Map output bytes=108 Map output materialized bytes=0 Input split bytes=104 Combine input records=0 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 File Input Format Counters Bytes Read=132 Finished
windows eclipse 下开发hadoop mapreduce,报空指针异常。
用三台ubuntu系统的服务器,搭建了hadoop集群,然后在windows下 用eclipse开发mapreduce,能连上hadoop,也能显示hdsf上的文件。自己写了mapreduce程序,run as hadoop 的时候,报空指针异常,什么localjob 之类的错误,什么原因求指点, 将工程打成jar包在linux hadoop环境用命令行运行是没问题的。。
hadoop mapreduce 数据分析 丢数据
最近发现hadoop的mapreduce程序会丢数据,不知道是什么原因,请教各位: hadoop环境,通过mapreduce程序分析hdfs上的数据,一天的数据是按小时存储的,每一个小时一个文件价,数据格式都是一样的,现在如果在16点这个文件价里有一条数据a,如果我用mr分析一整天的数据,数据a则丢失,如果单独跑16点这个文件夹里的数据,则数据a不会丢失,可以正常被分析出来,只要一加上其他时间段的数据,数据a就分析不出来,请问这是为什么? 最近在学习spark,我用spark程序跑同样的数据,整天的,不会有丢失的问题,的所以我肯定不是数据格式的问题 希望大家能帮我解决这个hadoop的问题,谢谢啦
hadoop mapreduce报错
java.lang.RuntimeException: Error caching map.xml: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive-root/root/935624e0-aea4-47d6-842c-32d42d506d4b/hive_2017-02-16_04-42-39_689_6740522155632742535-1/-mr-10004/7b69d4eb-6fe2-4c55-a6cd-ba4dcd5c2054/map.xml could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:183) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:399) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:776) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:714) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive-root/root/935624e0-aea4-47d6-842c-32d42d506d4b/hive_2017-02-16_04-42-39_689_6740522155632742535-1/-mr-10004/7b69d4eb-6fe2-4c55-a6cd-ba4dcd5c2054/map.xml could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) at org.apache.hadoop.ipc.Client.call(Client.java:1475) at org.apache.hadoop.ipc.Client.call(Client.java:1412) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy31.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy32.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1455) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1251) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:448) Job Submission failed with exception 'java.lang.RuntimeException(Error caching map.xml: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive-root/root/935624e0-aea4-47d6-842c-32d42d506d4b/hive_2017-02-16_04-42-39_689_6740522155632742535-1/-mr-10004/7b69d4eb-6fe2-4c55-a6cd-ba4dcd5c2054/map.xml could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) )' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. Error caching map.xml: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hive-root/root/935624e0-aea4-47d6-842c-32d42d506d4b/hive_2017-02-16_04-42-39_689_6740522155632742535-1/-mr-10004/7b69d4eb-6fe2-4c55-a6cd-ba4dcd5c2054/map.xml could only be replicated to 0 nodes instead of minReplication (=1). There are 2 datanode(s) running and 2 node(s) are excluded in this operation. at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1571) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getNewBlockTargets(FSNamesystem.java:3107) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3031) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:725) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:492) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
hadoop MapReduce 路径输入
MapReduce程序中要处理的文件在一个文件夹及它的子文件夹中,用什么方法可以处理这种情况,让所有的文件都能被处理
hadoop mapreduce 能帮我解决什么问题?
# 求救。。。 在mapreduce下,能够进行一次map后,对map的结果进行多次reduce呢?
eclipse 开发 hadoop mapreduce的方式
我是在windows下用eclipse下开发mapreduce 然后打成jar包,在linux下,用命令运行jar,这种方式好吗,有其他方式吗?
hadoop mapreduce 在编写好的程序下 运行程序出现错误,求错误所在
15/09/01 10:05:06 INFO mapred.JobClient: map 0% reduce 0% 15/09/01 10:05:22 INFO mapred.JobClient: Task Id : attempt_201509011003_0001_m_000002_0, Status : FAILED java.util.NoSuchElementException at java.util.StringTokenizer.nextToken(StringTokenizer.java:332) at com.hebut.mr.Score$Map.map(Score.java:37) at com.hebut.mr.Score$Map.map(Score.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249) 15/09/01 10:05:25 WARN mapred.JobClient: Error reading task outputSlave1.hadoop 15/09/01 10:05:25 WARN mapred.JobClient: Error reading task outputSlave1.hadoop 15/09/01 10:05:26 INFO mapred.JobClient: Task Id : attempt_201509011003_0001_m_000000_0, Status : FAILED java.util.NoSuchElementException at java.util.StringTokenizer.nextToken(StringTokenizer.java:332) at com.hebut.mr.Score$Map.map(Score.java:37) at com.hebut.mr.Score$Map.map(Score.java:1) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.mapred.Child.main(Child.java:249)
cdh hadoop mapreduce 运行时的问题:(有时候会出现,有时候不出现,急求大神帮助)
15/10/08 08:49:13 INFO mapreduce.Job: Job job_1419225162729_18465 running in uber mode : false 15/10/08 08:49:13 INFO mapreduce.Job: map 0% reduce 0% 15/10/08 08:49:13 INFO mapreduce.Job: Job job_1419225162729_18465 failed with state FAILED due to: Application application_1419225162729_18465 failed 1 times due to AM Container for appattempt_1419225162729_18465_000001 exited with exitCode: -1000 due to: java.io.IOException: Not able to initialize app-log directories in any of the configured local directories for app application_1419225162729_18465 at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.createAppLogDirs(DefaultContainerExecutor.java:459) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:91) at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:861) .Failing this attempt.. Failing the application. 15/10/08 08:49:13 INFO mapreduce.Job: Counters: 0 Moved: 'hdfs://oiddhnode02:8020/user/nmger/worktemp/2015100408' to trash at: hdfs://oiddhnode02:8020/user/nmger/.Trash/Current
hadoop mapreduce 统计所有的key-value中value为空的数目
遇到一个问题,望大家有空可以帮忙看看,小弟感激不尽! 现在在做信令解析,有一条信令:[2016-04-02 09:58:09,724] len:78;type:1002;msc:0E1F;bsc:3F17;time:2016-04-01 16:48:46.494;lac:13883;ci:8713;imsi:460004544938252;msisdn:13994482976;callType:0;disLen:11;disMsisdn:13503531697;remark:0; map0: 输入:一条信令(上面那一条) 通过解析,得到一个Object 输出:key:imsi value:{imsi:460004544938252,msisdn:13994482976} reduce0: 输入:key:imsi value:List<> {imsi:460004544938252,msisdn:''},{imsi:460004544938252,msisdn:13994482976},{imsi:460004544938252,msisdn:''} 在这里我想统计一下: 1、所有的msisdn为空的个数 2、imsi对应的msisdn为空的个数(List里的msisdn全为空计1) 现在问题是:要把统计的值(一行:日期 | 呼叫总量 | IMSI未解析总呼叫次数 | 占比1 | 总IMSI | 未解析总IMSI数 | 占比2)写到文件中,怎么判断我读的是最后一条信令呢?如果不行的话,把这个reduce的结果传给第二个mapreduce统计怎么实现呢? 希望有空的朋友多多指点!谢谢
hadoop一个mapreduce的JOB最短执行时间
如题,我想用hadoop来进行文本检索,想法是一个查询对应一个JOB,检索的话肯定时间要快。 但是我在eclipse中跑一个JOB时,即使是什么都不做,也需要7秒,用hadoop jar命令更久。 请问这个时间可以优化吗,还是Mapreduce初始JOB就需要这么久。还有一个奇怪的现象: JOB如果遍历文本集合来进行检索,竟然只需要6秒多,比什么都不干还快。
运行PI实例检查Hadoop集群是否搭建成功时显示“bash:hadoop: command not found”该怎么解决?
运行PI实例检查Hadoop集群是否搭建成功时执行命令: [hfut@master ~]$ cd ~/hadoop-2.5.2/share/hadoop/mapreduce/ [hfut@master mapreduce]$ hadoop jar home/hfut/hadoop-2.5.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.5.2.jar pi 10 10 第二条命令执行完毕后显示bash:hadoop: command not found 如下图所示: ![图片说明](https://img-ask.csdn.net/upload/202001/30/1580370011_357824.png) 请问该怎么解决?
用eclipse连接虚拟机hadoop集群执行MapReduce程序,但是报以下错误,请问如何解决?
# 说明:eclipse中关于hadoop的各项advanced parameter参数均已按配置文件进行配置。但在执行过程中还是报如下错误,请问如何解决。 # 执行日志: 2018-09-22 22:59:11,429 INFO [org.apache.commons.beanutils.FluentPropertyBeanIntrospector] - Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property. 2018-09-22 22:59:11,443 WARN [org.apache.hadoop.metrics2.impl.MetricsConfig] - Cannot locate configuration: tried hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties 2018-09-22 22:59:11,496 INFO [org.apache.hadoop.metrics2.impl.MetricsSystemImpl] - Scheduled Metric snapshot period at 10 second(s). 2018-09-22 22:59:11,496 INFO [org.apache.hadoop.metrics2.impl.MetricsSystemImpl] - JobTracker metrics system started 2018-09-22 22:59:20,863 WARN [org.apache.hadoop.mapreduce.JobResourceUploader] - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2018-09-22 22:59:20,879 WARN [org.apache.hadoop.mapreduce.JobResourceUploader] - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2018-09-22 22:59:20,928 INFO [org.apache.hadoop.mapreduce.lib.input.FileInputFormat] - Total input files to process : 1 2018-09-22 22:59:20,984 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - number of splits:1 2018-09-22 22:59:21,072 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Submitting tokens for job: job_local1513265977_0001 2018-09-22 22:59:21,074 INFO [org.apache.hadoop.mapreduce.JobSubmitter] - Executing with tokens: [] 2018-09-22 22:59:21,950 INFO [org.apache.hadoop.mapred.LocalDistributedCacheManager] - Creating symlink: \tmp\hadoop-启政先生\mapred\local\1537628361150\movies.csv <- G:\java_workspace\MapReduce_DEMO/movies.csv 2018-09-22 22:59:21,995 WARN [org.apache.hadoop.fs.FileUtil] - Command 'E:\hadoop-3.0.0\bin\winutils.exe symlink G:\java_workspace\MapReduce_DEMO\movies.csv \tmp\hadoop-启政先生\mapred\local\1537628361150\movies.csv' failed 1 with: CreateSymbolicLink error (1314): ??????????? 2018-09-22 22:59:21,995 WARN [org.apache.hadoop.mapred.LocalDistributedCacheManager] - Failed to create symlink: \tmp\hadoop-启政先生\mapred\local\1537628361150\movies.csv <- G:\java_workspace\MapReduce_DEMO/movies.csv 2018-09-22 22:59:21,996 INFO [org.apache.hadoop.mapred.LocalDistributedCacheManager] - Localized hdfs://192.168.5.110:9000/temp/input/movies.csv as file:/tmp/hadoop-启政先生/mapred/local/1537628361150/movies.csv 2018-09-22 22:59:22,046 INFO [org.apache.hadoop.mapreduce.Job] - The url to track the job: http://localhost:8080/ 2018-09-22 22:59:22,047 INFO [org.apache.hadoop.mapreduce.Job] - Running job: job_local1513265977_0001 2018-09-22 22:59:22,047 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter set in config null 2018-09-22 22:59:22,051 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - File Output Committer Algorithm version is 2 2018-09-22 22:59:22,051 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 2018-09-22 22:59:22,052 INFO [org.apache.hadoop.mapred.LocalJobRunner] - OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter 2018-09-22 22:59:22,100 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Waiting for map tasks 2018-09-22 22:59:22,101 INFO [org.apache.hadoop.mapred.LocalJobRunner] - Starting task: attempt_local1513265977_0001_m_000000_0 2018-09-22 22:59:22,120 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - File Output Committer Algorithm version is 2 2018-09-22 22:59:22,120 INFO [org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter] - FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false 2018-09-22 22:59:22,128 INFO [org.apache.hadoop.yarn.util.ProcfsBasedProcessTree] - ProcfsBasedProcessTree currently is supported only on Linux. 2018-09-22 22:59:22,169 INFO [org.apache.hadoop.mapred.Task] - Using ResourceCalculatorProcessTree : org.apache.hadoop.yarn.util.WindowsBasedProcessTree@7ef907ef 2018-09-22 22:59:22,172 INFO [org.apache.hadoop.mapred.MapTask] - Processing split: hdfs://192.168.5.110:9000/temp/input/ratings.csv:0+2438233 ----------cachePath=/temp/input/movies.csv---------- 2018-09-22 22:59:22,226 INFO [org.apache.hadoop.mapred.LocalJobRunner] - map task executor complete. 2018-09-22 22:59:22,233 WARN [org.apache.hadoop.mapred.LocalJobRunner] - job_local1513265977_0001 java.lang.Exception: java.io.FileNotFoundException: \temp\input\movies.csv (系统找不到指定的路径。) at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552) Caused by: java.io.FileNotFoundException: \temp\input\movies.csv (系统找不到指定的路径。) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(Unknown Source) at java.io.FileInputStream.<init>(Unknown Source) at java.io.FileInputStream.<init>(Unknown Source) at java.io.FileReader.<init>(Unknown Source) at MovieJoinExercise1.MovieJoin$MovieJoinMapper.setup(MovieJoin.java:79) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:794) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 2018-09-22 22:59:23,051 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1513265977_0001 running in uber mode : false 2018-09-22 22:59:23,052 INFO [org.apache.hadoop.mapreduce.Job] - map 0% reduce 0% 2018-09-22 22:59:23,053 INFO [org.apache.hadoop.mapreduce.Job] - Job job_local1513265977_0001 failed with state FAILED due to: NA 2018-09-22 22:59:23,058 INFO [org.apache.hadoop.mapreduce.Job] - Counters: 0
运行mapredurce出现Method threw 'java.lang.IllegalStateException' exception. Cannot evaluate org.apache.hadoop.mapreduce.Job.toString()
执行下述代码后在,创建job后会有上述异常,但是可以执行到最后,但是job没有提交上去执行,在历史里也看不到有执行记录求帮助新手o(╥﹏╥)o。 package MapReducer; import com.sun.org.apache.bcel.internal.generic.RETURN; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.conf.Configured; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.input.TextInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; import org.apache.hadoop.util.Tool; import org.apache.hadoop.util.ToolRunner; import java.io.File; import java.io.IOException; import java.net.URI; import java.net.URISyntaxException; import java.util.StringTokenizer; /** * @Describe MapReducer第一个读取文档并计数 * @Author zhanglei * @Date 2019/11/18 22:53 **/ public class WordCountApp extends Configured implements Tool { public int run(String[] strings) throws Exception { String input_path="hdfs://192.168.91.130:8020/data/wc.txt"; String output_path="hdfs://192.168.91.130:8020/data/outputwc"; Configuration configuration = getConf(); final FileSystem fileSystem = FileSystem.get(new URI(input_path),configuration); if(fileSystem.exists(new Path(output_path))){ fileSystem.delete(new Path(output_path),true); } Job job = Job.getInstance(configuration,"WordCountApp"); job.setJarByClass(WordCountApp.class); job.setMapperClass(WordCountMapper.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); job.setReducerClass(WordCountReducer.class); job.setInputFormatClass(TextInputFormat.class); Path inpath = new Path(input_path); FileInputFormat.addInputPath(job,inpath); job.setOutputFormatClass(TextOutputFormat.class); Path outpath = new Path(output_path); FileOutputFormat.setOutputPath(job,outpath); return job.waitForCompletion(true) ? 0:1; } //继承 public static class WordCountMapper extends Mapper<Object,Text,Text,IntWritable>{ private final static IntWritable one= new IntWritable(1); private Text word = new Text(); public void map(Object key,Text value,Context context) throws IOException, InterruptedException { Text t = value; StringTokenizer itr = new StringTokenizer(value.toString()); while(itr.hasMoreTokens()){ word.set(itr.nextToken()); context.write(word,one); } } } public static class WordCountReducer extends Reducer<Object,Text,Text,IntWritable>{ private final static IntWritable res= new IntWritable(1); public void reduce(Text key,Iterable<IntWritable> values,Context context) throws IOException, InterruptedException { int sum = 0; for(IntWritable val:values){ sum+=val.get(); } res.set(sum); context.write(key,res); } } public static void main(String[] args) throws Exception { int exitCode = ToolRunner.run(new WordCountApp(), args); System.exit(exitCode); } }
自己写的hadoop ,MapReduce程序不能并行
** 学习hadoop有一段时间了,在写hadoop 程序时,尽管是根据官方例子,套着模板写出的,但是不能达到真正意义上的并行,也就是说,各分机没有任务运行。 ------------------------------------------------------------------------ ** 运行环境如下: 操作系统: centOS6.3 32位, jdk1.7, hadoop-1.0.3, 1台master,3台worker。 为了具体说明问题,程序如下: package campus; import java.io.IOException; import java.net.URI; import java.util.Set; import java.util.TreeSet; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.Text; import org.apache.hadoop.io.compress.CompressionCodec; import org.apache.hadoop.io.compress.GzipCodec; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.Reducer.Context; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; public class TestSmsMR { // map public static class TSmsMap extends Mapper<Object, Text, Text, Text> { private static Text keyWord = new Text(); //第一个结点 private static Text valueWord = new Text(); //第二个结点 public void map(Object key, Text value, Context context) { // value: tag: 0 1 u String line = value.toString(); String[] arr = line.split(" |\u0009|\\|"); // 通过 空格、 \t、 | 分割字符串 if ( !(arr[0].equals(arr[1])) ) { try { String tmpKey = arr[0]; String tmpValue = ""; for(int i = 1; i < arr.length; i ++){ tmpValue += arr[i] + " "; } keyWord.set(tmpKey); valueWord.set(tmpValue); // 数据是非对称的,这就需要使用一次 write context.write(keyWord, valueWord); // context.write(valueWord, keyWord); //添加这句的话,必须先看图文件,如果重复则不需要这一行 } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } // 这种方法可行 } } } //reduce write<Text,Set<Text>> public static class TSmsReduce extends Reducer<Text, Text, Text, Text> { private static Text keyStr = new Text(); private static Text valueStr = new Text(); public void reduce(Text key, Iterable<Text> values,Context context) { String writeKey = key.toString(); String writeValues = ""; for (Text val : values) { writeValues += val.toString() + "\t"; } keyStr.set(writeKey); valueStr.set(writeValues); // System.out.println("writeKey: " + writeKey + "\twriteValues: " + writeValues); try { context.write(keyStr, valueStr); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } catch (InterruptedException e) { // TODO Auto-generated catch block e.printStackTrace(); } } } public static void preExectue(String inputPath, String outputPath) throws Exception { Configuration conf = new Configuration(); // conf.setBoolean("mapred.compress.map.output", true); conf.setBoolean("mapred.output.compress", true); // conf.setIfUnset("mapred.map.output.compression.type", "BLOCK"); conf.setClass("mapred.map.output.compression.codec", GzipCodec.class, CompressionCodec.class); conf.addResource(new Path("/usr/hadoop/conf/core-site.xml")); conf.addResource(new Path("/usr/hadoop/conf/hdfs-site.xml")); // 如果 outputPath 存在,那么先删除 Path outPutPath = new Path(outputPath); FileSystem fs = FileSystem.get(URI.create(outputPath), conf); fs.delete(outPutPath); // 自己添加路径 String[] ars = new String[] { inputPath, outputPath }; String[] otherArgs = new GenericOptionsParser(conf, ars) .getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: sotTest <in> <out>"); System.exit(2); } Job job = new Job(conf, "TestSmsMR"); job.setJarByClass(TestSmsMR.class); job.setMapperClass(TSmsMap.class); job.setReducerClass(TSmsReduce.class); // job.setNumReduceTasks(4); job.setOutputKeyClass(Text.class); job.setOutputValueClass(Text.class); // FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); if (job.waitForCompletion(true)) { System.out.println("The preprocess mapreduce has finished!"); } } //main函数测试都好着,为什么就不能并行呢 public static void main(String[] args) throws Exception { Long startTime = System.currentTimeMillis(); String srcPath = "campusSms"; String dstPath = "campusSmsLabelOut"; preExectue(srcPath,dstPath); Long runTime = System.currentTimeMillis() - startTime; System.out.println("run time: " + runTime); } } 还是觉得问题出在这个函数上: public static void preExectue(String inputPath, String outputPath) 运行前提是: 环境已搭建好,而且测试主机分机都能正常通信,且主机从机都起来了。希望解答时,能多考虑些编程方面的问题。 该程序运行起来,就是在主机上跑,MapReduce机制到分机并没有得到任务,运行的数据250M,按照hadoop默认块64M来算,也应该分为4块,3台分机应得到执行任务的,可是程序就是不能并行。 请有经验的hadoop学习实践者给予指导。 在集群上运行Pi值计算,都能看到并行。就是自己写的程序,不行!......
hadoop单词统计报错Job job_1581768459583_0001 failed
3个节点hadoop01、hadoop02、hadoop03 hadoop01是主节点 hadoop01、hadoop02、hadoop03是从节点,目前集群已搭建好,jps查看三个节点运行都很正常,而且UI也能正常显示,但是使用hadoop自带的hadoop-mapreduce-examples-2.7.4.jar的wordcount进行单词统计时报错如下,请高人指点,看不懂呀: ```[root@hadoop01 mapreduce]# hadoop jar hadoop-mapreduce-examples-2.7.4.jar wordcount /wordcount/input /wordcount/output 20/02/15 20:14:25 INFO client.RMProxy: Connecting to ResourceManager at hadoop01/192.168.233.132:8032 20/02/15 20:14:27 INFO input.FileInputFormat: Total input paths to process : 1 20/02/15 20:14:27 INFO mapreduce.JobSubmitter: number of splits:1 20/02/15 20:14:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1581768459583_0001 20/02/15 20:14:28 INFO impl.YarnClientImpl: Submitted application application_1581768459583_0001 20/02/15 20:14:28 INFO mapreduce.Job: The url to track the job: http://hadoop01:8088/proxy/application_1581768459583_0001/ 20/02/15 20:14:28 INFO mapreduce.Job: Running job: job_1581768459583_0001 20/02/15 20:15:38 INFO mapreduce.Job: Job job_1581768459583_0001 running in uber mode : false 20/02/15 20:15:38 INFO mapreduce.Job: map 0% reduce 0% 20/02/15 20:15:38 INFO mapreduce.Job: Job job_1581768459583_0001 failed with state FAILED due to: Application application_1581768459583_0001 failed 2 times due to Error launching appattempt_1581768459583_0001_000002. Got exception: java.io.IOException: Failed on local exception: java.io.IOException: java.io.IOException: Connection reset by peer; Host Details : local host is: "hadoop01.com/79.124.78.101"; destination host is: "79.124.78.101":43276; at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1413) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy83.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy84.startContainers(Unknown Source) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:250) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.IOException: java.io.IOException: Connection reset by peer at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:651) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:738) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:376) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1452) ... 16 more Caused by: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read(BufferedInputStream.java:265) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:561) at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:376) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:730) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:726) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:726) ... 19 more . Failing the application. 20/02/15 20:15:38 INFO mapreduce.Job: Counters: 0 ```
终于明白阿里百度这样的大公司,为什么面试经常拿ThreadLocal考验求职者了
点击上面↑「爱开发」关注我们每晚10点,捕获技术思考和创业资源洞察什么是ThreadLocalThreadLocal是一个本地线程副本变量工具类,各个线程都拥有一份线程私...
《奇巧淫技》系列-python!!每天早上八点自动发送天气预报邮件到QQ邮箱
将代码部署服务器,每日早上定时获取到天气数据,并发送到邮箱。 也可以说是一个小人工智障。 思路可以运用在不同地方,主要介绍的是思路。
面试官问我:什么是消息队列?什么场景需要他?用了会出现什么问题?
你知道的越多,你不知道的越多 点赞再看,养成习惯 GitHub上已经开源 https://github.com/JavaFamily 有一线大厂面试点脑图、个人联系方式和人才交流群,欢迎Star和完善 前言 消息队列在互联网技术存储方面使用如此广泛,几乎所有的后端技术面试官都要在消息队列的使用和原理方面对小伙伴们进行360°的刁难。 作为一个在互联网公司面一次拿一次Offer的面霸...
8年经验面试官详解 Java 面试秘诀
作者 |胡书敏 责编 | 刘静 出品 | CSDN(ID:CSDNnews) 本人目前在一家知名外企担任架构师,而且最近八年来,在多家外企和互联网公司担任Java技术面试官,前后累计面试了有两三百位候选人。在本文里,就将结合本人的面试经验,针对Java初学者、Java初级开发和Java开发,给出若干准备简历和准备面试的建议。 Java程序员准备和投递简历的实...
究竟你适不适合买Mac?
我清晰的记得,刚买的macbook pro回到家,开机后第一件事情,就是上了淘宝网,花了500元钱,找了一个上门维修电脑的师傅,上门给我装了一个windows系统。。。。。。 表砍我。。。 当时买mac的初衷,只是想要个固态硬盘的笔记本,用来运行一些复杂的扑克软件。而看了当时所有的SSD笔记本后,最终决定,还是买个好(xiong)看(da)的。 已经有好几个朋友问我mba怎么样了,所以今天尽量客观...
MyBatis研习录(01)——MyBatis概述与入门
MyBatis 是一款优秀的持久层框架,它支持定制化 SQL、存储过程以及高级映射。MyBatis原本是apache的一个开源项目iBatis, 2010年该项目由apache software foundation 迁移到了google code并改名为MyBatis 。2013年11月MyBatis又迁移到Github。
程序员一般通过什么途径接私活?
二哥,你好,我想知道一般程序猿都如何接私活,我也想接,能告诉我一些方法吗? 上面是一个读者“烦不烦”问我的一个问题。其实不止是“烦不烦”,还有很多读者问过我类似这样的问题。 我接的私活不算多,挣到的钱也没有多少,加起来不到 20W。说实话,这个数目说出来我是有点心虚的,毕竟太少了,大家轻喷。但我想,恰好配得上“一般程序员”这个称号啊。毕竟苍蝇再小也是肉,我也算是有经验的人了。 唾弃接私活、做外...
Python爬虫爬取淘宝,京东商品信息
小编是一个理科生,不善长说一些废话。简单介绍下原理然后直接上代码。 使用的工具(Python+pycharm2019.3+selenium+xpath+chromedriver)其中要使用pycharm也可以私聊我selenium是一个框架可以通过pip下载 pip installselenium -ihttps://pypi.tuna.tsinghua.edu.cn/simple/ ...
阿里程序员写了一个新手都写不出的低级bug,被骂惨了。
这种新手都不会范的错,居然被一个工作好几年的小伙子写出来,差点被当场开除了。
Java工作4年来应聘要16K最后没要,细节如下。。。
前奏: 今天2B哥和大家分享一位前几天面试的一位应聘者,工作4年26岁,统招本科。 以下就是他的简历和面试情况。 基本情况: 专业技能: 1、&nbsp;熟悉Sping了解SpringMVC、SpringBoot、Mybatis等框架、了解SpringCloud微服务 2、&nbsp;熟悉常用项目管理工具:SVN、GIT、MAVEN、Jenkins 3、&nbsp;熟悉Nginx、tomca...
Python爬虫精简步骤1 获取数据
爬虫,从本质上来说,就是利用程序在网上拿到对我们有价值的数据。 爬虫能做很多事,能做商业分析,也能做生活助手,比如:分析北京近两年二手房成交均价是多少?广州的Python工程师平均薪资是多少?北京哪家餐厅粤菜最好吃?等等。 这是个人利用爬虫所做到的事情,而公司,同样可以利用爬虫来实现巨大的商业价值。比如你所熟悉的搜索引擎——百度和谷歌,它们的核心技术之一也是爬虫,而且是超级爬虫。 从搜索巨头到人工...
Python绘图,圣诞树,花,爱心 | Turtle篇
每周每日,分享Python实战代码,入门资料,进阶资料,基础语法,爬虫,数据分析,web网站,机器学习,深度学习等等。 公众号回复【进群】沟通交流吧,QQ扫码进群学习吧 微信群 QQ群 1.画圣诞树 import turtle screen = turtle.Screen() screen.setup(800,600) circle = turtle.Turtle()...
作为一个程序员,CPU的这些硬核知识你必须会!
CPU对每个程序员来说,是个既熟悉又陌生的东西? 如果你只知道CPU是中央处理器的话,那可能对你并没有什么用,那么作为程序员的我们,必须要搞懂的就是CPU这家伙是如何运行的,尤其要搞懂它里面的寄存器是怎么一回事,因为这将让你从底层明白程序的运行机制。 随我一起,来好好认识下CPU这货吧 把CPU掰开来看 对于CPU来说,我们首先就要搞明白它是怎么回事,也就是它的内部构造,当然,CPU那么牛的一个东...
破14亿,Python分析我国存在哪些人口危机!
一、背景 二、爬取数据 三、数据分析 1、总人口 2、男女人口比例 3、人口城镇化 4、人口增长率 5、人口老化(抚养比) 6、各省人口 7、世界人口 四、遇到的问题 遇到的问题 1、数据分页,需要获取从1949-2018年数据,观察到有近20年参数:LAST20,由此推测获取近70年的参数可设置为:LAST70 2、2019年数据没有放上去,可以手动添加上去 3、将数据进行 行列转换 4、列名...
web前端javascript+jquery知识点总结
1.Javascript 语法.用途 javascript 在前端网页中占有非常重要的地位,可以用于验证表单,制作特效等功能,它是一种描述语言,也是一种基于对象(Object)和事件驱动并具有安全性的脚本语言 ...
Python实战:抓肺炎疫情实时数据,画2019-nCoV疫情地图
今天,群里白垩老师问如何用python画武汉肺炎疫情地图。白垩老师是研究海洋生态与地球生物的学者,国家重点实验室成员,于不惑之年学习python,实为我等学习楷模。先前我并没有关注武汉肺炎的具体数据,也没有画过类似的数据分布图。于是就拿了两个小时,专门研究了一下,遂成此文。
听说想当黑客的都玩过这个Monyer游戏(1~14攻略)
第零关 进入传送门开始第0关(游戏链接) 请点击链接进入第1关: 连接在左边→ ←连接在右边 看不到啊。。。。(只能看到一堆大佬做完的留名,也能看到菜鸡的我,在后面~~) 直接fn+f12吧 &lt;span&gt;连接在左边→&lt;/span&gt; &lt;a href="first.php"&gt;&lt;/a&gt; &lt;span&gt;←连接在右边&lt;/span&gt; o...
在家远程办公效率低?那你一定要收好这个「在家办公」神器!
相信大家都已经收到国务院延长春节假期的消息,接下来,在家远程办公可能将会持续一段时间。 但是问题来了。远程办公不是人在电脑前就当坐班了,相反,对于沟通效率,文件协作,以及信息安全都有着极高的要求。有着非常多的挑战,比如: 1在异地互相不见面的会议上,如何提高沟通效率? 2文件之间的来往反馈如何做到及时性?如何保证信息安全? 3如何规划安排每天工作,以及如何进行成果验收? ...... ...
作为一个程序员,内存和磁盘的这些事情,你不得不知道啊!!!
截止目前,我已经分享了如下几篇文章: 一个程序在计算机中是如何运行的?超级干货!!! 作为一个程序员,CPU的这些硬核知识你必须会! 作为一个程序员,内存的这些硬核知识你必须懂! 这些知识可以说是我们之前都不太重视的基础知识,可能大家在上大学的时候都学习过了,但是嘞,当时由于老师讲解的没那么有趣,又加上这些知识本身就比较枯燥,所以嘞,大家当初几乎等于没学。 再说啦,学习这些,也看不出来有什么用啊!...
渗透测试-灰鸽子远控木马
木马概述 灰鸽子( Huigezi),原本该软件适用于公司和家庭管理,其功能十分强大,不但能监视摄像头、键盘记录、监控桌面、文件操作等。还提供了黑客专用功能,如:伪装系统图标、随意更换启动项名称和表述、随意更换端口、运行后自删除、毫无提示安装等,并采用反弹链接这种缺陷设计,使得使用者拥有最高权限,一经破解即无法控制。最终导致被黑客恶意使用。原作者的灰鸽子被定义为是一款集多种控制方式于一体的木马程序...
Python:爬取疫情每日数据
前言 目前每天各大平台,如腾讯、今日头条都会更新疫情每日数据,他们的数据源都是一样的,主要都是通过各地的卫健委官网通报。 以全国、湖北和上海为例,分别为以下三个网站: 国家卫健委官网:http://www.nhc.gov.cn/xcs/yqtb/list_gzbd.shtml 湖北卫健委官网:http://wjw.hubei.gov.cn/bmdt/ztzl/fkxxgzbdgrfyyq/xxfb...
这个世界上人真的分三六九等,你信吗?
偶然间,在知乎上看到一个问题 一时间,勾起了我深深的回忆。 以前在厂里打过两次工,做过家教,干过辅导班,做过中介。零下几度的晚上,贴过广告,满脸、满手地长冻疮。 再回首那段岁月,虽然苦,但让我学会了坚持和忍耐。让我明白了,在这个世界上,无论环境多么的恶劣,只要心存希望,星星之火,亦可燎原。 下文是原回答,希望能对你能有所启发。 如果我说,这个世界上人真的分三六九等,...
B 站上有哪些很好的学习资源?
哇说起B站,在小九眼里就是宝藏般的存在,放年假宅在家时一天刷6、7个小时不在话下,更别提今年的跨年晚会,我简直是跪着看完的!! 最早大家聚在在B站是为了追番,再后来我在上面刷欧美新歌和漂亮小姐姐的舞蹈视频,最近两年我和周围的朋友们已经把B站当作学习教室了,而且学习成本还免费,真是个励志的好平台ヽ(.◕ฺˇд ˇ◕ฺ;)ノ 下面我们就来盘点一下B站上优质的学习资源: 综合类 Oeasy: 综合...
雷火神山直播超两亿,Web播放器事件监听是怎么实现的?
Web播放器解决了在手机浏览器和PC浏览器上播放音视频数据的问题,让视音频内容可以不依赖用户安装App,就能进行播放以及在社交平台进行传播。在视频业务大数据平台中,播放数据的统计分析非常重要,所以Web播放器在使用过程中,需要对其内部的数据进行收集并上报至服务端,此时,就需要对发生在其内部的一些播放行为进行事件监听。 那么Web播放器事件监听是怎么实现的呢? 01 监听事件明细表 名...
3万字总结,Mysql优化之精髓
本文知识点较多,篇幅较长,请耐心学习 MySQL已经成为时下关系型数据库产品的中坚力量,备受互联网大厂的青睐,出门面试想进BAT,想拿高工资,不会点MySQL优化知识,拿offer的成功率会大大下降。 为什么要优化 系统的吞吐量瓶颈往往出现在数据库的访问速度上 随着应用程序的运行,数据库的中的数据会越来越多,处理时间会相应变慢 数据是存放在磁盘上的,读写速度无法和内存相比 如何优化 设计...
Python新型冠状病毒疫情数据自动爬取+统计+发送报告+数据屏幕(三)发送篇
今天介绍的项目是使用 Itchat 发送统计报告 项目功能设计: 定时爬取疫情数据存入Mysql 进行数据分析制作疫情报告 使用itchat给亲人朋友发送分析报告 基于Django做数据屏幕 使用Tableau做数据分析 来看看最终效果 目前已经完成,预计2月12日前更新 使用 itchat 发送数据统计报告 itchat 是一个基于 web微信的一个框架,但微信官方并不允许使用这...
作为程序员的我,大学四年一直自学,全靠这些实用工具和学习网站!
我本人因为高中沉迷于爱情,导致学业荒废,后来高考,毫无疑问进入了一所普普通通的大学,实在惭愧???? 我又是那么好强,现在学历不行,没办法改变的事情了,所以,进入大学开始,我就下定决心,一定要让自己掌握更多的技能,尤其选择了计算机这个行业,一定要多学习技术。 在进入大学学习不久后,我就认清了一个现实:我这个大学的整体教学质量和学习风气,真的一言难尽,懂的人自然知道怎么回事? 怎么办?我该如何更好的提升自...
粒子群算法求解物流配送路线问题(python)
1.Matlab实现粒子群算法的程序代码:https://www.cnblogs.com/kexinxin/p/9858664.html matlab代码求解函数最优值:https://blog.csdn.net/zyqblog/article/details/80829043 讲解通俗易懂,有数学实例的博文:https://blog.csdn.net/daaikuaichuan/article/...
教你如何编写第一个简单的爬虫
很多人知道爬虫,也很想利用爬虫去爬取自己想要的数据,那么爬虫到底怎么用呢?今天就教大家编写一个简单的爬虫。 下面以爬取笔者的个人博客网站为例获取第一篇文章的标题名称,教大家学会一个简单的爬虫。 第一步:获取页面 #!/usr/bin/python # coding: utf-8 import requests #引入包requests link = "http://www.santostang....
前端JS初级面试题二 (。•ˇ‸ˇ•。)老铁们!快来瞧瞧自己都会了么
1. 传统事件绑定和符合W3C标准的事件绑定有什么区别? 传统事件绑定 &lt;div onclick=""&gt;123&lt;/div&gt; div1.onclick = function(){}; &lt;button onmouseover=""&gt;&lt;/button&gt; 注意: 如果给同一个元素绑定了两次或多次相同类型的事件,那么后面的绑定会覆盖前面的绑定 (不支持DOM事...
相关热词 c#导入fbx c#中屏蔽键盘某个键 c#正态概率密度 c#和数据库登陆界面设计 c# 高斯消去法 c# codedom c#读取cad文件文本 c# 控制全局鼠标移动 c# temp 目录 bytes初始化 c#
立即提问