hadoop 运行本地jar包无法读取文件。

在本地运行时可以读取workspace中的txt文件,将java文件和txt一起打包之后上传至Hadoop运行,jar包中的txt文件没有被读取也没有报错。

2个回答

使用-files option加上本地文件路径即可。

你本地读取的时候文件路径是本地的,但是上传到hadoop以后,txt可能都不跟你的jar在一个机器上,你需要用hadoop的路径去读取

qq_35516286
qq_35516286 不管路径是Hadoop中的路径还是jar包中的路径,都无法读取
一年多之前 回复
Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
其他相关推荐
spark读取hdfs中lzo文件时hadoop版本冲突
各位大神跪求lzo-hadoop.jar支持hadoop-2.6版本的,或者是解决方法,本人想要用spark读取hdfs中*.lzo格式的压缩文件, 但是当前lzo-hadoop.jar包只支持hadoop-1.2.1,跪求解决办法!很急在线等!!!! 邮箱island_lonely@163.com
Hadoop运行WordCount程序有输入,不生成输出的文件
![图片说明](https://img-ask.csdn.net/upload/201812/21/1545407846_986988.jpg) 代码如下: import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.util.GenericOptionsParser; import java.io.IOException; import java.util.Iterator; import java.util.StringTokenizer; public class WordCountMapReduce { public static void main(String[] args) throws Exception{ Configuration configuration = new Configuration(); String[] otherArgs = (new GenericOptionsParser(configuration, args)).getRemainingArgs(); if (otherArgs.length < 2){ System.out.println("Usage:wordcount<in>[<in>···]<out>"); System.exit(2); } */ /** * 设置环境参数 *//* Job job = Job.getInstance(configuration, "wordcount"); */ /** * 设置整个程序的类名 *//* job.setJarByClass(WordCountMapReduce.class); */ /** * 添加Mapper类 *//* job.setMapperClass(WordCountMapReduce.WordCountMapper.class); */ /** * ? *//* //job.setCombinerClass(WordCountMapReduce.WordCountReducer.class); */ /** * 添加Reducer类 *//* job.setReducerClass(WordCountMapReduce.WordCountReducer.class); */ /** * 设置输出类型 *//* job.setOutputKeyClass(Text.class); */ /** * 设置输出类型 *//* job.setOutputValueClass(IntWritable.class); for (int i = 0;i < otherArgs.length - 1;++i){ */ /** * 设置输入文件 *//* FileInputFormat.addInputPath(job, new Path(otherArgs[i])); } */ /** * 设置输出文件 *//* FileOutputFormat.setOutputPath(job, new Path(otherArgs[otherArgs.length - 1])); System.exit(job.waitForCompletion(true)?0:1); } //map程序 public static class WordCountMapper extends Mapper<Object, Text, Text, IntWritable> { */ /* * map方法是提供给map task进程来调用的,map task进程是每读取一行文本来调用一次我们自定义的map方法 * map task在调用map方法时,传递的参数: * 一行的起始偏移量LongWritable作为key * 一行的文本内容Text作为value *//* private static final IntWritable one = new IntWritable(1); private Text word = new Text(); public WordCountMapper() { } @Override protected void map(Object key, Text value, Mapper<Object, Text, Text, IntWritable>.Context context) throws IOException, InterruptedException { StringTokenizer stringTokenizer = new StringTokenizer(value.toString()); while (stringTokenizer.hasMoreTokens()) { this.word.set(stringTokenizer.nextToken()); context.write(this.word, one); } } } //reduce程序 */ /* * KEYIN:对应mapper阶段输出的key类型 * VALUEIN:对应mapper阶段输出的value类型 * KEYOUT:reduce处理完之后输出的结果kv对中key的类型 * VALUEOUT:reduce处理完之后输出的结果kv对中value的类型 *//* public class WordCountReducer extends Reducer<Text, IntWritable, Text, IntWritable> { */ /* * reduce方法提供给reduce task进程来调用 * * reduce task会将shuffle阶段分发过来的大量kv数据对进行聚合,聚合的机制是相同key的kv对聚合为一组 * 然后reduce task对每一组聚合kv调用一次我们自定义的reduce方法 * 比如:<hello,1><hello,1><hello,1><tom,1><tom,1><tom,1> * hello组会调用一次reduce方法进行处理,tom组也会调用一次reduce方法进行处理 * 调用时传递的参数: * key:一组kv中的key * values:一组kv中所有value的迭代器 *//* private IntWritable intWritable = new IntWritable(); public WordCountReducer(){ } public void intWritable(Text key, Iterable<IntWritable>values, Reducer<Text, IntWritable, Text, IntWritable>.Context context)throws IOException, InterruptedException{ int sum = 0; IntWritable val; for (Iterator i$ = values.iterator(); i$.hasNext(); sum += val.get()){ val = (IntWritable)i$.next(); } this.intWritable.set(sum); context.write(key, this.intWritable); } } }
Android通过HDFS API 上传和读取文件
Android通过HDFS API 上传和读取文件需要什么jar包引入,然后要添加什么配置文件到项目,配在哪里,然后怎么连接上,如果可以给出核心的连接代码给我参考下,有劳大家了!!
Eclipse上运行MapReduce程序时,win10系统用户名中间有空格导致tmp文件生成&读取错误
报错如下: ``` SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/D:/Java/Dev/Maven/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.10.0/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/Java/Dev/Maven/.m2/repository/org/slf4j/slf4j-simple/1.6.6/slf4j-simple-1.6.6.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/D:/Java/Dev/Maven/.m2/repository/org/slf4j/slf4j-log4j12/1.7.25/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2019-09-05 10:27:02,488 WARN [main] impl.MetricsConfig (MetricsConfig.java:134) - Cannot locate configuration: tried hadoop-metrics2-jobtracker.properties,hadoop-metrics2.properties 2019-09-05 10:27:04,715 WARN [main] mapreduce.JobResourceUploader (JobResourceUploader.java:147) - Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 2019-09-05 10:27:04,743 WARN [main] mapreduce.JobResourceUploader (JobResourceUploader.java:480) - No job jar file set. User classes may not be found. See Job or Job#setJar(String). 2019-09-05 10:27:10,228 WARN [pool-8-thread-1] impl.MetricsSystemImpl (MetricsSystemImpl.java:151) - JobTracker metrics system already initialized! 2019-09-05 10:27:10,326 WARN [Thread-6] mapred.LocalJobRunner$Job (LocalJobRunner.java:590) - job_local64686135_0001 java.lang.Exception: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1 at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492) ~[hadoop-mapreduce-client-common-3.1.2.jar:?] at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:559) [hadoop-mapreduce-client-common-3.1.2.jar:?] Caused by: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in localfetcher#1 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:134) ~[hadoop-mapreduce-client-core-3.1.2.jar:?] at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:377) ~[hadoop-mapreduce-client-core-3.1.2.jar:?] at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:347) ~[hadoop-mapreduce-client-common-3.1.2.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_221] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_221] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_221] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_221] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_221] Caused by: java.io.FileNotFoundException: File D:/tmp/hadoop-William%20Scott/mapred/local/localRunner/icss/jobcache/job_local64686135_0001/attempt_local64686135_0001_m_000000_0/output/file.out.index does not exist at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641) ~[hadoop-common-3.1.2.jar:?] at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:930) ~[hadoop-common-3.1.2.jar:?] at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631) ~[hadoop-common-3.1.2.jar:?] at org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:211) ~[hadoop-common-3.1.2.jar:?] at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899) ~[hadoop-common-3.1.2.jar:?] at org.apache.hadoop.io.SecureIOUtils.openFSDataInputStream(SecureIOUtils.java:152) ~[hadoop-common-3.1.2.jar:?] at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:71) ~[hadoop-mapreduce-client-core-3.1.2.jar:?] at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:62) ~[hadoop-mapreduce-client-core-3.1.2.jar:?] at org.apache.hadoop.mapred.SpillRecord.<init>(SpillRecord.java:57) ~[hadoop-mapreduce-client-core-3.1.2.jar:?] at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.copyMapOutput(LocalFetcher.java:125) ~[hadoop-mapreduce-client-core-3.1.2.jar:?] at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.doCopy(LocalFetcher.java:103) ~[hadoop-mapreduce-client-core-3.1.2.jar:?] at org.apache.hadoop.mapreduce.task.reduce.LocalFetcher.run(LocalFetcher.java:86) ~[hadoop-mapreduce-client-core-3.1.2.jar:?] ``` 目前的情况是win10是用微软账号的登录的,姓名之间会自动生成一个空格,不是太方便更改账户。Hadoop运行环境是放在D盘的,但不是根目录。 请问有没有办法让本地的tmp文件换个地方生成,或者更改hadoop-William%20Scott文件夹的名字。 谢谢。
【急!】java访问hbase出错,无明显异常,无法读取对应表的信息(需求就是从Hbase之中的某张表读取信息)
有点着急请哪位大神帮忙一下 ## 本地Hosts配置(对应的机上面的hosts的文件配置都一样): -------------------------- 127.0.0.1 localhost 192.168.0.25 Master.Hadoop 192.168.0.26 Slave1.Hadoop 192.168.0.27 Slave2.Hadoop 192.168.0.28 Slave3.Hadoop ## hbase的基本信息: -------------------------- http://master.hadoop:60010/master-status HBase Root Directory: hdfs://Master.Hadoop:9000/hbase ## 代码 -------------------------- Configuration configuration = HBaseConfiguration.create(); configuration = HBaseConfiguration.create(); configuration.set("hbase.zookeeper.property.clientPort", "2181"); configuration.set("hbase.zookeeper.quorum", "Master.Hadoop,Slave1.Hadoop,Slave2.Hadoop,Slave1.Hadoop"); configuration.set("hbase.master", "Master.Hadoop:60010"); //"Master.Hadoop:60010" System.setProperty("hadoop.home.dir", "C:\\Program Files\\hadoop\\hadoop-common-2.2.0-bin-master"); connection = ConnectionFactory.createConnection(configuration); System.out.println( connection ); Admin admin = connection.getAdmin(); System.out.println( admin ); TableName tableName1 = TableName.valueOf("hbase:meta"); System.out.println( tableName1 ); System.out.println( admin.tableExists(tableName1) ); Table table = connection.getTable(tableName1); HTableDescriptor a = table.getTableDescriptor(); System.out.println( a ); ## 日志信息: -------------------------- "C:\Program Files\Java\jdk1.8.0_181\bin\java.exe" "-javaagent:C:\Program Files\JetBrains\IntelliJ IDEA 2018.2.3\lib\idea_rt.jar=57269:C:\Program Files\JetBrains\IntelliJ IDEA 2018.2.3\bin" -Dfile.encoding=UTF-8 -classpath "C:\Program Files\Java\jdk1.8.0_181\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\rt.jar;C:\Users\Administrator\IdeaProjects\TestHbase\out\production\TestHbase;D:\lib\xz-1.0.jar;D:\lib\asm-3.1.jar;D:\lib\avro-1.7.4.jar;D:\lib\common-1.0.jar;D:\lib\domain-1.0.jar;D:\lib\jfinal-3.1.jar;D:\lib\joni-2.1.2.jar;D:\lib\noggit-0.6.jar;D:\lib\jsch-0.1.42.jar;D:\lib\service-1.0.jar;D:\lib\xmlenc-0.52.jar;D:\lib\druid-1.0.31.jar;D:\lib\guava-12.0.1.jar;D:\lib\jcifs-1.3.17.jar;D:\lib\jetty-6.1.26.jar;D:\lib\jsr305-1.3.9.jar;D:\lib\log4j-1.2.16.jar;D:\lib\cos-26Dec2008.jar;D:\lib\paranamer-2.3.jar;D:\lib\activation-1.1.jar;D:\lib\commons-el-1.0.jar;D:\lib\commons-io-2.4.jar;D:\lib\httpcore-4.4.4.jar;D:\lib\httpmime-4.4.1.jar;D:\lib\jaxb-api-2.2.2.jar;D:\lib\jcodings-1.0.8.jar;D:\lib\jsp-2.1-6.1.14.jar;D:\lib\stax-api-1.0-2.jar;D:\lib\cglib-nodep-3.1.jar;D:\lib\commons-cli-1.2.jar;D:\lib\commons-net-3.1.jar;D:\lib\disruptor-3.3.0.jar;D:\lib\fastjson-1.2.37.jar;D:\lib\jersey-core-1.9.jar;D:\lib\servlet-api-2.4.jar;D:\lib\slf4j-api-1.6.6.jar;D:\lib\stax2-api-3.1.4.jar;D:\lib\zookeeper-3.4.6.jar;D:\lib\commons-lang-2.6.jar;D:\lib\commons-math-2.2.jar;D:\lib\httpclient-4.5.2.jar;D:\lib\solr-solrj-6.1.0.jar;D:\lib\freemarker-2.3.23.jar;D:\lib\hadoop-auth-2.5.1.jar;D:\lib\hadoop-hdfs-2.7.4.jar;D:\lib\jersey-server-1.9.jar;D:\lib\jetty-util-6.1.26.jar;D:\lib\netty-3.6.2.Final.jar;D:\lib\api-util-1.0.0-M20.jar;D:\lib\commons-codec-1.11.jar;D:\lib\hbase-client-1.2.3.jar;D:\lib\hbase-common-1.2.3.jar;D:\lib\hbase-server-1.4.0.jar;D:\lib\jsp-api-2.1-6.1.14.jar;D:\lib\leveldbjni-all-1.8.jar;D:\lib\metrics-core-2.2.0.jar;D:\lib\metrics-core-3.1.2.jar;D:\lib\commons-logging-1.2.jar;D:\lib\commons-math3-3.1.1.jar;D:\lib\hadoop-client-2.7.4.jar;D:\lib\hadoop-common-2.5.1.jar;D:\lib\hbase-metrics-1.4.0.jar;D:\lib\jamon-runtime-2.4.1.jar;D:\lib\protobuf-java-2.5.0.jar;D:\lib\slf4j-log4j12-1.6.6.jar;D:\lib\snappy-java-1.0.4.1.jar;D:\lib\commons-digester-1.8.jar;D:\lib\hbase-protocol-1.2.3.jar;D:\lib\jackson-jaxrs-1.9.13.jar;D:\lib\jcl-over-slf4j-1.7.7.jar;D:\lib\commons-daemon-1.0.13.jar;D:\lib\hadoop-yarn-api-2.7.4.jar;D:\lib\hbase-procedure-1.4.0.jar;D:\lib\jasper-runtime-5.5.23.jar;D:\lib\api-asn1-api-1.0.0-M20.jar;D:\lib\commons-compress-1.4.1.jar;D:\lib\commons-httpclient-3.1.jar;D:\lib\jasper-compiler-5.5.23.jar;D:\lib\jetty-sslengine-6.1.26.jar;D:\lib\netty-all-4.0.23.Final.jar;D:\lib\servlet-api-2.5-6.1.14.jar;D:\lib\apacheds-i18n-2.0.0-M15.jar;D:\lib\commons-beanutils-1.7.0.jar;D:\lib\hbase-annotations-1.2.3.jar;D:\lib\hbase-metrics-api-1.4.0.jar;D:\lib\hbase-prefix-tree-1.4.0.jar;D:\lib\jackson-core-asl-1.9.13.jar;D:\lib\woodstox-core-asl-4.4.1.jar;D:\lib\hadoop-annotations-2.5.1.jar;D:\lib\hadoop-yarn-client-2.7.4.jar;D:\lib\hadoop-yarn-common-2.5.1.jar;D:\lib\hbase-common-1.4.0-tests.jar;D:\lib\commons-collections-3.2.2.jar;D:\lib\commons-configuration-1.6.jar;D:\lib\hbase-hadoop-compat-1.4.0.jar;D:\lib\jackson-mapper-asl-1.9.13.jar;D:\lib\hbase-hadoop2-compat-1.4.0.jar;D:\lib\mysql-connector-java-5.1.38.jar;D:\lib\commons-beanutils-core-1.8.0.jar;D:\lib\findbugs-annotations-1.3.9-1.jar;D:\lib\htrace-core-3.1.0-incubating.jar;D:\lib\hadoop-yarn-server-common-2.7.4.jar;D:\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\lib\hadoop-mapreduce-client-app-2.7.4.jar;D:\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\lib\hadoop-mapreduce-client-common-2.7.4.jar;D:\lib\hadoop-mapreduce-client-shuffle-2.7.4.jar;D:\lib\hadoop-mapreduce-client-jobclient-2.7.4.jar" TestHbase [DEBUG] [10:23:53] org.apache.hadoop.security.Groups - Creating new Groups object [DEBUG] [10:23:53] org.apache.hadoop.util.NativeCodeLoader - Trying to load the custom-built native-hadoop library... [DEBUG] [10:23:53] org.apache.hadoop.util.NativeCodeLoader - Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path [DEBUG] [10:23:53] org.apache.hadoop.util.NativeCodeLoader - java.library.path=C:\Program Files\Java\jdk1.8.0_181\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\ProgramData\Oracle\Java\javapath;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;. [WARN ] [10:23:53] org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable [DEBUG] [10:23:53] org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback - Falling back to shell based [DEBUG] [10:23:53] org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback - Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping [DEBUG] [10:23:53] org.apache.hadoop.security.Groups - Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000 [DEBUG] [10:23:53] org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[Rate of successful kerberos logins and latency (milliseconds)], valueName=Time) [DEBUG] [10:23:53] org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[Rate of failed kerberos logins and latency (milliseconds)], valueName=Time) [DEBUG] [10:23:53] org.apache.hadoop.metrics2.lib.MutableMetricsFactory - field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(about=, sampleName=Ops, always=false, type=DEFAULT, value=[GetGroups], valueName=Time) [DEBUG] [10:23:53] org.apache.hadoop.metrics2.impl.MetricsSystemImpl - UgiMetrics, User and group related metrics [DEBUG] [10:23:54] org.apache.hadoop.security.authentication.util.KerberosName - Kerberos krb5 configuration not found, setting default realm to empty [DEBUG] [10:23:54] org.apache.hadoop.security.UserGroupInformation - hadoop login [DEBUG] [10:23:54] org.apache.hadoop.security.UserGroupInformation - hadoop login commit [DEBUG] [10:23:54] org.apache.hadoop.security.UserGroupInformation - using local user:NTUserPrincipal: admin [DEBUG] [10:23:54] org.apache.hadoop.security.UserGroupInformation - UGI loginUser:admin (auth:SIMPLE) [INFO ] [10:23:55] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Process identifier=hconnection-0x17d0685f connecting to ZooKeeper ensemble=192.168.0.25:2181 [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:host.name=WIN-SSJFMH6ELVT [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_181 [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Oracle Corporation [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:java.home=C:\Program Files\Java\jdk1.8.0_181\jre [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=C:\Program Files\Java\jdk1.8.0_181\jre\lib\charsets.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\deploy.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\access-bridge-64.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\cldrdata.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\dnsns.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\jaccess.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\jfxrt.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\localedata.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\nashorn.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunec.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunjce_provider.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunmscapi.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\sunpkcs11.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\ext\zipfs.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\javaws.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jce.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jfr.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jfxswt.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\jsse.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\management-agent.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\plugin.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\resources.jar;C:\Program Files\Java\jdk1.8.0_181\jre\lib\rt.jar;C:\Users\Administrator\IdeaProjects\TestHbase\out\production\TestHbase;D:\lib\xz-1.0.jar;D:\lib\asm-3.1.jar;D:\lib\avro-1.7.4.jar;D:\lib\common-1.0.jar;D:\lib\domain-1.0.jar;D:\lib\jfinal-3.1.jar;D:\lib\joni-2.1.2.jar;D:\lib\noggit-0.6.jar;D:\lib\jsch-0.1.42.jar;D:\lib\service-1.0.jar;D:\lib\xmlenc-0.52.jar;D:\lib\druid-1.0.31.jar;D:\lib\guava-12.0.1.jar;D:\lib\jcifs-1.3.17.jar;D:\lib\jetty-6.1.26.jar;D:\lib\jsr305-1.3.9.jar;D:\lib\log4j-1.2.16.jar;D:\lib\cos-26Dec2008.jar;D:\lib\paranamer-2.3.jar;D:\lib\activation-1.1.jar;D:\lib\commons-el-1.0.jar;D:\lib\commons-io-2.4.jar;D:\lib\httpcore-4.4.4.jar;D:\lib\httpmime-4.4.1.jar;D:\lib\jaxb-api-2.2.2.jar;D:\lib\jcodings-1.0.8.jar;D:\lib\jsp-2.1-6.1.14.jar;D:\lib\stax-api-1.0-2.jar;D:\lib\cglib-nodep-3.1.jar;D:\lib\commons-cli-1.2.jar;D:\lib\commons-net-3.1.jar;D:\lib\disruptor-3.3.0.jar;D:\lib\fastjson-1.2.37.jar;D:\lib\jersey-core-1.9.jar;D:\lib\servlet-api-2.4.jar;D:\lib\slf4j-api-1.6.6.jar;D:\lib\stax2-api-3.1.4.jar;D:\lib\zookeeper-3.4.6.jar;D:\lib\commons-lang-2.6.jar;D:\lib\commons-math-2.2.jar;D:\lib\httpclient-4.5.2.jar;D:\lib\solr-solrj-6.1.0.jar;D:\lib\freemarker-2.3.23.jar;D:\lib\hadoop-auth-2.5.1.jar;D:\lib\hadoop-hdfs-2.7.4.jar;D:\lib\jersey-server-1.9.jar;D:\lib\jetty-util-6.1.26.jar;D:\lib\netty-3.6.2.Final.jar;D:\lib\api-util-1.0.0-M20.jar;D:\lib\commons-codec-1.11.jar;D:\lib\hbase-client-1.2.3.jar;D:\lib\hbase-common-1.2.3.jar;D:\lib\hbase-server-1.4.0.jar;D:\lib\jsp-api-2.1-6.1.14.jar;D:\lib\leveldbjni-all-1.8.jar;D:\lib\metrics-core-2.2.0.jar;D:\lib\metrics-core-3.1.2.jar;D:\lib\commons-logging-1.2.jar;D:\lib\commons-math3-3.1.1.jar;D:\lib\hadoop-client-2.7.4.jar;D:\lib\hadoop-common-2.5.1.jar;D:\lib\hbase-metrics-1.4.0.jar;D:\lib\jamon-runtime-2.4.1.jar;D:\lib\protobuf-java-2.5.0.jar;D:\lib\slf4j-log4j12-1.6.6.jar;D:\lib\snappy-java-1.0.4.1.jar;D:\lib\commons-digester-1.8.jar;D:\lib\hbase-protocol-1.2.3.jar;D:\lib\jackson-jaxrs-1.9.13.jar;D:\lib\jcl-over-slf4j-1.7.7.jar;D:\lib\commons-daemon-1.0.13.jar;D:\lib\hadoop-yarn-api-2.7.4.jar;D:\lib\hbase-procedure-1.4.0.jar;D:\lib\jasper-runtime-5.5.23.jar;D:\lib\api-asn1-api-1.0.0-M20.jar;D:\lib\commons-compress-1.4.1.jar;D:\lib\commons-httpclient-3.1.jar;D:\lib\jasper-compiler-5.5.23.jar;D:\lib\jetty-sslengine-6.1.26.jar;D:\lib\netty-all-4.0.23.Final.jar;D:\lib\servlet-api-2.5-6.1.14.jar;D:\lib\apacheds-i18n-2.0.0-M15.jar;D:\lib\commons-beanutils-1.7.0.jar;D:\lib\hbase-annotations-1.2.3.jar;D:\lib\hbase-metrics-api-1.4.0.jar;D:\lib\hbase-prefix-tree-1.4.0.jar;D:\lib\jackson-core-asl-1.9.13.jar;D:\lib\woodstox-core-asl-4.4.1.jar;D:\lib\hadoop-annotations-2.5.1.jar;D:\lib\hadoop-yarn-client-2.7.4.jar;D:\lib\hadoop-yarn-common-2.5.1.jar;D:\lib\hbase-common-1.4.0-tests.jar;D:\lib\commons-collections-3.2.2.jar;D:\lib\commons-configuration-1.6.jar;D:\lib\hbase-hadoop-compat-1.4.0.jar;D:\lib\jackson-mapper-asl-1.9.13.jar;D:\lib\hbase-hadoop2-compat-1.4.0.jar;D:\lib\mysql-connector-java-5.1.38.jar;D:\lib\commons-beanutils-core-1.8.0.jar;D:\lib\findbugs-annotations-1.3.9-1.jar;D:\lib\htrace-core-3.1.0-incubating.jar;D:\lib\hadoop-yarn-server-common-2.7.4.jar;D:\lib\apacheds-kerberos-codec-2.0.0-M15.jar;D:\lib\hadoop-mapreduce-client-app-2.7.4.jar;D:\lib\hadoop-mapreduce-client-core-2.5.1.jar;D:\lib\hadoop-mapreduce-client-common-2.7.4.jar;D:\lib\hadoop-mapreduce-client-shuffle-2.7.4.jar;D:\lib\hadoop-mapreduce-client-jobclient-2.7.4.jar;C:\Program Files\JetBrains\IntelliJ IDEA 2018.2.3\lib\idea_rt.jar [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=C:\Program Files\Java\jdk1.8.0_181\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\ProgramData\Oracle\Java\javapath;C:\Program Files\Java\jdk1.7.0_51\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;. [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\1\ [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:os.name=Windows Server 2008 R2 [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:os.version=6.1 [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:user.name=admin [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:user.home=C:\Users\Administrator [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Client environment:user.dir=C:\Users\Administrator\IdeaProjects\TestHbase [INFO ] [10:23:55] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192.168.0.25:2181 sessionTimeout=90000 watcher=hconnection-0x17d0685f0x0, quorum=192.168.0.25:2181, baseZNode=/hbase [DEBUG] [10:23:55] org.apache.zookeeper.ClientCnxn - zookeeper.disableAutoWatchReset is false [INFO ] [10:23:55] org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.0.25/192.168.0.25:2181. Will not attempt to authenticate using SASL (unknown error) [INFO ] [10:23:55] org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.0.25/192.168.0.25:2181, initiating session [DEBUG] [10:23:55] org.apache.zookeeper.ClientCnxn - Session establishment request sent on 192.168.0.25/192.168.0.25:2181 [INFO ] [10:23:56] org.apache.zookeeper.ClientCnxn - Session establishment complete on server 192.168.0.25/192.168.0.25:2181, sessionid = 0x165a4d303830022, negotiated timeout = 40000 [DEBUG] [10:23:56] org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher - hconnection-0x17d0685f0x0, quorum=192.168.0.25:2181, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null [DEBUG] [10:23:56] org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher - hconnection-0x17d0685f-0x165a4d303830022 connected [DEBUG] [10:23:56] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 1,3 replyHeader:: 1,184683596486,0 request:: '/hbase/hbaseid,F response:: s{4294967310,184683593733,1421399758722,1536068918058,938,0,0,0,60,0,4294967310} [DEBUG] [10:23:56] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 2,4 replyHeader:: 2,184683596486,0 request:: '/hbase/hbaseid,F response:: #ffffffff000133135313538404d61737465722e4861646f6f7066626162613563302d313737332d343731342d613630622d643233626232623865373831,s{4294967310,184683593733,1421399758722,1536068918058,938,0,0,0,60,0,4294967310} [DEBUG] [10:23:56] org.apache.hadoop.hbase.ipc.AbstractRpcClient - Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7c1e2a9e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=false, bind address=null hconnection-0x17d0685f org.apache.hadoop.hbase.client.HBaseAdmin@272ed83b hbase:meta true [DEBUG] [10:23:56] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 3,3 replyHeader:: 3,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:23:56] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 4,4 replyHeader:: 4,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:23:56] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:23:56] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [DEBUG] [10:23:57] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 5,3 replyHeader:: 5,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:23:57] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 6,4 replyHeader:: 6,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:23:57] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:23:57] org.apache.hadoop.hbase.ipc.RpcClientImpl - Not trying to connect to Master.Hadoop/192.168.0.25:60000 this server is in the failed servers list [DEBUG] [10:23:57] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 7,3 replyHeader:: 7,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:23:57] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 8,4 replyHeader:: 8,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:23:57] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:23:57] org.apache.hadoop.hbase.ipc.RpcClientImpl - Not trying to connect to Master.Hadoop/192.168.0.25:60000 this server is in the failed servers list [DEBUG] [10:23:57] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 9,3 replyHeader:: 9,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:23:57] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 10,4 replyHeader:: 10,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:23:57] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:23:57] org.apache.hadoop.hbase.ipc.RpcClientImpl - Not trying to connect to Master.Hadoop/192.168.0.25:60000 this server is in the failed servers list [DEBUG] [10:23:58] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 11,3 replyHeader:: 11,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:23:58] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 12,4 replyHeader:: 12,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:23:58] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:23:58] org.apache.hadoop.hbase.ipc.RpcClientImpl - Not trying to connect to Master.Hadoop/192.168.0.25:60000 this server is in the failed servers list [DEBUG] [10:23:59] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 13,3 replyHeader:: 13,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:23:59] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 14,4 replyHeader:: 14,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:23:59] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:23:59] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [DEBUG] [10:24:01] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 15,3 replyHeader:: 15,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:24:01] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 16,4 replyHeader:: 16,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:24:01] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:24:01] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [DEBUG] [10:24:05] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 17,3 replyHeader:: 17,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:24:05] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 18,4 replyHeader:: 18,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:24:05] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:24:05] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [DEBUG] [10:24:15] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 19,3 replyHeader:: 19,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:24:15] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 11ms [DEBUG] [10:24:15] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 20,4 replyHeader:: 20,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:24:15] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:24:15] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [DEBUG] [10:24:25] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 21,3 replyHeader:: 21,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:24:25] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 3ms [DEBUG] [10:24:25] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 22,4 replyHeader:: 22,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:24:25] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:24:25] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [DEBUG] [10:24:35] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 23,3 replyHeader:: 23,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:24:35] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 1ms [DEBUG] [10:24:35] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 24,4 replyHeader:: 24,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:24:35] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:24:35] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:24:35] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=10, retries=35, started=38894 ms ago, cancelled=false, msg= [DEBUG] [10:24:45] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 25,3 replyHeader:: 25,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:24:45] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 1ms [DEBUG] [10:24:45] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 26,4 replyHeader:: 26,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:24:45] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:24:45] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:24:45] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=11, retries=35, started=48956 ms ago, cancelled=false, msg= [DEBUG] [10:24:58] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 1ms [DEBUG] [10:25:05] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 27,3 replyHeader:: 27,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:25:05] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 28,4 replyHeader:: 28,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:25:05] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:25:05] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:25:05] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=12, retries=35, started=69113 ms ago, cancelled=false, msg= [DEBUG] [10:25:19] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 1ms [DEBUG] [10:25:25] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 29,3 replyHeader:: 29,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:25:25] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 30,4 replyHeader:: 30,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:25:25] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:25:25] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:25:25] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=13, retries=35, started=89176 ms ago, cancelled=false, msg= [DEBUG] [10:25:39] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 0ms [DEBUG] [10:25:45] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 31,3 replyHeader:: 31,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:25:45] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 32,4 replyHeader:: 32,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:25:45] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:25:45] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:25:45] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=14, retries=35, started=109299 ms ago, cancelled=false, msg= [DEBUG] [10:25:59] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 0ms [DEBUG] [10:26:05] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 33,3 replyHeader:: 33,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:26:05] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 34,4 replyHeader:: 34,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:26:05] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:26:05] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:26:05] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=15, retries=35, started=129364 ms ago, cancelled=false, msg= [DEBUG] [10:26:19] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 0ms [DEBUG] [10:26:26] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 35,3 replyHeader:: 35,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:26:26] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 36,4 replyHeader:: 36,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:26:26] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:26:26] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:26:26] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=16, retries=35, started=149567 ms ago, cancelled=false, msg= [DEBUG] [10:26:39] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 0ms [DEBUG] [10:26:46] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 37,3 replyHeader:: 37,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:26:46] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 38,4 replyHeader:: 38,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:26:46] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:26:46] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:26:46] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=17, retries=35, started=169742 ms ago, cancelled=false, msg= [DEBUG] [10:26:59] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 1ms [DEBUG] [10:27:06] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 39,3 replyHeader:: 39,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:27:06] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 40,4 replyHeader:: 40,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:27:06] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:27:06] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:27:06] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=18, retries=35, started=189961 ms ago, cancelled=false, msg= [DEBUG] [10:27:19] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 0ms [DEBUG] [10:27:26] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 41,3 replyHeader:: 41,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:27:26] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 42,4 replyHeader:: 42,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:27:26] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:27:26] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:27:26] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=19, retries=35, started=210070 ms ago, cancelled=false, msg= [DEBUG] [10:27:39] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 2ms [DEBUG] [10:27:46] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 43,3 replyHeader:: 43,184683596486,0 request:: '/hbase,F response:: s{4294967298,4294967298,1421399756338,1421399756338,0,1956,0,0,0,12,184683593750} [DEBUG] [10:27:46] org.apache.zookeeper.ClientCnxn - Reading reply sessionid:0x165a4d303830022, packet:: clientPath:null serverPath:null finished:false header:: 44,4 replyHeader:: 44,184683596486,0 request:: '/hbase/master,F response:: #ffffffff000133135313538404d61737465722e4861646f6f70004d61737465722e4861646f6f702c36303030302c31353336303638393135363739,s{184683593731,184683593731,1536068917156,1536068917156,0,0,0,316840575388352512,59,0,184683593731} [DEBUG] [10:27:46] org.apache.hadoop.hbase.ipc.RpcClientImpl - Use SIMPLE authentication for service MasterService, sasl=false [DEBUG] [10:27:46] org.apache.hadoop.hbase.ipc.RpcClientImpl - Connecting to Master.Hadoop/192.168.0.25:60000 [INFO ] [10:27:46] org.apache.hadoop.hbase.client.RpcRetryingCaller - Call exception, tries=20, retries=35, started=230074 ms ago, cancelled=false, msg= [DEBUG] [10:28:00] org.apache.zookeeper.ClientCnxn - Got ping response for sessionid: 0x165a4d303830022 after 1ms
用java读取hdfs的.lzo_deflate文件报错
linux环境没有问题,hadoop环境、配置也没有问题,并且通过hdoop fs -text 指令能正常打开该压缩文件。但是用java读取就报错了,请大神帮忙看看,谢谢 代码如下: public static void main(String[] args) { String uri = "/daas/****/MBLDPI3G.2016081823_10.1471532401822.lzo_deflate"; Configuration conf = new Configuration(); String path = "/software/servers/hadoop-2.6.3-bin/hadoop-2.6.3/etc/hadoop/"; conf.addResource(new Path(path + "core-site.xml")); conf.addResource(new Path(path + "hdfs-site.xml")); conf.addResource(new Path(path + "mapred-site.xml")); try { CompressionCodecFactory factory = new CompressionCodecFactory(conf); CompressionCodec codec = factory.getCodec(new Path(uri)); if (codec == null) { System.out.println("Codec for " + uri + " not found."); } else { CompressionInputStream in = null; try { in = codec.createInputStream(new java.io.FileInputStream(uri)); byte[] buffer = new byte[100]; int len = in.read(buffer); while (len > 0) { System.out.write(buffer, 0, len); len = in.read(buffer); } } finally { if (in != null) { in.close(); } } } } catch (Exception e) { e.printStackTrace(); } } 报错信息如下: log4j:WARN No appenders could be found for logger (org.apache.hadoop.util.NativeCodeLoader). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. java.io.FileNotFoundException: /daas/***/MBLDPI3G.2016081823_10.1471532401822.lzo_deflate (没有那个文件或目录) at java.io.FileInputStream.open(Native Method) at java.io.FileInputStream.<init>(FileInputStream.java:146) at java.io.FileInputStream.<init>(FileInputStream.java:101) at FileDecompressor.main(FileDecompressor.java:53) 加载的jar包: <classpathentry kind="lib" path="lib/commons-cli-1.2.jar"/> <classpathentry kind="lib" path="lib/commons-collections-3.2.2.jar"/> <classpathentry kind="lib" path="lib/commons-configuration-1.6.jar"/> <classpathentry kind="lib" path="lib/commons-lang-2.6.jar"/> <classpathentry kind="lib" path="lib/commons-logging-1.1.3.jar"/> <classpathentry kind="lib" path="lib/guava-18.0.jar"/> <classpathentry kind="lib" path="lib/hadoop-auth-2.6.3.jar"/> <classpathentry kind="lib" path="lib/hadoop-common-2.6.3.jar"/> <classpathentry kind="lib" path="lib/hadoop-hdfs-2.6.3.jar"/> <classpathentry kind="lib" path="lib/htrace-core-3.0.4.jar"/> <classpathentry kind="lib" path="lib/log4j-1.2.17.jar"/> <classpathentry kind="lib" path="lib/protobuf-java-2.5.0.jar"/> <classpathentry kind="lib" path="lib/slf4j-api-1.7.5.jar"/> <classpathentry kind="lib" path="lib/slf4j-log4j12-1.7.5.jar"/> <classpathentry kind="lib" path="lib/hadoop-lzo-0.4.20.jar"/>
spark 读取不到hive metastore 获取不到数据库
直接上异常 ``` Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/data01/hadoop/yarn/local/filecache/355/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 19/08/13 19:53:17 INFO SignalUtils: Registered signal handler for TERM 19/08/13 19:53:17 INFO SignalUtils: Registered signal handler for HUP 19/08/13 19:53:17 INFO SignalUtils: Registered signal handler for INT 19/08/13 19:53:17 INFO SecurityManager: Changing view acls to: yarn,hdfs 19/08/13 19:53:17 INFO SecurityManager: Changing modify acls to: yarn,hdfs 19/08/13 19:53:17 INFO SecurityManager: Changing view acls groups to: 19/08/13 19:53:17 INFO SecurityManager: Changing modify acls groups to: 19/08/13 19:53:17 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); groups with view permissions: Set(); users with modify permissions: Set(yarn, hdfs); groups with modify permissions: Set() 19/08/13 19:53:18 INFO ApplicationMaster: Preparing Local resources 19/08/13 19:53:19 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1565610088533_0087_000001 19/08/13 19:53:19 INFO ApplicationMaster: Starting the user application in a separate Thread 19/08/13 19:53:19 INFO ApplicationMaster: Waiting for spark context initialization... 19/08/13 19:53:19 INFO SparkContext: Running Spark version 2.3.0.2.6.5.0-292 19/08/13 19:53:19 INFO SparkContext: Submitted application: voice_stream 19/08/13 19:53:19 INFO SecurityManager: Changing view acls to: yarn,hdfs 19/08/13 19:53:19 INFO SecurityManager: Changing modify acls to: yarn,hdfs 19/08/13 19:53:19 INFO SecurityManager: Changing view acls groups to: 19/08/13 19:53:19 INFO SecurityManager: Changing modify acls groups to: 19/08/13 19:53:19 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); groups with view permissions: Set(); users with modify permissions: Set(yarn, hdfs); groups with modify permissions: Set() 19/08/13 19:53:19 INFO Utils: Successfully started service 'sparkDriver' on port 20410. 19/08/13 19:53:19 INFO SparkEnv: Registering MapOutputTracker 19/08/13 19:53:19 INFO SparkEnv: Registering BlockManagerMaster 19/08/13 19:53:19 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 19/08/13 19:53:19 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 19/08/13 19:53:19 INFO DiskBlockManager: Created local directory at /data01/hadoop/yarn/local/usercache/hdfs/appcache/application_1565610088533_0087/blockmgr-94d35b97-43b2-496e-a4cb-73ecd3ed186c 19/08/13 19:53:19 INFO MemoryStore: MemoryStore started with capacity 366.3 MB 19/08/13 19:53:19 INFO SparkEnv: Registering OutputCommitCoordinator 19/08/13 19:53:19 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 19/08/13 19:53:19 INFO Utils: Successfully started service 'SparkUI' on port 28852. 19/08/13 19:53:19 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://datanode02:28852 19/08/13 19:53:19 INFO YarnClusterScheduler: Created YarnClusterScheduler 19/08/13 19:53:20 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1565610088533_0087 and attemptId Some(appattempt_1565610088533_0087_000001) 19/08/13 19:53:20 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 31984. 19/08/13 19:53:20 INFO NettyBlockTransferService: Server created on datanode02:31984 19/08/13 19:53:20 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 19/08/13 19:53:20 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, datanode02, 31984, None) 19/08/13 19:53:20 INFO BlockManagerMasterEndpoint: Registering block manager datanode02:31984 with 366.3 MB RAM, BlockManagerId(driver, datanode02, 31984, None) 19/08/13 19:53:20 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, datanode02, 31984, None) 19/08/13 19:53:20 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, datanode02, 31984, None) 19/08/13 19:53:20 INFO EventLoggingListener: Logging events to hdfs:/spark2-history/application_1565610088533_0087_1 19/08/13 19:53:20 INFO ApplicationMaster: =============================================================================== YARN executor launch context: env: CLASSPATH -> {{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>/usr/hdp/2.6.5.0-292/hadoop/conf<CPS>/usr/hdp/2.6.5.0-292/hadoop/*<CPS>/usr/hdp/2.6.5.0-292/hadoop/lib/*<CPS>/usr/hdp/current/hadoop-hdfs-client/*<CPS>/usr/hdp/current/hadoop-hdfs-client/lib/*<CPS>/usr/hdp/current/hadoop-yarn-client/*<CPS>/usr/hdp/current/hadoop-yarn-client/lib/*<CPS>/usr/hdp/current/ext/hadoop/*<CPS>$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.5.0-292/hadoop/lib/hadoop-lzo-0.6.0.2.6.5.0-292.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/*<CPS>{{PWD}}/__spark_conf__/__hadoop_conf__ SPARK_YARN_STAGING_DIR -> *********(redacted) SPARK_USER -> *********(redacted) command: LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" \ {{JAVA_HOME}}/bin/java \ -server \ -Xmx5120m \ -Djava.io.tmpdir={{PWD}}/tmp \ '-Dspark.history.ui.port=18081' \ '-Dspark.rpc.message.maxSize=100' \ -Dspark.yarn.app.container.log.dir=<LOG_DIR> \ -XX:OnOutOfMemoryError='kill %p' \ org.apache.spark.executor.CoarseGrainedExecutorBackend \ --driver-url \ spark://CoarseGrainedScheduler@datanode02:20410 \ --executor-id \ <executorId> \ --hostname \ <hostname> \ --cores \ 2 \ --app-id \ application_1565610088533_0087 \ --user-class-path \ file:$PWD/__app__.jar \ --user-class-path \ file:$PWD/hadoop-common-2.7.3.jar \ --user-class-path \ file:$PWD/guava-12.0.1.jar \ --user-class-path \ file:$PWD/hbase-server-1.2.8.jar \ --user-class-path \ file:$PWD/hbase-protocol-1.2.8.jar \ --user-class-path \ file:$PWD/hbase-client-1.2.8.jar \ --user-class-path \ file:$PWD/hbase-common-1.2.8.jar \ --user-class-path \ file:$PWD/mysql-connector-java-5.1.44-bin.jar \ --user-class-path \ file:$PWD/spark-streaming-kafka-0-8-assembly_2.11-2.3.2.jar \ --user-class-path \ file:$PWD/spark-examples_2.11-1.6.0-typesafe-001.jar \ --user-class-path \ file:$PWD/fastjson-1.2.7.jar \ 1><LOG_DIR>/stdout \ 2><LOG_DIR>/stderr resources: spark-streaming-kafka-0-8-assembly_2.11-2.3.2.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/spark-streaming-kafka-0-8-assembly_2.11-2.3.2.jar" } size: 12271027 timestamp: 1565697198603 type: FILE visibility: PRIVATE spark-examples_2.11-1.6.0-typesafe-001.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/spark-examples_2.11-1.6.0-typesafe-001.jar" } size: 1867746 timestamp: 1565697198751 type: FILE visibility: PRIVATE hbase-server-1.2.8.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/hbase-server-1.2.8.jar" } size: 4197896 timestamp: 1565697197770 type: FILE visibility: PRIVATE hbase-common-1.2.8.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/hbase-common-1.2.8.jar" } size: 570163 timestamp: 1565697198318 type: FILE visibility: PRIVATE __app__.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/spark_history_data2.jar" } size: 44924 timestamp: 1565697197260 type: FILE visibility: PRIVATE guava-12.0.1.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/guava-12.0.1.jar" } size: 1795932 timestamp: 1565697197614 type: FILE visibility: PRIVATE hbase-client-1.2.8.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/hbase-client-1.2.8.jar" } size: 1306401 timestamp: 1565697198180 type: FILE visibility: PRIVATE __spark_conf__ -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/__spark_conf__.zip" } size: 273513 timestamp: 1565697199131 type: ARCHIVE visibility: PRIVATE fastjson-1.2.7.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/fastjson-1.2.7.jar" } size: 417221 timestamp: 1565697198865 type: FILE visibility: PRIVATE hbase-protocol-1.2.8.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/hbase-protocol-1.2.8.jar" } size: 4366252 timestamp: 1565697198023 type: FILE visibility: PRIVATE __spark_libs__ -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/hdp/apps/2.6.5.0-292/spark2/spark2-hdp-yarn-archive.tar.gz" } size: 227600110 timestamp: 1549953820247 type: ARCHIVE visibility: PUBLIC mysql-connector-java-5.1.44-bin.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/mysql-connector-java-5.1.44-bin.jar" } size: 999635 timestamp: 1565697198445 type: FILE visibility: PRIVATE hadoop-common-2.7.3.jar -> resource { scheme: "hdfs" host: "CID-042fb939-95b4-4b74-91b8-9f94b999bdf7" port: -1 file: "/user/hdfs/.sparkStaging/application_1565610088533_0087/hadoop-common-2.7.3.jar" } size: 3479293 timestamp: 1565697197476 type: FILE visibility: PRIVATE =============================================================================== 19/08/13 19:53:20 INFO RMProxy: Connecting to ResourceManager at namenode02/10.1.38.38:8030 19/08/13 19:53:20 INFO YarnRMClient: Registering the ApplicationMaster 19/08/13 19:53:20 INFO YarnAllocator: Will request 3 executor container(s), each with 2 core(s) and 5632 MB memory (including 512 MB of overhead) 19/08/13 19:53:20 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@datanode02:20410) 19/08/13 19:53:20 INFO YarnAllocator: Submitted 3 unlocalized container requests. 19/08/13 19:53:20 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals 19/08/13 19:53:20 INFO AMRMClientImpl: Received new token for : datanode03:45454 19/08/13 19:53:21 INFO YarnAllocator: Launching container container_e20_1565610088533_0087_01_000002 on host datanode03 for executor with ID 1 19/08/13 19:53:21 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. 19/08/13 19:53:21 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 19/08/13 19:53:21 INFO ContainerManagementProtocolProxy: Opening proxy : datanode03:45454 19/08/13 19:53:21 INFO AMRMClientImpl: Received new token for : datanode01:45454 19/08/13 19:53:21 INFO YarnAllocator: Launching container container_e20_1565610088533_0087_01_000003 on host datanode01 for executor with ID 2 19/08/13 19:53:21 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. 19/08/13 19:53:21 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 19/08/13 19:53:21 INFO ContainerManagementProtocolProxy: Opening proxy : datanode01:45454 19/08/13 19:53:22 INFO AMRMClientImpl: Received new token for : datanode02:45454 19/08/13 19:53:22 INFO YarnAllocator: Launching container container_e20_1565610088533_0087_01_000004 on host datanode02 for executor with ID 3 19/08/13 19:53:22 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. 19/08/13 19:53:22 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 19/08/13 19:53:22 INFO ContainerManagementProtocolProxy: Opening proxy : datanode02:45454 19/08/13 19:53:24 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.1.198.144:41122) with ID 1 19/08/13 19:53:25 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.1.229.163:24656) with ID 3 19/08/13 19:53:25 INFO BlockManagerMasterEndpoint: Registering block manager datanode03:3328 with 2.5 GB RAM, BlockManagerId(1, datanode03, 3328, None) 19/08/13 19:53:25 INFO BlockManagerMasterEndpoint: Registering block manager datanode02:28863 with 2.5 GB RAM, BlockManagerId(3, datanode02, 28863, None) 19/08/13 19:53:25 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.1.229.158:64276) with ID 2 19/08/13 19:53:25 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 19/08/13 19:53:25 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done 19/08/13 19:53:25 INFO BlockManagerMasterEndpoint: Registering block manager datanode01:20487 with 2.5 GB RAM, BlockManagerId(2, datanode01, 20487, None) 19/08/13 19:53:25 WARN SparkContext: Using an existing SparkContext; some configuration may not take effect. 19/08/13 19:53:25 INFO SparkContext: Starting job: start at VoiceApplication2.java:128 19/08/13 19:53:25 INFO DAGScheduler: Registering RDD 1 (start at VoiceApplication2.java:128) 19/08/13 19:53:25 INFO DAGScheduler: Got job 0 (start at VoiceApplication2.java:128) with 20 output partitions 19/08/13 19:53:25 INFO DAGScheduler: Final stage: ResultStage 1 (start at VoiceApplication2.java:128) 19/08/13 19:53:25 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0) 19/08/13 19:53:25 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0) 19/08/13 19:53:26 INFO DAGScheduler: Submitting ShuffleMapStage 0 (MapPartitionsRDD[1] at start at VoiceApplication2.java:128), which has no missing parents 19/08/13 19:53:26 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.1 KB, free 366.3 MB) 19/08/13 19:53:26 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2011.0 B, free 366.3 MB) 19/08/13 19:53:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on datanode02:31984 (size: 2011.0 B, free: 366.3 MB) 19/08/13 19:53:26 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1039 19/08/13 19:53:26 INFO DAGScheduler: Submitting 50 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[1] at start at VoiceApplication2.java:128) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 19/08/13 19:53:26 INFO YarnClusterScheduler: Adding task set 0.0 with 50 tasks 19/08/13 19:53:26 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, datanode02, executor 3, partition 0, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, datanode03, executor 1, partition 1, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, datanode01, executor 2, partition 2, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, datanode02, executor 3, partition 3, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, datanode03, executor 1, partition 4, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, datanode01, executor 2, partition 5, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on datanode02:28863 (size: 2011.0 B, free: 2.5 GB) 19/08/13 19:53:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on datanode03:3328 (size: 2011.0 B, free: 2.5 GB) 19/08/13 19:53:26 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on datanode01:20487 (size: 2011.0 B, free: 2.5 GB) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, datanode02, executor 3, partition 6, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, datanode02, executor 3, partition 7, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 693 ms on datanode02 (executor 3) (1/50) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 712 ms on datanode02 (executor 3) (2/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, datanode02, executor 3, partition 8, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 21 ms on datanode02 (executor 3) (3/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, datanode02, executor 3, partition 9, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 26 ms on datanode02 (executor 3) (4/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, datanode02, executor 3, partition 10, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 23 ms on datanode02 (executor 3) (5/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, datanode02, executor 3, partition 11, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 25 ms on datanode02 (executor 3) (6/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, datanode02, executor 3, partition 12, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 18 ms on datanode02 (executor 3) (7/50) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 14 ms on datanode02 (executor 3) (8/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, datanode02, executor 3, partition 13, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, datanode02, executor 3, partition 14, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 16 ms on datanode02 (executor 3) (9/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, datanode02, executor 3, partition 15, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 22 ms on datanode02 (executor 3) (10/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, datanode02, executor 3, partition 16, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 14) in 16 ms on datanode02 (executor 3) (11/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, datanode02, executor 3, partition 17, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 15.0 in stage 0.0 (TID 15) in 13 ms on datanode02 (executor 3) (12/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, datanode01, executor 2, partition 18, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, datanode01, executor 2, partition 19, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 787 ms on datanode01 (executor 2) (13/50) 19/08/13 19:53:26 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 789 ms on datanode01 (executor 2) (14/50) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, datanode03, executor 1, partition 20, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:26 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, datanode03, executor 1, partition 21, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 905 ms on datanode03 (executor 1) (15/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 907 ms on datanode03 (executor 1) (16/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, datanode02, executor 3, partition 22, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, datanode02, executor 3, partition 23, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 24, datanode01, executor 2, partition 24, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 18.0 in stage 0.0 (TID 18) in 124 ms on datanode01 (executor 2) (17/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 16.0 in stage 0.0 (TID 16) in 134 ms on datanode02 (executor 3) (18/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, datanode01, executor 2, partition 25, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 26, datanode03, executor 1, partition 26, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 17.0 in stage 0.0 (TID 17) in 134 ms on datanode02 (executor 3) (19/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 20.0 in stage 0.0 (TID 20) in 122 ms on datanode03 (executor 1) (20/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 27.0 in stage 0.0 (TID 27, datanode03, executor 1, partition 27, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 19.0 in stage 0.0 (TID 19) in 127 ms on datanode01 (executor 2) (21/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 21.0 in stage 0.0 (TID 21) in 123 ms on datanode03 (executor 1) (22/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 28.0 in stage 0.0 (TID 28, datanode02, executor 3, partition 28, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 29.0 in stage 0.0 (TID 29, datanode02, executor 3, partition 29, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 22.0 in stage 0.0 (TID 22) in 19 ms on datanode02 (executor 3) (23/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 23.0 in stage 0.0 (TID 23) in 18 ms on datanode02 (executor 3) (24/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 30.0 in stage 0.0 (TID 30, datanode01, executor 2, partition 30, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 31.0 in stage 0.0 (TID 31, datanode01, executor 2, partition 31, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 25.0 in stage 0.0 (TID 25) in 27 ms on datanode01 (executor 2) (25/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 24.0 in stage 0.0 (TID 24) in 29 ms on datanode01 (executor 2) (26/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 32.0 in stage 0.0 (TID 32, datanode02, executor 3, partition 32, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 29.0 in stage 0.0 (TID 29) in 16 ms on datanode02 (executor 3) (27/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 33.0 in stage 0.0 (TID 33, datanode03, executor 1, partition 33, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 26.0 in stage 0.0 (TID 26) in 30 ms on datanode03 (executor 1) (28/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 34.0 in stage 0.0 (TID 34, datanode02, executor 3, partition 34, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 28.0 in stage 0.0 (TID 28) in 21 ms on datanode02 (executor 3) (29/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 35.0 in stage 0.0 (TID 35, datanode03, executor 1, partition 35, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 27.0 in stage 0.0 (TID 27) in 32 ms on datanode03 (executor 1) (30/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 36.0 in stage 0.0 (TID 36, datanode02, executor 3, partition 36, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 32.0 in stage 0.0 (TID 32) in 11 ms on datanode02 (executor 3) (31/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 37.0 in stage 0.0 (TID 37, datanode01, executor 2, partition 37, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 30.0 in stage 0.0 (TID 30) in 18 ms on datanode01 (executor 2) (32/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 38.0 in stage 0.0 (TID 38, datanode01, executor 2, partition 38, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 31.0 in stage 0.0 (TID 31) in 20 ms on datanode01 (executor 2) (33/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 39.0 in stage 0.0 (TID 39, datanode03, executor 1, partition 39, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 33.0 in stage 0.0 (TID 33) in 17 ms on datanode03 (executor 1) (34/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 34.0 in stage 0.0 (TID 34) in 17 ms on datanode02 (executor 3) (35/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 40.0 in stage 0.0 (TID 40, datanode02, executor 3, partition 40, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 41.0 in stage 0.0 (TID 41, datanode03, executor 1, partition 41, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 35.0 in stage 0.0 (TID 35) in 17 ms on datanode03 (executor 1) (36/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 42.0 in stage 0.0 (TID 42, datanode02, executor 3, partition 42, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 36.0 in stage 0.0 (TID 36) in 16 ms on datanode02 (executor 3) (37/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 43.0 in stage 0.0 (TID 43, datanode01, executor 2, partition 43, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 37.0 in stage 0.0 (TID 37) in 16 ms on datanode01 (executor 2) (38/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 44.0 in stage 0.0 (TID 44, datanode02, executor 3, partition 44, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 45.0 in stage 0.0 (TID 45, datanode02, executor 3, partition 45, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 40.0 in stage 0.0 (TID 40) in 14 ms on datanode02 (executor 3) (39/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 42.0 in stage 0.0 (TID 42) in 11 ms on datanode02 (executor 3) (40/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 46.0 in stage 0.0 (TID 46, datanode03, executor 1, partition 46, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 39.0 in stage 0.0 (TID 39) in 20 ms on datanode03 (executor 1) (41/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 47.0 in stage 0.0 (TID 47, datanode03, executor 1, partition 47, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 41.0 in stage 0.0 (TID 41) in 20 ms on datanode03 (executor 1) (42/50) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 48.0 in stage 0.0 (TID 48, datanode01, executor 2, partition 48, PROCESS_LOCAL, 7831 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 49.0 in stage 0.0 (TID 49, datanode01, executor 2, partition 49, PROCESS_LOCAL, 7888 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 43.0 in stage 0.0 (TID 43) in 18 ms on datanode01 (executor 2) (43/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 38.0 in stage 0.0 (TID 38) in 31 ms on datanode01 (executor 2) (44/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 45.0 in stage 0.0 (TID 45) in 11 ms on datanode02 (executor 3) (45/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 44.0 in stage 0.0 (TID 44) in 16 ms on datanode02 (executor 3) (46/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 46.0 in stage 0.0 (TID 46) in 18 ms on datanode03 (executor 1) (47/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 48.0 in stage 0.0 (TID 48) in 15 ms on datanode01 (executor 2) (48/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 47.0 in stage 0.0 (TID 47) in 15 ms on datanode03 (executor 1) (49/50) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 49.0 in stage 0.0 (TID 49) in 25 ms on datanode01 (executor 2) (50/50) 19/08/13 19:53:27 INFO YarnClusterScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 19/08/13 19:53:27 INFO DAGScheduler: ShuffleMapStage 0 (start at VoiceApplication2.java:128) finished in 1.174 s 19/08/13 19:53:27 INFO DAGScheduler: looking for newly runnable stages 19/08/13 19:53:27 INFO DAGScheduler: running: Set() 19/08/13 19:53:27 INFO DAGScheduler: waiting: Set(ResultStage 1) 19/08/13 19:53:27 INFO DAGScheduler: failed: Set() 19/08/13 19:53:27 INFO DAGScheduler: Submitting ResultStage 1 (ShuffledRDD[2] at start at VoiceApplication2.java:128), which has no missing parents 19/08/13 19:53:27 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.2 KB, free 366.3 MB) 19/08/13 19:53:27 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1979.0 B, free 366.3 MB) 19/08/13 19:53:27 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on datanode02:31984 (size: 1979.0 B, free: 366.3 MB) 19/08/13 19:53:27 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1039 19/08/13 19:53:27 INFO DAGScheduler: Submitting 20 missing tasks from ResultStage 1 (ShuffledRDD[2] at start at VoiceApplication2.java:128) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)) 19/08/13 19:53:27 INFO YarnClusterScheduler: Adding task set 1.0 with 20 tasks 19/08/13 19:53:27 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 50, datanode03, executor 1, partition 0, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 51, datanode02, executor 3, partition 1, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 52, datanode01, executor 2, partition 3, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 53, datanode03, executor 1, partition 2, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 4.0 in stage 1.0 (TID 54, datanode02, executor 3, partition 4, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 5.0 in stage 1.0 (TID 55, datanode01, executor 2, partition 5, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on datanode02:28863 (size: 1979.0 B, free: 2.5 GB) 19/08/13 19:53:27 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on datanode01:20487 (size: 1979.0 B, free: 2.5 GB) 19/08/13 19:53:27 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on datanode03:3328 (size: 1979.0 B, free: 2.5 GB) 19/08/13 19:53:27 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.1.229.163:24656 19/08/13 19:53:27 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.1.198.144:41122 19/08/13 19:53:27 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 10.1.229.158:64276 19/08/13 19:53:27 INFO TaskSetManager: Starting task 7.0 in stage 1.0 (TID 56, datanode03, executor 1, partition 7, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 53) in 192 ms on datanode03 (executor 1) (1/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 8.0 in stage 1.0 (TID 57, datanode03, executor 1, partition 8, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 7.0 in stage 1.0 (TID 56) in 25 ms on datanode03 (executor 1) (2/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 6.0 in stage 1.0 (TID 58, datanode02, executor 3, partition 6, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 51) in 220 ms on datanode02 (executor 3) (3/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 14.0 in stage 1.0 (TID 59, datanode03, executor 1, partition 14, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 8.0 in stage 1.0 (TID 57) in 17 ms on datanode03 (executor 1) (4/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 16.0 in stage 1.0 (TID 60, datanode03, executor 1, partition 16, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 14.0 in stage 1.0 (TID 59) in 15 ms on datanode03 (executor 1) (5/20) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 16.0 in stage 1.0 (TID 60) in 21 ms on datanode03 (executor 1) (6/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 9.0 in stage 1.0 (TID 61, datanode02, executor 3, partition 9, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 4.0 in stage 1.0 (TID 54) in 269 ms on datanode02 (executor 3) (7/20) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 50) in 339 ms on datanode03 (executor 1) (8/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 10.0 in stage 1.0 (TID 62, datanode02, executor 3, partition 10, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 6.0 in stage 1.0 (TID 58) in 56 ms on datanode02 (executor 3) (9/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 11.0 in stage 1.0 (TID 63, datanode01, executor 2, partition 11, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 5.0 in stage 1.0 (TID 55) in 284 ms on datanode01 (executor 2) (10/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 12.0 in stage 1.0 (TID 64, datanode01, executor 2, partition 12, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 52) in 287 ms on datanode01 (executor 2) (11/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 13.0 in stage 1.0 (TID 65, datanode02, executor 3, partition 13, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 15.0 in stage 1.0 (TID 66, datanode02, executor 3, partition 15, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 10.0 in stage 1.0 (TID 62) in 25 ms on datanode02 (executor 3) (12/20) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 9.0 in stage 1.0 (TID 61) in 29 ms on datanode02 (executor 3) (13/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 17.0 in stage 1.0 (TID 67, datanode02, executor 3, partition 17, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 15.0 in stage 1.0 (TID 66) in 13 ms on datanode02 (executor 3) (14/20) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 13.0 in stage 1.0 (TID 65) in 16 ms on datanode02 (executor 3) (15/20) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 18.0 in stage 1.0 (TID 68, datanode02, executor 3, partition 18, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Starting task 19.0 in stage 1.0 (TID 69, datanode01, executor 2, partition 19, NODE_LOCAL, 7638 bytes) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 11.0 in stage 1.0 (TID 63) in 30 ms on datanode01 (executor 2) (16/20) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 12.0 in stage 1.0 (TID 64) in 30 ms on datanode01 (executor 2) (17/20) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 17.0 in stage 1.0 (TID 67) in 17 ms on datanode02 (executor 3) (18/20) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 19.0 in stage 1.0 (TID 69) in 13 ms on datanode01 (executor 2) (19/20) 19/08/13 19:53:27 INFO TaskSetManager: Finished task 18.0 in stage 1.0 (TID 68) in 20 ms on datanode02 (executor 3) (20/20) 19/08/13 19:53:27 INFO YarnClusterScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 19/08/13 19:53:27 INFO DAGScheduler: ResultStage 1 (start at VoiceApplication2.java:128) finished in 0.406 s 19/08/13 19:53:27 INFO DAGScheduler: Job 0 finished: start at VoiceApplication2.java:128, took 1.850883 s 19/08/13 19:53:27 INFO ReceiverTracker: Starting 1 receivers 19/08/13 19:53:27 INFO ReceiverTracker: ReceiverTracker started 19/08/13 19:53:27 INFO KafkaInputDStream: Slide time = 60000 ms 19/08/13 19:53:27 INFO KafkaInputDStream: Storage level = Serialized 1x Replicated 19/08/13 19:53:27 INFO KafkaInputDStream: Checkpoint interval = null 19/08/13 19:53:27 INFO KafkaInputDStream: Remember interval = 60000 ms 19/08/13 19:53:27 INFO KafkaInputDStream: Initialized and validated org.apache.spark.streaming.kafka.KafkaInputDStream@5fd3dc81 19/08/13 19:53:27 INFO ForEachDStream: Slide time = 60000 ms 19/08/13 19:53:27 INFO ForEachDStream: Storage level = Serialized 1x Replicated 19/08/13 19:53:27 INFO ForEachDStream: Checkpoint interval = null 19/08/13 19:53:27 INFO ForEachDStream: Remember interval = 60000 ms 19/08/13 19:53:27 INFO ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@4044ec97 19/08/13 19:53:27 INFO KafkaInputDStream: Slide time = 60000 ms 19/08/13 19:53:27 INFO KafkaInputDStream: Storage level = Serialized 1x Replicated 19/08/13 19:53:27 INFO KafkaInputDStream: Checkpoint interval = null 19/08/13 19:53:27 INFO KafkaInputDStream: Remember interval = 60000 ms 19/08/13 19:53:27 INFO KafkaInputDStream: Initialized and validated org.apache.spark.streaming.kafka.KafkaInputDStream@5fd3dc81 19/08/13 19:53:27 INFO MappedDStream: Slide time = 60000 ms 19/08/13 19:53:27 INFO MappedDStream: Storage level = Serialized 1x Replicated 19/08/13 19:53:27 INFO MappedDStream: Checkpoint interval = null 19/08/13 19:53:27 INFO MappedDStream: Remember interval = 60000 ms 19/08/13 19:53:27 INFO MappedDStream: Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@5dd4b960 19/08/13 19:53:27 INFO ForEachDStream: Slide time = 60000 ms 19/08/13 19:53:27 INFO ForEachDStream: Storage level = Serialized 1x Replicated 19/08/13 19:53:27 INFO ForEachDStream: Checkpoint interval = null 19/08/13 19:53:27 INFO ForEachDStream: Remember interval = 60000 ms 19/08/13 19:53:27 INFO ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@132d0c3c 19/08/13 19:53:27 INFO KafkaInputDStream: Slide time = 60000 ms 19/08/13 19:53:27 INFO KafkaInputDStream: Storage level = Serialized 1x Replicated 19/08/13 19:53:27 INFO KafkaInputDStream: Checkpoint interval = null 19/08/13 19:53:27 INFO KafkaInputDStream: Remember interval = 60000 ms 19/08/13 19:53:27 INFO KafkaInputDStream: Initialized and validated org.apache.spark.streaming.kafka.KafkaInputDStream@5fd3dc81 19/08/13 19:53:27 INFO MappedDStream: Slide time = 60000 ms 19/08/13 19:53:27 INFO MappedDStream: Storage level = Serialized 1x Replicated 19/08/13 19:53:27 INFO MappedDStream: Checkpoint interval = null 19/08/13 19:53:27 INFO MappedDStream: Remember interval = 60000 ms 19/08/13 19:53:27 INFO MappedDStream: Initialized and validated org.apache.spark.streaming.dstream.MappedDStream@5dd4b960 19/08/13 19:53:27 INFO ForEachDStream: Slide time = 60000 ms 19/08/13 19:53:27 INFO ForEachDStream: Storage level = Serialized 1x Replicated 19/08/13 19:53:27 INFO ForEachDStream: Checkpoint interval = null 19/08/13 19:53:27 INFO ForEachDStream: Remember interval = 60000 ms 19/08/13 19:53:27 INFO ForEachDStream: Initialized and validated org.apache.spark.streaming.dstream.ForEachDStream@525bed0c 19/08/13 19:53:27 INFO DAGScheduler: Got job 1 (start at VoiceApplication2.java:128) with 1 output partitions 19/08/13 19:53:27 INFO DAGScheduler: Final stage: ResultStage 2 (start at VoiceApplication2.java:128) 19/08/13 19:53:27 INFO DAGScheduler: Parents of final stage: List() 19/08/13 19:53:27 INFO DAGScheduler: Missing parents: List() 19/08/13 19:53:27 INFO DAGScheduler: Submitting ResultStage 2 (Receiver 0 ParallelCollectionRDD[3] at makeRDD at ReceiverTracker.scala:613), which has no missing parents 19/08/13 19:53:27 INFO ReceiverTracker: Receiver 0 started 19/08/13 19:53:27 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 133.5 KB, free 366.2 MB) 19/08/13 19:53:27 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 36.3 KB, free 366.1 MB) 19/08/13 19:53:27 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on datanode02:31984 (size: 36.3 KB, free: 366.3 MB) 19/08/13 19:53:27 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:1039 19/08/13 19:53:27 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (Receiver 0 ParallelCollectionRDD[3] at makeRDD at ReceiverTracker.scala:613) (first 15 tasks are for partitions Vector(0)) 19/08/13 19:53:27 INFO YarnClusterScheduler: Adding task set 2.0 with 1 tasks 19/08/13 19:53:27 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 70, datanode01, executor 2, partition 0, PROCESS_LOCAL, 8757 bytes) 19/08/13 19:53:27 INFO RecurringTimer: Started timer for JobGenerator at time 1565697240000 19/08/13 19:53:27 INFO JobGenerator: Started JobGenerator at 1565697240000 ms 19/08/13 19:53:27 INFO JobScheduler: Started JobScheduler 19/08/13 19:53:27 INFO StreamingContext: StreamingContext started 19/08/13 19:53:27 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on datanode01:20487 (size: 36.3 KB, free: 2.5 GB) 19/08/13 19:53:27 INFO ReceiverTracker: Registered receiver for stream 0 from 10.1.229.158:64276 19/08/13 19:54:00 INFO JobScheduler: Added jobs for time 1565697240000 ms 19/08/13 19:54:00 INFO JobScheduler: Starting job streaming job 1565697240000 ms.0 from job set of time 1565697240000 ms 19/08/13 19:54:00 INFO JobScheduler: Starting job streaming job 1565697240000 ms.1 from job set of time 1565697240000 ms 19/08/13 19:54:00 INFO JobScheduler: Finished job streaming job 1565697240000 ms.1 from job set of time 1565697240000 ms 19/08/13 19:54:00 INFO JobScheduler: Finished job streaming job 1565697240000 ms.0 from job set of time 1565697240000 ms 19/08/13 19:54:00 INFO JobScheduler: Starting job streaming job 1565697240000 ms.2 from job set of time 1565697240000 ms 19/08/13 19:54:00 INFO SharedState: loading hive config file: file:/data01/hadoop/yarn/local/usercache/hdfs/filecache/85431/__spark_conf__.zip/__hadoop_conf__/hive-site.xml 19/08/13 19:54:00 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('hdfs://CID-042fb939-95b4-4b74-91b8-9f94b999bdf7/apps/hive/warehouse'). 19/08/13 19:54:00 INFO SharedState: Warehouse path is 'hdfs://CID-042fb939-95b4-4b74-91b8-9f94b999bdf7/apps/hive/warehouse'. 19/08/13 19:54:00 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint 19/08/13 19:54:00 INFO BlockManagerInfo: Removed broadcast_1_piece0 on datanode02:31984 in memory (size: 1979.0 B, free: 366.3 MB) 19/08/13 19:54:00 INFO BlockManagerInfo: Removed broadcast_1_piece0 on datanode02:28863 in memory (size: 1979.0 B, free: 2.5 GB) 19/08/13 19:54:00 INFO BlockManagerInfo: Removed broadcast_1_piece0 on datanode01:20487 in memory (size: 1979.0 B, free: 2.5 GB) 19/08/13 19:54:00 INFO BlockManagerInfo: Removed broadcast_1_piece0 on datanode03:3328 in memory (size: 1979.0 B, free: 2.5 GB) 19/08/13 19:54:02 INFO CodeGenerator: Code generated in 175.416957 ms 19/08/13 19:54:02 INFO JobScheduler: Finished job streaming job 1565697240000 ms.2 from job set of time 1565697240000 ms 19/08/13 19:54:02 ERROR JobScheduler: Error running job streaming job 1565697240000 ms.2 org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'meta_voice' not found; at org.apache.spark.sql.catalyst.catalog.ExternalCatalog.requireDbExists(ExternalCatalog.scala:40) at org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.tableExists(InMemoryCatalog.scala:331) at org.apache.spark.sql.catalyst.catalog.SessionCatalog.tableExists(SessionCatalog.scala:388) at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:398) at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:393) at com.stream.VoiceApplication2$2.call(VoiceApplication2.java:122) at com.stream.VoiceApplication2$2.call(VoiceApplication2.java:115) at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$2.apply(JavaDStreamLike.scala:280) at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$2.apply(JavaDStreamLike.scala:280) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/08/13 19:54:02 ERROR ApplicationMaster: User class threw exception: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'meta_voice' not found; org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'meta_voice' not found; at org.apache.spark.sql.catalyst.catalog.ExternalCatalog.requireDbExists(ExternalCatalog.scala:40) at org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.tableExists(InMemoryCatalog.scala:331) at org.apache.spark.sql.catalyst.catalog.SessionCatalog.tableExists(SessionCatalog.scala:388) at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:398) at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:393) at com.stream.VoiceApplication2$2.call(VoiceApplication2.java:122) at com.stream.VoiceApplication2$2.call(VoiceApplication2.java:115) at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$2.apply(JavaDStreamLike.scala:280) at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$2.apply(JavaDStreamLike.scala:280) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 19/08/13 19:54:02 INFO ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: org.apache.spark.sql.catalyst.analysis.NoSuchDatabaseException: Database 'meta_voice' not found; at org.apache.spark.sql.catalyst.catalog.ExternalCatalog.requireDbExists(ExternalCatalog.scala:40) at org.apache.spark.sql.catalyst.catalog.InMemoryCatalog.tableExists(InMemoryCatalog.scala:331) at org.apache.spark.sql.catalyst.catalog.SessionCatalog.tableExists(SessionCatalog.scala:388) at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:398) at org.apache.spark.sql.DataFrameWriter.saveAsTable(DataFrameWriter.scala:393) at com.stream.VoiceApplication2$2.call(VoiceApplication2.java:122) at com.stream.VoiceApplication2$2.call(VoiceApplication2.java:115) at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$2.apply(JavaDStreamLike.scala:280) at org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$2.apply(JavaDStreamLike.scala:280) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51) at org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50) at org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.streaming.scheduler.Job.run(Job.scala:39) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) at org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ) 19/08/13 19:54:02 INFO StreamingContext: Invoking stop(stopGracefully=true) from shutdown hook 19/08/13 19:54:02 INFO ReceiverTracker: Sent stop signal to all 1 receivers 19/08/13 19:54:02 ERROR ReceiverTracker: Deregistered receiver for stream 0: Stopped by driver 19/08/13 19:54:02 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 70) in 35055 ms on datanode01 (executor 2) (1/1) 19/08/13 19:54:02 INFO YarnClusterScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool 19/08/13 19:54:02 INFO DAGScheduler: ResultStage 2 (start at VoiceApplication2.java:128) finished in 35.086 s 19/08/13 19:54:02 INFO ReceiverTracker: Waiting for receiver job to terminate gracefully 19/08/13 19:54:02 INFO ReceiverTracker: Waited for receiver job to terminate gracefully 19/08/13 19:54:02 INFO ReceiverTracker: All of the receivers have deregistered successfully 19/08/13 19:54:02 INFO ReceiverTracker: ReceiverTracker stopped 19/08/13 19:54:02 INFO JobGenerator: Stopping JobGenerator gracefully 19/08/13 19:54:02 INFO JobGenerator: Waiting for all received blocks to be consumed for job generation 19/08/13 19:54:02 INFO JobGenerator: Waited for all received blocks to be consumed for job generation 19/08/13 19:54:12 WARN ShutdownHookManager: ShutdownHook '$anon$2' timeout, java.util.concurrent.TimeoutException java.util.concurrent.TimeoutException at java.util.concurrent.FutureTask.get(FutureTask.java:205) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:67) 19/08/13 19:54:12 ERROR Utils: Uncaught exception in thread pool-1-thread-1 java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at org.apache.spark.streaming.util.RecurringTimer.stop(RecurringTimer.scala:86) at org.apache.spark.streaming.scheduler.JobGenerator.stop(JobGenerator.scala:137) at org.apache.spark.streaming.scheduler.JobScheduler.stop(JobScheduler.scala:123) at org.apache.spark.streaming.StreamingContext$$anonfun$stop$1.apply$mcV$sp(StreamingContext.scala:681) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1357) at org.apache.spark.streaming.StreamingContext.stop(StreamingContext.scala:680) at org.apache.spark.streaming.StreamingContext.org$apache$spark$streaming$StreamingContext$$stopOnShutdown(StreamingContext.scala:714) at org.apache.spark.streaming.StreamingContext$$anonfun$start$1.apply$mcV$sp(StreamingContext.scala:599) at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:216) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1$$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:188) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1988) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anonfun$runAll$1.apply(ShutdownHookManager.scala:188) at scala.util.Try$.apply(Try.scala:192) at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188) at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ```
spark(自带hive)不能读取主子表的数据
【问题详细描述】 spark(自带hive)读取不了主子表的数据,非主表的数据可以读取。spark版本:spark-1.3.0-bin-hadoop2.4 使用的jar包: spark-sequoiadb-1.12.jar sequoiadb-driver-1.12.jar hadoop-sequoiadb-1.12.jar hive-sequoiadb-1.12.jar postgresql-9.4-1201-jdbc41.jar 查询主表错误如下: select * from test201607_cs.tb_order limit 1 ; Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 16.0 failed 4 times, most recent failure: Lost task 0.3 in stage 16.0 (TID 362, sdb-223.3golden.hq): com.sequoiadb.exception.BaseException: errorType:SDB_DMS_CS_NOTEXIST,Collection space does not exist Exception Detail:test201607_cs at com.sequoiadb.base.Sequoiadb.getCollectionSpace(Sequoiadb.java:598) at com.sequoiadb.hive.SdbReader.<init>(SdbReader.java:145) at com.sequoiadb.hive.SdbHiveInputFormat.getRecordReader(SdbHiveInputFormat.java:120) at org.apache.spark.rdd.HadoopRDD$anon$1.<init>(HadoopRDD.scala:236) at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:212) 复制代码 查询非主表结果: select * from test201607_cs.test_hive limit 1 ; +----------+ | shop_id | +----------+ | 10048 | +----------+
spark2.3.3跨集群读取hive2.4.2
问题描述: 旧集群为spark2.1.0,hive2.4.2。新集群为cdh的spark2.3.3+hive3.0.0。hdfs不在一起。我尝试用spark2.3.3去读旧集群的hive2.4.2。在spark-submit的时候--files添加了 hive-site.xml。 里面定义了 ``` spark.sql.warehouse.dir=hdfs://master:9000/apps/hive/warehouse hive.metastore.uris=thrift://master:9083 ``` 这里的master为旧集群的地址。 当我将依赖包打进要执行的jar的时候执行抛出如下异常: ``` class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator ``` 而当我仅执行original的jar包, 依赖包选择spark-submit --jars的方式引入时,则抛出这个异常 ``` org.apache.thrift.TApplicationException: Invalid method name: 'get_all_functions' ```
爬虫福利二 之 妹子图网MM批量下载
爬虫福利一:27报网MM批量下载    点击 看了本文,相信大家对爬虫一定会产生强烈的兴趣,激励自己去学习爬虫,在这里提前祝:大家学有所成! 目标网站:妹子图网 环境:Python3.x 相关第三方模块:requests、beautifulsoup4 Re:各位在测试时只需要将代码里的变量 path 指定为你当前系统要保存的路径,使用 python xxx.py 或IDE运行即可。
Java学习的正确打开方式
在博主认为,对于入门级学习java的最佳学习方法莫过于视频+博客+书籍+总结,前三者博主将淋漓尽致地挥毫于这篇博客文章中,至于总结在于个人,实际上越到后面你会发现学习的最好方式就是阅读参考官方文档其次就是国内的书籍,博客次之,这又是一个层次了,这里暂时不提后面再谈。博主将为各位入门java保驾护航,各位只管冲鸭!!!上天是公平的,只要不辜负时间,时间自然不会辜负你。 何谓学习?博主所理解的学习,它
程序员必须掌握的核心算法有哪些?
由于我之前一直强调数据结构以及算法学习的重要性,所以就有一些读者经常问我,数据结构与算法应该要学习到哪个程度呢?,说实话,这个问题我不知道要怎么回答你,主要取决于你想学习到哪些程度,不过针对这个问题,我稍微总结一下我学过的算法知识点,以及我觉得值得学习的算法。这些算法与数据结构的学习大多数是零散的,并没有一本把他们全部覆盖的书籍。下面是我觉得值得学习的一些算法以及数据结构,当然,我也会整理一些看过
大学四年自学走来,这些私藏的实用工具/学习网站我贡献出来了
大学四年,看课本是不可能一直看课本的了,对于学习,特别是自学,善于搜索网上的一些资源来辅助,还是非常有必要的,下面我就把这几年私藏的各种资源,网站贡献出来给你们。主要有:电子书搜索、实用工具、在线视频学习网站、非视频学习网站、软件下载、面试/求职必备网站。 注意:文中提到的所有资源,文末我都给你整理好了,你们只管拿去,如果觉得不错,转发、分享就是最大的支持了。 一、PDF搜索网站推荐 对于大部
linux系列之常用运维命令整理笔录
本博客记录工作中需要的linux运维命令,大学时候开始接触linux,会一些基本操作,可是都没有整理起来,加上是做开发,不做运维,有些命令忘记了,所以现在整理成博客,当然vi,文件操作等就不介绍了,慢慢积累一些其它拓展的命令,博客不定时更新 顺便拉下票,我在参加csdn博客之星竞选,欢迎投票支持,每个QQ或者微信每天都可以投5票,扫二维码即可,http://m234140.nofollow.ax.
Vue + Spring Boot 项目实战(十四):用户认证方案与完善的访问拦截
本篇文章主要讲解 token、session 等用户认证方案的区别并分析常见误区,以及如何通过前后端的配合实现完善的访问拦截,为下一步权限控制的实现打下基础。
比特币原理详解
一、什么是比特币 比特币是一种电子货币,是一种基于密码学的货币,在2008年11月1日由中本聪发表比特币白皮书,文中提出了一种去中心化的电子记账系统,我们平时的电子现金是银行来记账,因为银行的背后是国家信用。去中心化电子记账系统是参与者共同记账。比特币可以防止主权危机、信用风险。其好处不多做赘述,这一层面介绍的文章很多,本文主要从更深层的技术原理角度进行介绍。 二、问题引入  假设现有4个人
程序员接私活怎样防止做完了不给钱?
首先跟大家说明一点,我们做 IT 类的外包开发,是非标品开发,所以很有可能在开发过程中会有这样那样的需求修改,而这种需求修改很容易造成扯皮,进而影响到费用支付,甚至出现做完了项目收不到钱的情况。 那么,怎么保证自己的薪酬安全呢? 我们在开工前,一定要做好一些证据方面的准备(也就是“讨薪”的理论依据),这其中最重要的就是需求文档和验收标准。一定要让需求方提供这两个文档资料作为开发的基础。之后开发
网页实现一个简单的音乐播放器(大佬别看。(⊙﹏⊙))
今天闲着无事,就想写点东西。然后听了下歌,就打算写个播放器。 于是乎用h5 audio的加上js简单的播放器完工了。 欢迎 改进 留言。 演示地点跳到演示地点 html代码如下`&lt;!DOCTYPE html&gt; &lt;html&gt; &lt;head&gt; &lt;title&gt;music&lt;/title&gt; &lt;meta charset="utf-8"&gt
Python十大装B语法
Python 是一种代表简单思想的语言,其语法相对简单,很容易上手。不过,如果就此小视 Python 语法的精妙和深邃,那就大错特错了。本文精心筛选了最能展现 Python 语法之精妙的十个知识点,并附上详细的实例代码。如能在实战中融会贯通、灵活使用,必将使代码更为精炼、高效,同时也会极大提升代码B格,使之看上去更老练,读起来更优雅。 1. for - else 什么?不是 if 和 else 才
数据库优化 - SQL优化
前面一篇文章从实例的角度进行数据库优化,通过配置一些参数让数据库性能达到最优。但是一些“不好”的SQL也会导致数据库查询变慢,影响业务流程。本文从SQL角度进行数据库优化,提升SQL运行效率。 判断问题SQL 判断SQL是否有问题时可以通过两个表象进行判断: 系统级别表象 CPU消耗严重 IO等待严重 页面响应时间过长
2019年11月中国大陆编程语言排行榜
2019年11月2日,我统计了某招聘网站,获得有效程序员招聘数据9万条。针对招聘信息,提取编程语言关键字,并统计如下: 编程语言比例 rank pl_ percentage 1 java 33.62% 2 c/c++ 16.42% 3 c_sharp 12.82% 4 javascript 12.31% 5 python 7.93% 6 go 7.25% 7
通俗易懂地给女朋友讲:线程池的内部原理
餐厅的约会 餐盘在灯光的照耀下格外晶莹洁白,女朋友拿起红酒杯轻轻地抿了一小口,对我说:“经常听你说线程池,到底线程池到底是个什么原理?”我楞了一下,心里想女朋友今天是怎么了,怎么突然问出这么专业的问题,但做为一个专业人士在女朋友面前也不能露怯啊,想了一下便说:“我先给你讲讲我前同事老王的故事吧!” 大龄程序员老王 老王是一个已经北漂十多年的程序员,岁数大了,加班加不动了,升迁也无望,于是拿着手里
经典算法(5)杨辉三角
写在前面: 我是 扬帆向海,这个昵称来源于我的名字以及女朋友的名字。我热爱技术、热爱开源、热爱编程。技术是开源的、知识是共享的。 这博客是对自己学习的一点点总结及记录,如果您对 Java、算法 感兴趣,可以关注我的动态,我们一起学习。 用知识改变命运,让我们的家人过上更好的生活。 目录一、杨辉三角的介绍二、杨辉三角的算法思想三、代码实现1.第一种写法2.第二种写法 一、杨辉三角的介绍 百度
腾讯算法面试题:64匹马8个跑道需要多少轮才能选出最快的四匹?
昨天,有网友私信我,说去阿里面试,彻底的被打击到了。问了为什么网上大量使用ThreadLocal的源码都会加上private static?他被难住了,因为他从来都没有考虑过这个问题。无独有偶,今天笔者又发现有网友吐槽了一道腾讯的面试题,我们一起来看看。 腾讯算法面试题:64匹马8个跑道需要多少轮才能选出最快的四匹? 在互联网职场论坛,一名程序员发帖求助到。二面腾讯,其中一个算法题:64匹
面试官:你连RESTful都不知道我怎么敢要你?
面试官:了解RESTful吗? 我:听说过。 面试官:那什么是RESTful? 我:就是用起来很规范,挺好的 面试官:是RESTful挺好的,还是自我感觉挺好的 我:都挺好的。 面试官:… 把门关上。 我:… 要干嘛?先关上再说。 面试官:我说出去把门关上。 我:what ?,夺门而去 文章目录01 前言02 RESTful的来源03 RESTful6大原则1. C-S架构2. 无状态3.统一的接
为啥国人偏爱Mybatis,而老外喜欢Hibernate/JPA呢?
关于SQL和ORM的争论,永远都不会终止,我也一直在思考这个问题。昨天又跟群里的小伙伴进行了一番讨论,感触还是有一些,于是就有了今天这篇文。 声明:本文不会下关于Mybatis和JPA两个持久层框架哪个更好这样的结论。只是摆事实,讲道理,所以,请各位看官勿喷。 一、事件起因 关于Mybatis和JPA孰优孰劣的问题,争论已经很多年了。一直也没有结论,毕竟每个人的喜好和习惯是大不相同的。我也看
SQL-小白最佳入门sql查询一
一 说明 如果是初学者,建议去网上寻找安装Mysql的文章安装,以及使用navicat连接数据库,以后的示例基本是使用mysql数据库管理系统; 二 准备前提 需要建立一张学生表,列分别是id,名称,年龄,学生信息;本示例中文章篇幅原因SQL注释略; 建表语句: CREATE TABLE `student` ( `id` int(11) NOT NULL AUTO_INCREMENT, `
项目中的if else太多了,该怎么重构?
介绍 最近跟着公司的大佬开发了一款IM系统,类似QQ和微信哈,就是聊天软件。我们有一部分业务逻辑是这样的 if (msgType = "文本") { // dosomething } else if(msgType = "图片") { // doshomething } else if(msgType = "视频") { // doshomething } else { // dosho
“狗屁不通文章生成器”登顶GitHub热榜,分分钟写出万字形式主义大作
一、垃圾文字生成器介绍 最近在浏览GitHub的时候,发现了这样一个骨骼清奇的雷人项目,而且热度还特别高。 项目中文名:狗屁不通文章生成器 项目英文名:BullshitGenerator 根据作者的介绍,他是偶尔需要一些中文文字用于GUI开发时测试文本渲染,因此开发了这个废话生成器。但由于生成的废话实在是太过富于哲理,所以最近已经被小伙伴们给玩坏了。 他的文风可能是这样的: 你发现,
程序员:我终于知道post和get的区别
IT界知名的程序员曾说:对于那些月薪三万以下,自称IT工程师的码农们,其实我们从来没有把他们归为我们IT工程师的队伍。他们虽然总是以IT工程师自居,但只是他们一厢情愿罢了。 此话一出,不知激起了多少(码农)程序员的愤怒,却又无可奈何,于是码农问程序员。 码农:你知道get和post请求到底有什么区别? 程序员:你看这篇就知道了。 码农:你月薪三万了? 程序员:嗯。 码农:你是怎么做到的? 程序员:
《程序人生》系列-这个程序员只用了20行代码就拿了冠军
你知道的越多,你不知道的越多 点赞再看,养成习惯GitHub上已经开源https://github.com/JavaFamily,有一线大厂面试点脑图,欢迎Star和完善 前言 这一期不算《吊打面试官》系列的,所有没前言我直接开始。 絮叨 本来应该是没有这期的,看过我上期的小伙伴应该是知道的嘛,双十一比较忙嘛,要值班又要去帮忙拍摄年会的视频素材,还得搞个程序员一天的Vlog,还要写BU
加快推动区块链技术和产业创新发展,2019可信区块链峰会在京召开
      11月8日,由中国信息通信研究院、中国通信标准化协会、中国互联网协会、可信区块链推进计划联合主办,科技行者协办的2019可信区块链峰会将在北京悠唐皇冠假日酒店开幕。   区块链技术被认为是继蒸汽机、电力、互联网之后,下一代颠覆性的核心技术。如果说蒸汽机释放了人类的生产力,电力解决了人类基本的生活需求,互联网彻底改变了信息传递的方式,区块链作为构造信任的技术有重要的价值。   1
程序员把地府后台管理系统做出来了,还有3.0版本!12月7号最新消息:已在开发中有github地址
第一幕:缘起 听说阎王爷要做个生死簿后台管理系统,我们派去了一个程序员…… 996程序员做的梦: 第一场:团队招募 为了应对地府管理危机,阎王打算找“人”开发一套地府后台管理系统,于是就在地府总经办群中发了项目需求。 话说还是中国电信的信号好,地府都是满格,哈哈!!! 经常会有外行朋友问:看某网站做的不错,功能也简单,你帮忙做一下? 而这次,面对这样的需求,这个程序员
Android 9.0系统新特性,对刘海屏设备进行适配
其实Android 9.0系统已经是去年推出的“老”系统了,这个系统中新增了一个比较重要的特性,就是对刘海屏设备进行了支持。一直以来我也都有打算针对这个新特性好好地写一篇文章,但是为什么直到拖到了Android 10.0系统都发布了才开始写这篇文章呢?当然,一是因为我这段时间确实比较忙,今年几乎绝大部分的业余时间都放到写新书上了。但是最主要的原因并不是这个,而是因为刘海屏设备的适配存在一定的特殊性
网易云6亿用户音乐推荐算法
网易云音乐是音乐爱好者的集聚地,云音乐推荐系统致力于通过 AI 算法的落地,实现用户千人千面的个性化推荐,为用户带来不一样的听歌体验。 本次分享重点介绍 AI 算法在音乐推荐中的应用实践,以及在算法落地过程中遇到的挑战和解决方案。 将从如下两个部分展开: AI 算法在音乐推荐中的应用 音乐场景下的 AI 思考 从 2013 年 4 月正式上线至今,网易云音乐平台持续提供着:乐屏社区、UGC
【技巧总结】位运算装逼指南
位算法的效率有多快我就不说,不信你可以去用 10 亿个数据模拟一下,今天给大家讲一讲位运算的一些经典例子。不过,最重要的不是看懂了这些例子就好,而是要在以后多去运用位运算这些技巧,当然,采用位运算,也是可以装逼的,不信,你往下看。我会从最简单的讲起,一道比一道难度递增,不过居然是讲技巧,那么也不会太难,相信你分分钟看懂。 判断奇偶数 判断一个数是基于还是偶数,相信很多人都做过,一般的做法的代码如下
日均350000亿接入量,腾讯TubeMQ性能超过Kafka
整理 | 夕颜出品 | AI科技大本营(ID:rgznai100) 【导读】近日,腾讯开源动作不断,相继开源了分布式消息中间件TubeMQ,基于最主流的 OpenJDK8开发的
8年经验面试官详解 Java 面试秘诀
    作者 | 胡书敏 责编 | 刘静 出品 | CSDN(ID:CSDNnews) 本人目前在一家知名外企担任架构师,而且最近八年来,在多家外企和互联网公司担任Java技术面试官,前后累计面试了有两三百位候选人。在本文里,就将结合本人的面试经验,针对Java初学者、Java初级开发和Java开发,给出若干准备简历和准备面试的建议。   Java程序员准备和投递简历的实
面试官如何考察你的思维方式?
1.两种思维方式在求职面试中,经常会考察这种问题:北京有多少量特斯拉汽车? 某胡同口的煎饼摊一年能卖出多少个煎饼? 深圳有多少个产品经理? 一辆公交车里能装下多少个乒乓球? 一
碎片化的时代,如何学习
今天周末,和大家聊聊学习这件事情。 在如今这个社会,我们的时间被各类 APP 撕的粉碎。 刷知乎、刷微博、刷朋友圈; 看论坛、看博客、看公号; 等等形形色色的信息和知识获取方式一个都不错过。 貌似学了很多,但是却感觉没什么用。 要解决上面这些问题,首先要分清楚一点,什么是信息,什么是知识。 那什么是信息呢? 你一切听到的、看到的,都是信息,比如微博上的明星出轨、微信中的表情大战、抖音上的...
so easy! 10行代码写个"狗屁不通"文章生成器
前几天,GitHub 有个开源项目特别火,只要输入标题就可以生成一篇长长的文章。背后实现代码一定很复杂吧,里面一定有很多高深莫测的机器学习等复杂算法不过,当我看了源代码之后这程序不到50
知乎高赞:中国有什么拿得出手的开源软件产品?(整理自本人原创回答)
知乎高赞:中国有什么拿得出手的开源软件产品? 在知乎上,有个问题问“中国有什么拿得出手的开源软件产品(在 GitHub 等社区受欢迎度较好的)?” 事实上,还不少呢~ 本人于2019.7.6进行了较为全面的 回答 - Bravo Yeung,获得该问题下回答中得最高赞(236赞和1枚专业勋章),对这些受欢迎的 Github 开源项目分类整理如下: 分布式计算、云平台相关工具类 1.SkyWalk
MySQL数据库总结
文章目录一、数据库简介二、MySQL数据类型(5.5版本)三、Sql语句(1)Sql语句简介(2)数据定义语言DDLcreate,alter,drop(3)数据操纵语言DMLupdate,insert,delete(4)数据控制语言DCLgrant,revoke(5)数据查询语言DQLselect(6)分组查询与分页查询group by,limit四、完整性约束(单表)五、多表查询六、MySQL数
记一次腾讯面试:进程之间究竟有哪些通信方式?如何通信? ---- 告别死记硬背
有一次面试的时候,被问到进程之间有哪些通信方式,不过由于之前没深入思考且整理过,说的并不好。想必大家也都知道进程有哪些通信方式,可是我猜很多人都是靠着”背“来记忆的,所以今天的这篇文章,讲给大家详细着讲解他们是如何通信的,让大家尽量能够理解他们之间的区别、优缺点等,这样的话,以后面试官让你举例子,你也能够顺手拈来。 1、管道 我们来看一条 Linux 的语句 netstat -tulnp | gr...
20行Python代码爬取王者荣耀全英雄皮肤
引言 王者荣耀大家都玩过吧,没玩过的也应该听说过,作为时下最火的手机MOBA游戏,咳咳,好像跑题了。我们今天的重点是爬取王者荣耀所有英雄的所有皮肤,而且仅仅使用20行Python代码即可完成。 准备工作 爬取皮肤本身并不难,难点在于分析,我们首先得得到皮肤图片的url地址,话不多说,我们马上来到王者荣耀的官网: 我们点击英雄资料,然后随意地选择一位英雄,接着F12打开调试台,找到英雄原皮肤的图片
程序设计的5个底层逻辑,决定你能走多快
阿里妹导读:肉眼看计算机是由CPU、内存、显示器这些硬件设备组成,但大部分人从事的是软件开发工作。计算机底层原理就是连通硬件和软件的桥梁,理解计算机底层原理才能在程序设计这条路上越走越快,越走越轻松。从操作系统层面去理解高级编程语言的执行过程,会发现好多软件设计都是同一种套路,很多语言特性都依赖于底层机制,今天董鹏为你一一揭秘。 结合 CPU 理解一行 Java 代码是怎么执行的 根据冯·诺...
张小龙-年薪近3亿的微信之父,他是如何做到的?
张小龙生于湖南邵东魏家桥镇, 家庭主要特点:穷。 不仅自己穷,亲戚也都很穷,可以说穷以类聚。爷爷做过铜匠,总的来说,标准的劳动阶级出身。 家有兄弟两人, 一个小龙,一个小虎。 小虎好动,与邻里打成一片, 小龙好静,喜好读书。 “文静的像个妹子。”张小龙的表哥如是说。 穷文富武,做个读书郎是个不错的选择。 87年至94年, 华中科技大学本硕连读。 本科就读电信系, 不喜欢上课
阿里靠什么武功秘籍渡过“双十一“的天量冲击
双十一大概会产生多大的数据量呢,可能大家没概念,举个例子央视拍了这么多年电视新闻节目,几十年下来他存了大概80P的数据。而今年双11一天,阿里要处理970P的数据,做为一个IT人,笔者认为今年”双十一“阿里最大的技术看点有有以下两个: 阿里的数据库,也就是刚刚拿下TPC冠军的OcceanBase,处理峰值也达到了骇人听闻的6100万次/秒, 阿里核心系统百分百上云了。 如果把信息系统比做一个武
西游记团队中如果需要裁掉一个人,会先裁掉谁?
2019年互联网寒冬,大批企业开始裁员,下图是网上流传的一张截图: 裁员不可避免,那如何才能做到不管大环境如何变化,自身不受影响呢? 我们先来看一个有意思的故事,如果西游记取经团队需要裁员一名,会裁掉谁呢,为什么? 西游记团队组成: 1.唐僧 作为团队teamleader,有很坚韧的品性和极高的原则性,不达目的不罢休,遇到任何问题,都没有退缩过,又很得上司支持和赏识(直接得到唐太宗的任命,既给
相关热词 c# 二进制截断字符串 c#实现窗体设计器 c#检测是否为微信 c# plc s1200 c#里氏转换原则 c# 主界面 c# do loop c#存为组套 模板 c# 停掉协程 c# rgb 读取图片
立即提问

相似问题

1
【急!】java访问hbase出错,无明显异常,无法读取对应表的信息(需求就是从Hbase之中的某张表读取信息)
1
flume上传文件到hadoop,没有文件时正常,有文件时抛出DistributeFileSystem not found?
1
kettle 7.1版本,连接hadoop2.7.3版本,无法读取hadoop文件目录
2
flink在hadoop yarn运行出错,报相应的jar找不到(self4j)
0
运行Hadoop中的Wordcount时报错?
1
在Linux hadoop环境中运行sh脚本,报异常java.io.IOException: No FileSystem for scheme: E
0
Hadoop运行WordCount程序有输入,不生成输出的文件
2
maven仓库导入jar包后,无法引用类
2
springboot使用hadoop中一个copyBytes功能,然后启动项目之后就提示缺少jar包
1
大数据hadoop3.x.x在win10环境下编译后的jar包
0
hadoop运行start-dfs.sh时,报master: ERROR: JAVA_HOME is not set and could not be found.
0
hadoop jar运行jar包问题!请问这种情况如何解决?谢谢
0
hadoop jar运行jar包遇到的问题?
1
Failed to retrieve data from /webhdfs/v1/?op=LISTSTATUS: Server Error,同时无法put文件到hdfs
1
hadoop本地调试在输入路径中指定到文件名字才不报错,但是不指定具体的文件名字就报空指针
0
部署单节点Hadoop下面这代码是什么意思
1
hadoop里面gz大文件切割为小文件?
4
Eclipse上运行MapReduce程序时,win10系统用户名中间有空格导致tmp文件生成&读取错误
1
Hadoop集群执行wordcount出现的一些报错信息
1
hadoop的DataNode节点的问题