大数据-输入命令为:hdfs dfs -ls / ，什么也没返回，HFDS根目录下一片空白—

输入命令为:hdfs dfs -ls / ，什么也没返回，HFDS根目录下一片空白

我的是hadoop3.0，伪分布式部署已经按照教程起来了，为什么什么都搜不到呢

liugen@ubuntu:/$ jps
80624 DataNode
93379 Jps
80819 SecondaryNameNode
80493 NameNode

编辑于：2018.10.24 10:23 发布于：2018.10.23 07:50

大数据

| 评论0 | 收藏 | 浏览11199 |

: 望隐少年

声望： 2

1个回答

按赞数排序

启动yarn，去 master:50070 上看一下，你的hdfs上是否有文件

发布于：2018.10.23 09:50

评论 3 |

| 已采纳

: 学编程的萌新

声望： 17

: ~信仰 回复望隐少年: 怎么解决的呢？不是一开始dfs就有几个目录吗？; 6 个月之前回复

: 望隐少年 谢谢大神，原来是我没有创建hdfs目录; 大约一年之前回复

: 望隐少年 谢谢大神我试试; 大约一年之前回复

: 插入链接

本地上传网络图片

上传中...

浏览

上传图片

插入图片

其他相关推荐

hdfs dfs -ls / 查询到的是本地文件目录，这是为什么？: 已经配置了hadoop的环境变量，也source过了，但是只要不在hadoop/bin下面使用hdfs dfs -ls / 查询到的就是本地文件目录，并且报错： ``` Warning: fs.defaultFS is not set when running "ls" command. ``` 如图： ![图片说明](https://img-ask.csdn.net/upload/201902/16/1550308582_593413.png) 但如果在bin目录下，显示正确，如图： ![图片说明](https://img-ask.csdn.net/upload/201902/16/1550307272_437929.png) 环境变量如图： ![图片说明](https://img-ask.csdn.net/upload/201902/16/1550307296_326384.png) 上网查了有人说core-site.xml有问题，我不是这个问题，配置文件如下： ![图片说明](https://img-ask.csdn.net/upload/201902/16/1550307367_203815.png) 请问有人知道这是怎么回事吗？

hadoop集群，hdfs dfs -ls / 目录出错: 搭建了一个hadoop集群，用hdfs dfs -ls /命令，列出的是本地系统的根目录。用hdfs dfs -ls hdfs://servicename/ 列出的目录才是hdfs上的目录，可能是什么原因？执行hive创建的目录也是在本地系统目录上。集群的配置如下集群规划：主机名 IP 安装的软件运行的进程 hadoop01 192.168.175.129 jdk、hadoop NameNode、DFSZKFailoverController(zkfc) hadoop02 192.168.175.127 jdk、hadoop NameNode、DFSZKFailoverController(zkfc) hadoop03 192.168.175.126 jdk、hadoop ResourceManager hadoop04 192.168.175.125 jdk、hadoop ResourceManager hadoop05 192.168.175.124 jdk、hadoop、zookeeper DataNode、NodeManager、JournalNode、QuorumPeerMain hadoop06 192.168.175.123 jdk、hadoop、zookeeper DataNode、NodeManager、JournalNode、QuorumPeerMain hadoop07 192.168.175.122 jdk、hadoop、zookeeper DataNode、NodeManager、JournalNode、QuorumPeerMain windows:NLB LINUX:LVS 1.liunx虚拟机安装后，虚拟机连接模式要选择host-only模式。然后分配IP(以hadoop01为例) DEVICE="eth0" BOOTPROTO="static" ### HWADDR="00:0C:29:3C:BF:E7" IPV6INIT="yes" NM_CONTROLLED="yes" ONBOOT="yes" TYPE="Ethernet" UUID="ce22eeca-ecde-4536-8cc2-ef0dc36d4a8c" IPADDR="192.168.175.129" ### NETMASK="255.255.255.0" ### GATEWAY="192.168.175.1" ### 2.修改主机名： vim /etc/sysconfig/network NETWORKING=yes HOSTNAME=hadoop01 ### 3.关闭防火墙： #查看防火墙状态 service iptables status #关闭防火墙 service iptables stop #查看防火墙开机启动状态 chkconfig iptables --list #关闭防火墙开机启动 chkconfig iptables off 4.免登录配置： #生成ssh免登陆密钥 #进入到我的home目录 cd ~/.ssh ssh-keygen -t rsa （四个回车）执行完这个命令后，会生成两个文件id_rsa（私钥）、id_rsa.pub（公钥）将公钥拷贝到要免登陆的机器上 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys 或若报错ssh-copy-id: ERROR: No identities found，是因为找不到公钥路径,加上-i然后再加上路径即可则用 $ ssh-copy-id -i ~/.ssh/id_rsa.pub user@remote_ip 5.主机IP映射关系(/etc/hosts每台机器上都要配置全部映射关系) 192.168.175.129 hadoop01 192.168.175.127 hadoop02 192.168.175.126 hadoop03 192.168.175.125 hadoop04 192.168.175.124 hadoop05 192.168.175.123 hadoop06 192.168.175.122 hadoop07 6./etc/profile下配置java环境变量： export JAVA_HOME=/lichangwu/jdk1.7.0_79 export PATH=$PATH:$JAVA_HOME/bin #刷新profile source /etc/profile 若版本报错,vi /etc/selinux/config,设置SELINUX=disabled,然后重启虚拟机 7.安装zookeeper: 1.安装配置zooekeeper集群（在hadoop05上）: 1.1解压 tar -zxvf zookeeper-3.4.6.tar.gz -C /lichangwu/ 1.2修改配置 cd /lichangwu/zookeeper-3.4.6/conf/ cp zoo_sample.cfg zoo.cfg vim zoo.cfg 修改：dataDir=/lichangwu/zookeeper-3.4.6/tmp 在最后添加： server.1=hadoop05:2888:3888 server.2=hadoop06:2888:3888 server.3=hadoop07:2888:3888 保存退出然后创建一个tmp文件夹 mkdir /lichangwu/zookeeper-3.4.6/tmp 再创建一个空文件 touch /lichangwu/zookeeper-3.4.6/tmp/myid 最后向该文件写入ID echo 1 > /lichangwu/zookeeper-3.4.6/tmp/myid 1.3将配置好的zookeeper拷贝到其他节点(首先分别在hadoop06、hadoop07根目录下创建一个lichangwu目录：mkdir /lichangwu) scp -r /lichangwu/zookeeper-3.4.6/ hadoop06:/lichangwu/ scp -r /lichangwu/zookeeper-3.4.6/ hadoop07:/lichangwu/ 注意：修改hadoop06、hadoop07对应/lichangwu/zookeeper-3.4.6/tmp/myid内容 itcast06： echo 2 > /lichangwu/zookeeper-3.4.6/tmp/myid itcast07： echo 3 > /lichangwu/zookeeper-3.4.6/tmp/myid 8.安装配置hadoop集群（在hadoop01上操作）: 2.1解压 tar -zxvf hadoop-2.4.1.tar.gz -C /lichangwu/ 2.2配置HDFS（hadoop2.0所有的配置文件都在$HADOOP_HOME/etc/hadoop目录下） #将hadoop添加到环境变量中 vim /etc/profile export JAVA_HOME=/lichangwu/jdk1.7.0_79 export HADOOP_HOME=/lichangwu/hadoop-2.4.1 export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin #hadoop2.0的配置文件全部在$HADOOP_HOME/etc/hadoop下 cd /lichangwu/hadoop-2.4.1/etc/hadoop 2.2.1修改hadoo-env.sh export JAVA_HOME=/lichangwu/jdk1.7.0_79 2.2.2修改core-site.xml <configuration>  <property> <name>fs.defaultFS</name> <value>hdfs://ns1</value> </property>  <property> <name>hadoop.tmp.dir</name> <value>/lichangwu/hadoop-2.4.1/tmp</value> </property>  <property> <name>ha.zookeeper.quorum</name> <value>hadoop05:2181,hadoop06:2181,hadoop07:2181</value> </property> </configuration> 2.2.3修改hdfs-site.xml <configuration>  <property> <name>dfs.nameservices</name> <value>ns1</value> </property>  <property> <name>dfs.ha.namenodes.ns1</name> <value>nn1,nn2</value> </property>  <property> <name>dfs.namenode.rpc-address.ns1.nn1</name> <value>hadoop01:9000</value> </property>  <property> <name>dfs.namenode.http-address.ns1.nn1</name> <value>hadoop01:50070</value> </property>  <property> <name>dfs.namenode.rpc-address.ns1.nn2</name> <value>hadoop02:9000</value> </property>  <property> <name>dfs.namenode.http-address.ns1.nn2</name> <value>hadoop02:50070</value> </property>  <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://hadoop05:8485;hadoop06:8485;hadoop07:8485/ns1</value> </property>  <property> <name>dfs.journalnode.edits.dir</name> <value>/lichangwu/hadoop-2.4.1/journal</value> </property>  <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property>  <property> <name>dfs.client.failover.proxy.provider.ns1</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property>  <property> <name>dfs.ha.fencing.methods</name> <value> sshfence shell(/bin/true) </value> </property>  <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/hadoop/.ssh/id_rsa</value> </property>  <property> <name>dfs.ha.fencing.ssh.connect-timeout</name> <value>30000</value> </property> </configuration> 2.2.4修改mapred-site.xml <configuration>  <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration> 2.2.5修改yarn-site.xml <configuration>  <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property>  <property> <name>yarn.resourcemanager.cluster-id</name> <value>yrc</value> </property>  <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property>  <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>hadoop03</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>hadoop04</value> </property>  <property> <name>yarn.resourcemanager.zk-address</name> <value>hadoop05:2181,hadoop06:2181,hadoop07:2181</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration> 2.2.6修改slaves(slaves是指定子节点的位置，因为要在itcast01上启动HDFS、在itcast03启动yarn，所以itcast01上的slaves文件指定的是datanode的位置，itcast03上的slaves文件指定的是nodemanager的位置) hadoop05 hadoop06 hadoop07 2.2.7配置免密码登陆 #首先要配置itcast01到hadoop02、hadoop03、hadoop04、hadoop05、hadoop06、hadoop07的免密码登陆 #在hadoop01上生产一对钥匙 ssh-keygen -t rsa #将公钥拷贝到其他节点，包括自己 ssh-coyp-id hadoop01 ssh-coyp-id hadoop02 ssh-coyp-id hadoop03 ssh-coyp-id hadoop04 ssh-coyp-id hadoop05 ssh-coyp-id hadoop06 ssh-coyp-id hadoop07 #配置hadoop03到hadoop04、hadoop05、hadoop06、hadoop07的免密码登陆 #在hadoop03上生产一对钥匙 ssh-keygen -t rsa #将公钥拷贝到其他节点 ssh-coyp-id hadoop04 ssh-coyp-id hadoop05 ssh-coyp-id hadoop06 ssh-coyp-id hadoop07 #注意：两个namenode之间要配置ssh免密码登陆，别忘了配置hadoop02到hadoop01的免登陆在hadoop02上生产一对钥匙 ssh-keygen -t rsa ssh-coyp-id -i hadoop01 2.4将配置好的hadoop拷贝到其他节点 scp -r hadoop-2.4.1/ hadoop02:/lichangwu/hadoop-2.4.1/ scp -r hadoop-2.4.1/ hadoop03:/lichangwu/hadoop-2.4.1/ scp -r hadoop-2.4.1/ hadoop04:/lichangwu/hadoop-2.4.1/ scp -r hadoop-2.4.1/ hadoop05:/lichangwu/hadoop-2.4.1/ scp -r hadoop-2.4.1/ hadoop06:/lichangwu/hadoop-2.4.1/ scp -r hadoop-2.4.1/ hadoop07:/lichangwu/hadoop-2.4.1/ ###注意：严格按照下面的步骤 2.5启动zookeeper集群（分别在hadoop05、hadoop06、hadoop07上启动zk） cd /lichangwu/zookeeper-3.4.6/bin/ ./zkServer.sh start #查看状态：一个leader，两个follower ./zkServer.sh status 2.6启动journalnode（分别在在hadoop05、hadoop06、hadoop07上执行） cd /lichangwu/hadoop-2.4.1 sbin/hadoop-daemon.sh start journalnode #运行jps命令检验，hadoop05、hadoop06、hadoop07上多了JournalNode进程 2.7格式化HDFS #在hadoop01上执行命令: hdfs namenode -format #格式化后会在根据core-site.xml中的hadoop.tmp.dir配置生成个文件，这里我配置的是/lichangwu/hadoop-2.4.1/tmp，然后将/lichangwu/hadoop-2.4.1/tmp拷贝到hadoop02的/lichangwu/hadoop-2.4.1/下。 scp -r tmp/ hadoop02:/lichangwu/hadoop-2.4.1/ 2.8格式化ZK(在hadoop01上执行即可) hdfs zkfc -formatZK 2.9启动HDFS(在hadoop01上执行) sbin/start-dfs.sh 2.10启动YARN(#####注意#####：是在hadoop03上执行start-yarn.sh，如果hadoop04上没有启动成功，则在hadoop04上再启动一次start-yarn.sh; 把namenode和resourcemanager分开是因为性能问题，因为他们都要占用大量资源，所以把他们分开了，他们分开了就要分别在不同的机器上启动) sbin/start-yarn.sh 到此，hadoop-2.4.1配置完毕，可以统计浏览器访问: http://192.168.175.129:50070 NameNode 'hadoop01:9000' (active) http://192.168.175.127:50070 NameNode 'hadoop02:9000' (standby)

HADOOP集群主机上运行HADOOP相关命令回车后响应慢: HADOOP集群主机上运行HADOOP相关命令回车后响应慢，如集群主机上输入hive\hbase shell或者hdfs dfs -ls / 回车后终端响应非常慢，至少10秒上才出来 Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/jars/hive-common-0.13.1-cdh5.3.1.jar!/hive-log4j.properties hive>

在HDFS中创建用户目录失败: 环境配置没错，启动无误，输入jps，显示了6个进程。 hadoop目录: /usr/lib/hadoop 权限是：777 所属者和所属组都为普通用户：sister 使用的命令： ./bin/hdfs dfs -mkdir -p /user/hadoop 问题：创建用户目录时，没报错，但是到根目录下没有发现 user/hadoop 文件，不知道是什么原因，请大神支招！谢谢了

Hadoop集群执行wordcount出现的一些报错信息: 我是一个Hadoop学习的新手，请大家帮助一下，非常的感谢！我自己在虚拟机使用docker搭建了一个Hadoop集群，docker镜像是使用的ubuntu18.04 首先我的Hadoop1主节点上开启了以下服务： ``` root@hadoop1:/usr/local/hadoop# jps 2058 NameNode 2266 SecondaryNameNode 2445 ResourceManager 2718 Jps ``` 下面是两个从节点的服务： ``` root@hadoop2:~# jps 294 DataNode 550 Jps 406 NodeManager ``` ``` root@hadoop3:~# jps 543 Jps 399 NodeManager 287 DataNode ``` hadoop1(主节点)在云端创建一个/data/input的文件夹结构 ``` root@hadoop1:/usr/local/hadoop#bin/hdfs dfs -mkdir -p /data/input WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.9.2.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release ``` 就是上面的一堆警告，下面我每执行一次bin/hdfs dfs都会有这个警告，请问这种警告对于整个Hadoop集群有没有影响，怎样将这个警告消除。 ``` 下面这是将test1文件推送带云端时也出现同样的报警 root@hadoop1:/usr/local/hadoop#bin/hdfs dfs -put test1 /data/input WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.9.2.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release ``` 这是查看推送到云端文件的时候也出现这种报警 ``` root@hadoop1:/usr/local/hadoop#bin/hdfs dfs -ls /data/input WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.9.2.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Found 1 items -rw-r--r-- 1 root supergroup 60 2019-09-15 08:07 /data/input/test1 ``` 这是执行share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar也出现这种报警 ``` root@hadoop1:/usr/local/hadoop#bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar wordcount /data/input/test1 /data/output/test1 WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.9.2.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release ``` 这是执行后查看wordcount的执行结果也出现这种报警 ``` root@hadoop1:/usr/local/hadoop# bin/hdfs dfs -cat /data/output/test1/part-r-00000 WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-2.9.2.jar) to method sun.security.krb5.Config.getInstance() WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release first 1 hello 2 is 2 my 2 test1 1 testwordcount 1 this 2 ``` 有哪位大神能否帮我看一下这个问题如何解决，非常感谢！

Failed to retrieve data from /webhdfs/v1/?op=LISTSTATUS: Server Error，同时无法put文件到hdfs: hadoop版本是3.1，ubuntu是18，问题一：浏览hdfs目录显示： Failed to retrieve data from /webhdfs/v1/?op=LISTSTATUS: Server Error 问题二： namenode的log如下： ``` 438 WARN org.eclipse.jetty.servlet.ServletHandler: Error for /webhdfs/v1/ java.lang.NoClassDefFoundError: javax/activation/DataSource at com.sun.xml.bind.v2.model.impl.RuntimeBuiltinLeafInfoImpl.<clinit>(RuntimeBuiltinLeafInfoImpl.java:457) at com.sun.xml.bind.v2.model.impl.RuntimeTypeInfoSetImpl.<init>(RuntimeTypeInfoSetImpl.java:65) at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.createTypeInfoSet(RuntimeModelBuilder.java:133) at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.createTypeInfoSet(RuntimeModelBuilder.java:85) at com.sun.xml.bind.v2.model.impl.ModelBuilder.<init>(ModelBuilder.java:156) at com.sun.xml.bind.v2.model.impl.RuntimeModelBuilder.<init>(RuntimeModelBuilder.java:93) at com.sun.xml.bind.v2.runtime.JAXBContextImpl.getTypeInfoSet(JAXBContextImpl.java:473) at com.sun.xml.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:319) at com.sun.xml.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1170) at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:145) at com.sun.xml.bind.v2.ContextFactory.createContext(ContextFactory.java:236) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:186) at javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:146) at javax.xml.bind.ContextFinder.find(ContextFinder.java:350) at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:446) at javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:409) at com.sun.jersey.server.impl.wadl.WadlApplicationContextImpl.<init>(WadlApplicationContextImpl.java:103) at com.sun.jersey.server.impl.wadl.WadlFactory.init(WadlFactory.java:100) at com.sun.jersey.server.impl.application.RootResourceUriRules.initWadl(RootResourceUriRules.java:169) at com.sun.jersey.server.impl.application.RootResourceUriRules.<init>(RootResourceUriRules.java:106) at com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1359) at com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:180) at com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:799) at com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:795) at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193) at com.sun.jersey.server.impl.application.WebApplicationImpl.initiate(WebApplicationImpl.java:795) at com.sun.jersey.server.impl.application.WebApplicationImpl.initiate(WebApplicationImpl.java:790) at com.sun.jersey.spi.container.servlet.ServletContainer.initiate(ServletContainer.java:509) at com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.initiate(ServletContainer.java:339) at com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:605) at com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:207) at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:394) at com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:577) at javax.servlet.GenericServlet.init(GenericServlet.java:244) at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:643) at org.eclipse.jetty.servlet.ServletHolder.getServlet(ServletHolder.java:499) at org.eclipse.jetty.servlet.ServletHolder.ensureInstance(ServletHolder.java:791) at org.eclipse.jetty.servlet.ServletHolder.prepare(ServletHolder.java:776) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:579) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.ClassNotFoundException: javax.activation.DataSource at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ... 65 more 2019-06-18 15:35:01,950 WARN org.eclipse.jetty.servlet.ServletHandler: /webhdfs/v1/ java.lang.NullPointerException at com.sun.jersey.spi.container.ContainerRequest.<init>(ContainerRequest.java:189) at com.sun.jersey.spi.container.servlet.WebComponent.createRequest(WebComponent.java:446) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:373) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:90) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.base/java.lang.Thread.run(Thread.java:834) 2019-06-18 15:39:17,698 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 3 Total time for transactions(ms): 56 Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 22 2019-06-18 15:39:25,202 WARN org.eclipse.jetty.servlet.ServletHandler: /webhdfs/v1/ java.lang.NullPointerException at com.sun.jersey.spi.container.ContainerRequest.<init>(ContainerRequest.java:189) at com.sun.jersey.spi.container.servlet.WebComponent.createRequest(WebComponent.java:446) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:373) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:90) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.base/java.lang.Thread.run(Thread.java:834) 2019-06-18 15:39:45,858 WARN org.eclipse.jetty.servlet.ServletHandler: /webhdfs/v1/ java.lang.NullPointerException at com.sun.jersey.spi.container.ContainerRequest.<init>(ContainerRequest.java:189) at com.sun.jersey.spi.container.servlet.WebComponent.createRequest(WebComponent.java:446) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:373) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:90) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1609) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:539) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.base/java.lang.Thread.run(Thread.java:834) ``` 附datanode日志： 2019-06-18 14:52:36,785 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting DataNode STARTUP_MSG: host = gx-virtual-machine/127.0.1.1 STARTUP_MSG: args = [] STARTUP_MSG: version = 3.2.0 STARTUP_MSG: classpath = /usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/kerb-core-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-util-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/common/lib/avro-1.7.7.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/re2j-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-databind-2.9.5.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-annotations-2.9.5.jar:/usr/local/hadoop/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-client-2.12.0.jar:/usr/local/hadoop/share/hadoop/common/lib/json-smart-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/usr/local/hadoop/share/hadoop/common/lib/kerby-pkix-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-annotations-3.2.0.jar:/usr/local/hadoop/share/hadoop/common/lib/token-provider-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr311-api-1.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-core-1.19.jar:/usr/local/hadoop/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-lang3-3.7.jar:/usr/local/hadoop/share/hadoop/common/lib/dnsjava-2.1.7.jar:/usr/local/hadoop/share/hadoop/common/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/common/lib/jsp-api-2.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-beanutils-1.9.3.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/common/lib/javax.servlet-api-3.1.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-server-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/common/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-recipes-2.12.0.jar:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/usr/local/hadoop/share/hadoop/common/lib/kerb-client-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/gson-2.2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-security-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-net-3.6.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-text-1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/httpclient-4.5.2.jar:/usr/local/hadoop/share/hadoop/common/lib/kerby-config-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/kerb-server-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-servlet-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/kerb-util-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/asm-5.0.4.jar:/usr/local/hadoop/share/hadoop/common/lib/zookeeper-3.4.13.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-codec-1.11.jar:/usr/local/hadoop/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/accessors-smart-1.2.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-xml-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/usr/local/hadoop/share/hadoop/common/lib/kerby-util-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-http-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/common/lib/metrics-core-3.2.4.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-configuration2-2.1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/commons-io-2.5.jar:/usr/local/hadoop/share/hadoop/common/lib/curator-framework-2.12.0.jar:/usr/local/hadoop/share/hadoop/common/lib/kerb-common-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/netty-3.10.5.Final.jar:/usr/local/hadoop/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/snappy-java-1.0.5.jar:/usr/local/hadoop/share/hadoop/common/lib/httpcore-4.4.4.jar:/usr/local/hadoop/share/hadoop/common/lib/kerby-asn1-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-json-1.19.jar:/usr/local/hadoop/share/hadoop/common/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-server-1.19.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-core-2.9.5.jar:/usr/local/hadoop/share/hadoop/common/lib/hadoop-auth-3.2.0.jar:/usr/local/hadoop/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/usr/local/hadoop/share/hadoop/common/lib/jsch-0.1.54.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-webapp-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/common/lib/stax2-api-3.1.4.jar:/usr/local/hadoop/share/hadoop/common/lib/jersey-servlet-1.19.jar:/usr/local/hadoop/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/usr/local/hadoop/share/hadoop/common/lib/jetty-io-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-3.2.0-tests.jar:/usr/local/hadoop/share/hadoop/common/hadoop-kms-3.2.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-common-3.2.0.jar:/usr/local/hadoop/share/hadoop/common/hadoop-nfs-3.2.0.jar:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/avro-1.7.7.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerb-crypto-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/re2j-1.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/json-simple-1.1.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-databind-2.9.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-annotations-2.9.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/xz-1.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/curator-client-2.12.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/json-smart-2.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerby-pkix-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/hadoop-annotations-3.2.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/token-provider-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr311-api-1.1.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-lang3-3.7.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/dnsjava-2.1.7.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/paranamer-2.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-beanutils-1.9.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-compress-1.4.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-server-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/guava-11.0.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/curator-recipes-2.12.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/gson-2.2.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-security-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-math3-3.1.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-net-3.6.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-text-1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/httpclient-4.5.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-servlet-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/okio-1.6.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-util-ajax-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/asm-5.0.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/zookeeper-3.4.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-codec-1.11.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerb-identity-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-xml-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-all-4.0.52.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-http-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/commons-io-2.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/curator-framework-2.12.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/netty-3.10.5.Final.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/nimbus-jose-jwt-4.41.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/httpcore-4.4.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jettison-1.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-json-1.19.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-core-2.9.5.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/hadoop-auth-3.2.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jsch-0.1.54.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-webapp-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/kerb-simplekdc-1.0.1.jar:/usr/local/hadoop/share/hadoop/hdfs/lib/jetty-io-9.3.24.v20180605.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.2.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-3.2.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-nfs-3.2.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-3.2.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-native-client-3.2.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-rbf-3.2.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.2.0.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-client-3.2.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-3.2.0-tests.jar:/usr/local/hadoop/share/hadoop/hdfs/hadoop-hdfs-rbf-3.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/usr/local/hadoop/share/hadoop/mapreduce/lib/junit-4.11.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-nativetask-3.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-uploader-3.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.0.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.0-tests.jar:/usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn:/usr/local/hadoop/share/hadoop/yarn/lib/guice-servlet-4.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/json-io-2.5.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-client-1.19.jar:/usr/local/hadoop/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/usr/local/hadoop/share/hadoop/yarn/lib/java-util-1.9.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/usr/local/hadoop/share/hadoop/yarn/lib/javax.inject-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/fst-2.50.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jersey-guice-1.19.jar:/usr/local/hadoop/share/hadoop/yarn/lib/guice-4.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/usr/local/hadoop/share/hadoop/yarn/lib/metrics-core-3.2.4.jar:/usr/local/hadoop/share/hadoop/yarn/lib/objenesis-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-base-2.9.5.jar:/usr/local/hadoop/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/local/hadoop/share/hadoop/yarn/lib/aopalliance-1.0.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.9.5.jar:/usr/local/hadoop/share/hadoop/yarn/lib/jackson-jaxrs-json-provider-2.9.5.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-nodemanager-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-registry-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-submarine-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-client-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-api-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-services-api-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-common-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-services-core-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-tests-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-common-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-router-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.2.0.jar:/usr/local/hadoop/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.2.0.jar STARTUP_MSG: build = https://github.com/apache/hadoop.git -r e97acb3bd8f3befd27418996fa5d4b50bf2e17bf; compiled by 'sunilg' on 2019-01-08T06:08Z STARTUP_MSG: java = 11.0.3 ************************************************************/ 2019-06-18 14:52:36,863 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX signal handlers for [TERM, HUP, INT] 2019-06-18 14:52:41,503 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for [DISK]file:/usr/local/hadoop/tmp/dfs/data 2019-06-18 14:52:42,424 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: Loaded properties from hadoop-metrics2.properties 2019-06-18 14:52:44,123 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric snapshot period at 10 second(s). 2019-06-18 14:52:44,123 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics system started 2019-06-18 14:52:46,504 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling 2019-06-18 14:52:46,511 INFO org.apache.hadoop.hdfs.server.datanode.BlockScanner: Initialized block scanner with targetBytesPerSec 1048576 2019-06-18 14:52:46,566 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname is gx-virtual-machine 2019-06-18 14:52:46,567 INFO org.apache.hadoop.hdfs.server.common.Util: dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file IO profiling 2019-06-18 14:52:46,592 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting DataNode with maxLockedMemory = 0 2019-06-18 14:52:46,798 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming server at /0.0.0.0:9866 2019-06-18 14:52:46,866 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwidth is 10485760 bytes/s 2019-06-18 14:52:46,866 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for balancing is 50 2019-06-18 14:52:47,198 INFO org.eclipse.jetty.util.log: Logging initialized @15269ms 2019-06-18 14:52:48,022 INFO org.apache.hadoop.security.authentication.server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2019-06-18 14:52:48,062 INFO org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.datanode is not defined 2019-06-18 14:52:48,161 INFO org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter) 2019-06-18 14:52:48,174 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context datanode 2019-06-18 14:52:48,174 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context static 2019-06-18 14:52:48,174 INFO org.apache.hadoop.http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs 2019-06-18 14:52:48,556 INFO org.apache.hadoop.http.HttpServer2: Jetty bound to port 44121 2019-06-18 14:52:48,580 INFO org.eclipse.jetty.server.Server: jetty-9.3.24.v20180605, build timestamp: 2018-06-06T01:11:56+08:00, git hash: 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827 2019-06-18 14:52:49,011 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@7876d598{/logs,file:///usr/local/hadoop/logs/,AVAILABLE} 2019-06-18 14:52:49,018 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.s.ServletContextHandler@5af28b27{/static,file:///usr/local/hadoop/share/hadoop/hdfs/webapps/static/,AVAILABLE} 2019-06-18 14:52:50,151 INFO org.eclipse.jetty.server.handler.ContextHandler: Started o.e.j.w.WebAppContext@547e29a4{/,file:///usr/local/hadoop/share/hadoop/hdfs/webapps/datanode/,AVAILABLE}{/datanode} 2019-06-18 14:52:50,242 INFO org.eclipse.jetty.server.AbstractConnector: Started ServerConnector@6f45a1a0{HTTP/1.1,[http/1.1]}{localhost:44121} 2019-06-18 14:52:50,243 INFO org.eclipse.jetty.server.Server: Started @18329ms 2019-06-18 14:52:52,165 INFO org.apache.hadoop.hdfs.server.datanode.web.DatanodeHttpServer: Listening HTTP traffic on /0.0.0.0:9864 2019-06-18 14:52:52,273 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: dnUserName = hadoop 2019-06-18 14:52:52,273 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: supergroup = supergroup 2019-06-18 14:52:52,242 INFO org.apache.hadoop.util.JvmPauseMonitor: Starting JVM pause monitor 2019-06-18 14:52:52,720 INFO org.apache.hadoop.ipc.CallQueueManager: Using callQueue: class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false. 2019-06-18 14:52:52,880 INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 9867 2019-06-18 14:52:54,839 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Opened IPC server at /0.0.0.0:9867 2019-06-18 14:52:55,160 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Refresh request received for nameservices: null 2019-06-18 14:52:55,365 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Starting BPOfferServices for nameservices: <default> 2019-06-18 14:52:55,418 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000 starting to offer service 2019-06-18 14:52:55,532 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting 2019-06-18 14:52:55,561 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9867: starting 2019-06-18 14:52:58,314 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Acknowledging ACTIVE Namenode during handshakeBlock pool <registering> (Datanode Uuid unassigned) service to localhost/127.0.0.1:9000 2019-06-18 14:52:58,329 INFO org.apache.hadoop.hdfs.server.common.Storage: Using 1 threads to upgrade data directories (dfs.datanode.parallel.volumes.load.threads.num=1, dataDirs=1) 2019-06-18 14:52:58,458 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /usr/local/hadoop/tmp/dfs/data/in_use.lock acquired by nodename 55815@gx-virtual-machine 2019-06-18 14:52:58,478 INFO org.apache.hadoop.hdfs.server.common.Storage: Storage directory with location [DISK]file:/usr/local/hadoop/tmp/dfs/data is not formatted for namespace 317473294. Formatting... 2019-06-18 14:52:58,479 INFO org.apache.hadoop.hdfs.server.common.Storage: Generated new storageID DS-8b3e1e6d-135a-433a-93bb-3e62598daf5e for directory /usr/local/hadoop/tmp/dfs/data 2019-06-18 14:52:58,749 INFO org.apache.hadoop.hdfs.server.common.Storage: Analyzing storage directories for bpid BP-200946205-127.0.1.1-1560840480894 2019-06-18 14:52:58,750 INFO org.apache.hadoop.hdfs.server.common.Storage: Locking is disabled for /usr/local/hadoop/tmp/dfs/data/current/BP-200946205-127.0.1.1-1560840480894 2019-06-18 14:52:58,753 INFO org.apache.hadoop.hdfs.server.common.Storage: Block pool storage directory for location [DISK]file:/usr/local/hadoop/tmp/dfs/data and block pool id BP-200946205-127.0.1.1-1560840480894 is not formatted. Formatting ... 2019-06-18 14:52:58,753 INFO org.apache.hadoop.hdfs.server.common.Storage: Formatting block pool BP-200946205-127.0.1.1-1560840480894 directory /usr/local/hadoop/tmp/dfs/data/current/BP-200946205-127.0.1.1-1560840480894/current 2019-06-18 14:52:58,772 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Setting up storage: nsid=317473294;bpid=BP-200946205-127.0.1.1-1560840480894;lv=-57;nsInfo=lv=-65;cid=CID-eb45654d-0bc6-4348-b02f-e03603e1ae37;nsid=317473294;c=1560840480894;bpid=BP-200946205-127.0.1.1-1560840480894;dnuuid=null 2019-06-18 14:52:58,776 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Generated and persisted new Datanode UUID 6a2049c6-1a18-437a-97bd-51c5bb65a639 2019-06-18 14:52:59,549 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added new volume: DS-8b3e1e6d-135a-433a-93bb-3e62598daf5e 2019-06-18 14:52:59,553 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Added volume - [DISK]file:/usr/local/hadoop/tmp/dfs/data, StorageType: DISK 2019-06-18 14:52:59,615 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Registered FSDatasetState MBean 2019-06-18 14:52:59,680 INFO org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker: Scheduling a check for /usr/local/hadoop/tmp/dfs/data 2019-06-18 14:52:59,801 INFO org.apache.hadoop.hdfs.server.datanode.checker.DatasetVolumeChecker: Scheduled health check for volume /usr/local/hadoop/tmp/dfs/data 2019-06-18 14:52:59,809 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding block pool BP-200946205-127.0.1.1-1560840480894 2019-06-18 14:52:59,839 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Scanning block pool BP-200946205-127.0.1.1-1560840480894 on volume /usr/local/hadoop/tmp/dfs/data... 2019-06-18 14:53:00,166 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time taken to scan block pool BP-200946205-127.0.1.1-1560840480894 on /usr/local/hadoop/tmp/dfs/data: 327ms 2019-06-18 14:53:00,168 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to scan all replicas for block pool BP-200946205-127.0.1.1-1560840480894: 359ms 2019-06-18 14:53:00,181 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Adding replicas to map for block pool BP-200946205-127.0.1.1-1560840480894 on volume /usr/local/hadoop/tmp/dfs/data... 2019-06-18 14:53:00,181 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.BlockPoolSlice: Replica Cache file: /usr/local/hadoop/tmp/dfs/data/current/BP-200946205-127.0.1.1-1560840480894/current/replicas doesn't exist 2019-06-18 14:53:00,198 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Time to add replicas to map for block pool BP-200946205-127.0.1.1-1560840480894 on volume /usr/local/hadoop/tmp/dfs/data: 17ms 2019-06-18 14:53:00,198 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl: Total time to add all replicas to map for block pool BP-200946205-127.0.1.1-1560840480894: 27ms 2019-06-18 14:53:00,208 INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: Now scanning bpid BP-200946205-127.0.1.1-1560840480894 on volume /usr/local/hadoop/tmp/dfs/data 2019-06-18 14:53:00,221 INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/usr/local/hadoop/tmp/dfs/data, DS-8b3e1e6d-135a-433a-93bb-3e62598daf5e): finished scanning block pool BP-200946205-127.0.1.1-1560840480894 2019-06-18 14:53:00,401 INFO org.apache.hadoop.hdfs.server.datanode.VolumeScanner: VolumeScanner(/usr/local/hadoop/tmp/dfs/data, DS-8b3e1e6d-135a-433a-93bb-3e62598daf5e): no suitable block pools found to scan. Waiting 1814399799 ms. 2019-06-18 14:53:00,418 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 2019/6/18 下午8:05 with interval of 21600000ms 2019-06-18 14:53:00,463 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool BP-200946205-127.0.1.1-1560840480894 (Datanode Uuid 6a2049c6-1a18-437a-97bd-51c5bb65a639) service to localhost/127.0.0.1:9000 beginning handshake with NN 2019-06-18 14:53:00,825 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Block pool Block pool BP-200946205-127.0.1.1-1560840480894 (Datanode Uuid 6a2049c6-1a18-437a-97bd-51c5bb65a639) service to localhost/127.0.0.1:9000 successfully registered with NN 2019-06-18 14:53:00,825 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: For namenode localhost/127.0.0.1:9000 using BLOCKREPORT_INTERVAL of 21600000msec CACHEREPORT_INTERVAL of 10000msec Initial delay: 0msec; heartBeatInterval=3000 2019-06-18 14:53:01,524 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Successfully sent block report 0xb210af820fa10abf, containing 1 storage report(s), of which we sent 1. The reports had 0 total blocks and used 1 RPC(s). This took 19 msec to generate and 231 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5. 2019-06-18 14:53:01,525 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Got finalize command for block pool BP-200946205-127.0.1.1-1560840480894 2019-06-18 15:44:37,567 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741825_1001 src: /127.0.0.1:34774 dest: /127.0.0.1:9866 2019-06-18 15:44:37,733 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:34774, dest: /127.0.0.1:9866, bytes: 8260, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1864191814_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741825_1001, duration(ns): 75831098 2019-06-18 15:44:37,737 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741825_1001, type=LAST_IN_PIPELINE terminating 2019-06-18 15:44:38,256 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741826_1002 src: /127.0.0.1:34776 dest: /127.0.0.1:9866 2019-06-18 15:44:38,266 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:34776, dest: /127.0.0.1:9866, bytes: 953, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1864191814_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741826_1002, duration(ns): 5252820 2019-06-18 15:44:38,266 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741826_1002, type=LAST_IN_PIPELINE terminating 2019-06-18 15:44:38,340 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741827_1003 src: /127.0.0.1:34778 dest: /127.0.0.1:9866 2019-06-18 15:44:38,365 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:34778, dest: /127.0.0.1:9866, bytes: 11392, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1864191814_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741827_1003, duration(ns): 19816531 2019-06-18 15:44:38,372 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741827_1003, type=LAST_IN_PIPELINE terminating 2019-06-18 15:44:38,428 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741828_1004 src: /127.0.0.1:34780 dest: /127.0.0.1:9866 2019-06-18 15:44:38,455 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:34780, dest: /127.0.0.1:9866, bytes: 1061, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1864191814_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741828_1004, duration(ns): 9820674 2019-06-18 15:44:38,464 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741828_1004, type=LAST_IN_PIPELINE terminating 2019-06-18 15:44:38,517 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741829_1005 src: /127.0.0.1:34782 dest: /127.0.0.1:9866 2019-06-18 15:44:38,537 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:34782, dest: /127.0.0.1:9866, bytes: 620, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1864191814_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741829_1005, duration(ns): 9424051 2019-06-18 15:44:38,537 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741829_1005, type=LAST_IN_PIPELINE terminating 2019-06-18 15:44:38,569 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741830_1006 src: /127.0.0.1:34784 dest: /127.0.0.1:9866 2019-06-18 15:44:38,579 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:34784, dest: /127.0.0.1:9866, bytes: 3518, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1864191814_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741830_1006, duration(ns): 6662498 2019-06-18 15:44:38,579 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741830_1006, type=LAST_IN_PIPELINE terminating 2019-06-18 15:44:38,642 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741831_1007 src: /127.0.0.1:34786 dest: /127.0.0.1:9866 2019-06-18 15:44:38,650 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:34786, dest: /127.0.0.1:9866, bytes: 682, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1864191814_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741831_1007, duration(ns): 5047916 2019-06-18 15:44:38,650 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741831_1007, type=LAST_IN_PIPELINE terminating 2019-06-18 15:44:38,713 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741832_1008 src: /127.0.0.1:34788 dest: /127.0.0.1:9866 2019-06-18 15:44:38,726 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:34788, dest: /127.0.0.1:9866, bytes: 758, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1864191814_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741832_1008, duration(ns): 8532382 2019-06-18 15:44:38,727 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741832_1008, type=LAST_IN_PIPELINE terminating 2019-06-18 15:44:38,789 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741833_1009 src: /127.0.0.1:34790 dest: /127.0.0.1:9866 2019-06-18 15:44:38,807 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:34790, dest: /127.0.0.1:9866, bytes: 690, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_1864191814_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741833_1009, duration(ns): 5589094 2019-06-18 15:44:38,813 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741833_1009, type=LAST_IN_PIPELINE terminating 2019-06-19 09:54:01,961 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741834_1010 src: /127.0.0.1:36578 dest: /127.0.0.1:9866 2019-06-19 09:54:02,003 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:36578, dest: /127.0.0.1:9866, bytes: 8260, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_853656761_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741834_1010, duration(ns): 32739756 2019-06-19 09:54:02,011 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741834_1010, type=LAST_IN_PIPELINE terminating 2019-06-19 09:54:02,125 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741835_1011 src: /127.0.0.1:36580 dest: /127.0.0.1:9866 2019-06-19 09:54:02,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:36580, dest: /127.0.0.1:9866, bytes: 953, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_853656761_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741835_1011, duration(ns): 12137675 2019-06-19 09:54:02,154 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741835_1011, type=LAST_IN_PIPELINE terminating 2019-06-19 09:54:02,235 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741836_1012 src: /127.0.0.1:36582 dest: /127.0.0.1:9866 2019-06-19 09:54:02,249 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:36582, dest: /127.0.0.1:9866, bytes: 11392, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_853656761_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741836_1012, duration(ns): 8740891 2019-06-19 09:54:02,249 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741836_1012, type=LAST_IN_PIPELINE terminating 2019-06-19 09:54:02,307 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741837_1013 src: /127.0.0.1:36584 dest: /127.0.0.1:9866 2019-06-19 09:54:02,322 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:36584, dest: /127.0.0.1:9866, bytes: 1061, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_853656761_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741837_1013, duration(ns): 8680367 2019-06-19 09:54:02,323 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741837_1013, type=LAST_IN_PIPELINE terminating 2019-06-19 09:54:02,399 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741838_1014 src: /127.0.0.1:36586 dest: /127.0.0.1:9866 2019-06-19 09:54:02,413 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:36586, dest: /127.0.0.1:9866, bytes: 620, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_853656761_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741838_1014, duration(ns): 8474258 2019-06-19 09:54:02,413 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741838_1014, type=LAST_IN_PIPELINE terminating 2019-06-19 09:54:02,491 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741839_1015 src: /127.0.0.1:36588 dest: /127.0.0.1:9866 2019-06-19 09:54:02,502 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:36588, dest: /127.0.0.1:9866, bytes: 3518, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_853656761_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741839_1015, duration(ns): 6946259 2019-06-19 09:54:02,503 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741839_1015, type=LAST_IN_PIPELINE terminating 2019-06-19 09:54:02,560 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741840_1016 src: /127.0.0.1:36590 dest: /127.0.0.1:9866 2019-06-19 09:54:02,571 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:36590, dest: /127.0.0.1:9866, bytes: 682, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_853656761_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741840_1016, duration(ns): 6602106 2019-06-19 09:54:02,571 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-200946205-127.0.1.1-1560840480894:blk_1073741840_1016, type=LAST_IN_PIPELINE terminating 2019-06-19 09:54:02,635 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-200946205-127.0.1.1-1560840480894:blk_1073741841_1017 src: /127.0.0.1:36592 dest: /127.0.0.1:9866 2019-06-19 09:54:02,650 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /127.0.0.1:36592, dest: /127.0.0.1:9866, bytes: 758, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_853656761_1, offset: 0, srvID: 6a2049c6-1a18-437a-97bd-51c5bb65a639, blockid: BP-200946205-127.0.1.1-1560840480894:blk_1073741841_1017, duration(ns): 9690339 2019-06-19 09:54:02,654 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder:

上传文件到HDFS，使用hadoop命令，一直报Retrying connect to server: (1)环境：我hadoop环境已经搭建好了，版本是2.6，hdfs的50070，MR的8088端口页面都可以显示，然后我想往hdfs上传文件。 (2)命令：hadoop fs -mkdir /root/1或./hdfs dfs -mkdir /root/1会一直报下面这个问题 (3)问题：其中42.123.125.237这个ip我也不知道是拿来的，myhadoop是我取的服务名，（还有8020端口是nameNode的吧） ![图片说明](https://img-ask.csdn.net/upload/201708/11/1502452427_763174.png) (4)IP说明： ![图片说明](https://img-ask.csdn.net/upload/201708/11/1502452446_130416.png) (5)配置： core-site.xml： ![图片说明](https://img-ask.csdn.net/upload/201708/11/1502452470_185793.png) hdfs-site.xml： ![图片说明](https://img-ask.csdn.net/upload/201708/11/1502452492_693348.png) ![图片说明](https://img-ask.csdn.net/upload/201708/11/1502449699_137831.png) hosts文件： ![图片说明](https://img-ask.csdn.net/upload/201708/11/1502449756_2107.png) profile: ![图片说明](https://img-ask.csdn.net/upload/201708/11/1502449801_929152.png)

Hadoop 同时启动HDFS、YARN出错: ``` liuye@liuye-VirtualBox:~$ start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh /usr/local/hadoop/bin/hdfs: 行 276: /usr/lib/jvm/java-7-openjdk-amd64/bin/java: 没有那个文件或目录 Starting namenodes on [] liuye@localhost's password: localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-liuye-namenode-liuye-VirtualBox.out localhost: /usr/local/hadoop/bin/hdfs: 行 276: /usr/lib/jvm/java-7-openjdk-amd64/bin/java: 没有那个文件或目录 liuye@localhost's password: localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-liuye-datanode-liuye-VirtualBox.out localhost: /usr/local/hadoop/bin/hdfs: 行 276: /usr/lib/jvm/java-7-openjdk-amd64/bin/java: 没有那个文件或目录 /usr/local/hadoop/bin/hdfs: 行 276: /usr/lib/jvm/java-7-openjdk-amd64/bin/java: 没有那个文件或目录 starting yarn daemons starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-liuye-resourcemanager-liuye-VirtualBox.out /usr/local/hadoop/bin/yarn: 行 284: /usr/lib/jvm/java-7-openjdk-amd64/bin/java: 没有那个文件或目录 liuye@localhost's password: localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-liuye-nodemanager-liuye-VirtualBox.out localhost: /usr/local/hadoop/bin/yarn: 行 284: /usr/lib/jvm/java-7-openjdk-amd64/bin/java: 没有那个文件或目录 liuye@liuye-VirtualBox:~$ ```

spark shell在存运算结果到hdfs时报java.io.IOException: Not a file: hdfs://mini1:9000/spark/res: scala> sc.textFile("hdfs://mini1:9000/spark").flatMap(_.split(" ")).map((_,1)).reduceByKey(_+_).saveAsTextFile("hdfs://mini1:9000/spark/res2") 执行上面的代码出错，这个目录在hdfs下是有的，而且就算没有也会创建。还有就是我运行的代码中是保存到res2目录，这里为什么报没有res目录 18/11/05 19:06:44 WARN SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes java.io.IOException: Not a file: hdfs://mini1:9000/spark/res at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:320) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:237) at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:65) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$reduceByKey$3.apply(PairRDDFunctions.scala:331) at org.apache.spark.rdd.PairRDDFunctions$$anonfun$reduceByKey$3.apply(PairRDDFunctions.scala:331) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111) at org.apache.spark.rdd.RDD.withScope(RDD.scala:316) at org.apache.spark.rdd.PairRDDFunctions.reduceByKey(PairRDDFunctions.scala:330) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:28) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:33) at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:35) at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:37) at $iwC$$iwC$$iwC$$iwC.<init>(<console>:39) at $iwC$$iwC$$iwC.<init>(<console>:41) at $iwC$$iwC.<init>(<console>:43) at $iwC.<init>(<console>:45) at <init>(<console>:47) at .<init>(<console>:51) at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657) at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665) at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

mapreduce编程出现错误：Job running in uber mode : false？: eclipse编写mapreduce程序，实现统计多个目录下所有文件的文件行数总数。代码如下： package ZhangBo; import java.io.IOException; //import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; //import org.apache.hadoop.mapreduce.lib.input.MultipleInputs; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; //import org.apache.hadoop.io.LongWritable; //import org.apache.hadoop.util.GenericOptionsParser; //import org.junit.Test; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; //import org.apache.log4j.MDC; //import org.apache.log4j.Logger; //import firstnight.testlog4jj; public class WordCount { //////log static Log logger = LogFactory.getLog(WordCount.class); public static class TokenizerMapper extends Mapper<LongWritable, Text, Text, IntWritable>{ //一行的起始位置 private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(LongWritable key, Text value, Context context ) throws IOException, InterruptedException { //进行分割 logger.debug("This is debug message."); logger.error("This is error message."); logger.info("ni hao "); context.write(word,one); } } //对输入的key value值进行处理转换成新的 key value值进行一系列框架自带的处理排序合并分句等等 public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values, Context context ) throws IOException, InterruptedException { int sum = 0; //循环遍历 IntWritable for (IntWritable val : values) { sum += val.get();//get 转 int } result.set(sum); context.write(key, result); } } private static FileSystem hdfs; private static String args1; private static String args2; private static String args3; Configuration conf1 = new Configuration(); private static void showDir(FileStatus fs) throws Exception { Path path = fs.getPath(); if (fs.isDir()) { args2 = path.toString(); if (args2 != args3) args1 = args1 +","+args2; System.out.println("目录：" + path); FileStatus[] f = hdfs.listStatus(path); if (f.length > 0) { for (FileStatus file:f) { showDir(file); args3 = file.getPath().toString(); } } } else { System.out.println("文件：" + path); } System.out.println("目录之和： "+args1); } public void test() throws Exception { hdfs = FileSystem.get(conf1); FileStatus[] fs = hdfs.listStatus(new Path("hdfs://C15/zbo")); args1 = "hdfs://C15/zbo"; if (fs.length > 0) { for (FileStatus f : fs) { showDir(f); } } else { System.out.println("没有什么好遍历的...."); } } public static void main(String[] args) throws Exception { new WordCount().test(); Configuration conf = new Configuration(); ////////////////////// Job job = new Job(conf,"word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); FileInputFormat.addInputPaths(job,args1); FileOutputFormat.setOutputPath(job, new Path("outbo")); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); System.exit(job.waitForCompletion(true)?0:1); } } 在目录hdfs://C15/zbo下创建目录和文件在eclipse上打jar包ho.jar 执行程序 :hadoop jar ZhangBo.WordCount 执行结果如下所示： [root@c15m1 /]# hadoop jar ho.jar ZhangBo.WordCount 文件：hdfs://C15/zbo/file1.txt 目录之和： hdfs://C15/zbo 文件：hdfs://C15/zbo/file2.txt 目录之和： hdfs://C15/zbo 目录：hdfs://C15/zbo/zbo 文件：hdfs://C15/zbo/zbo/file1.txt 目录之和： hdfs://C15/zbo,hdfs://C15/zbo/zbo 文件：hdfs://C15/zbo/zbo/file2.txt 目录之和： hdfs://C15/zbo,hdfs://C15/zbo/zbo 目录：hdfs://C15/zbo/zbo/zbo 文件：hdfs://C15/zbo/zbo/zbo/file1.txt 目录之和： hdfs://C15/zbo,hdfs://C15/zbo/zbo,hdfs://C15/zbo/zbo/zbo 文件：hdfs://C15/zbo/zbo/zbo/file2.txt 目录之和： hdfs://C15/zbo,hdfs://C15/zbo/zbo,hdfs://C15/zbo/zbo/zbo 目录之和： hdfs://C15/zbo,hdfs://C15/zbo/zbo,hdfs://C15/zbo/zbo/zbo 目录之和： hdfs://C15/zbo,hdfs://C15/zbo/zbo,hdfs://C15/zbo/zbo/zbo 15/11/17 08:55:37 INFO client.RMProxy: Connecting to ResourceManager at c15m1.ecld.com/132.121.94.213:8050 15/11/17 08:55:38 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 72820725 for hdfs on ha-hdfs:C15 15/11/17 08:55:38 INFO security.TokenCache: Got dt for hdfs://C15; Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:C15, Ident: (HDFS_DELEGATION_TOKEN token 72820725 for hdfs) 15/11/17 08:55:38 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 15/11/17 08:55:38 INFO input.FileInputFormat: Total input paths to process : 8 15/11/17 08:55:38 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 15/11/17 08:55:38 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev dbd51f0fb61f5347228a7a23fe0765ac1242fcdf] 15/11/17 08:55:47 INFO mapreduce.JobSubmitter: number of splits:8 15/11/17 08:55:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1440641224751_187144 15/11/17 08:55:48 INFO mapreduce.JobSubmitter: Kind: HDFS_DELEGATION_TOKEN, Service: ha-hdfs:C15, Ident: (HDFS_DELEGATION_TOKEN token 72820725 for hdfs) 15/11/17 08:55:48 INFO impl.YarnClientImpl: Submitted application application_1440641224751_187144 15/11/17 08:55:49 INFO mapreduce.Job: The url to track the job: http://c15m1.ecld.com:8088/proxy/application_1440641224751_187144/ 15/11/17 08:55:49 INFO mapreduce.Job: Running job: job_1440641224751_187144 15/11/17 08:55:51 INFO mapreduce.Job: Job job_1440641224751_187144 running in uber mode : false 15/11/17 08:55:51 INFO mapreduce.Job: map 0% reduce 0% 15/11/17 08:55:51 INFO mapreduce.Job: Job job_1440641224751_187144 failed with state FAILED due to: Application application_1440641224751_187144 failed 2 times due to AM Container for appattempt_1440641224751_187144_000002 exited with exitCode: -1000 due to: Application application_1440641224751_187144 initialization failed (exitCode=255) with output: Requested user hdfs is banned .Failing this attempt.. Failing the application. 15/11/17 08:55:51 INFO mapreduce.Job: Counters: 0 问题：running in uber mode : false是什么问题？

Azkaban和Hadoop2.5.1集成出现的问题: Using Hadoop from /usr/local/hadoop-suite/hadoop Using Hive from /usr/local/hadoop-suite/hive bin/.. /usr/local/jdk/lib/tools.jar:/usr/local/jdk/lib/dt.jar:bin/../lib/azkaban-common-2.6.4.jar:bin/../lib/azkaban-webserver-2.6.4.jar:bin/../lib/commons-codec-1.9.jar:bin/../lib/commons-collections-3.2.1.jar:bin/../lib/commons-configuration-1.8.jar:bin/../lib/commons-dbcp-1.4.jar:bin/../lib/commons-dbutils-1.5.jar:bin/../lib/commons-email-1.2.jar:bin/../lib/commons-fileupload-1.2.1.jar:bin/../lib/commons-io-2.4.jar:bin/../lib/commons-jexl-2.1.1.jar:bin/../lib/commons-lang-2.6.jar:bin/../lib/commons-logging-1.1.1.jar:bin/../lib/commons-pool-1.6.jar:bin/../lib/data-1.15.7.jar:bin/../lib/gradle-plugins-1.15.7.jar:bin/../lib/guava-13.0.1.jar:bin/../lib/h2-1.3.170.jar:bin/../lib/httpclient-4.2.1.jar:bin/../lib/httpcore-4.2.1.jar:bin/../lib/jackson-core-2.3.2.jar:bin/../lib/jackson-core-asl-1.9.5.jar:bin/../lib/jackson-mapper-asl-1.9.5.jar:bin/../lib/jetty-6.1.26.jar:bin/../lib/jetty-util-6.1.26.jar:bin/../lib/joda-time-2.0.jar:bin/../lib/jopt-simple-4.3.jar:bin/../lib/li-jersey-uri-1.15.7.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/mail-1.4.5.jar:bin/../lib/mysql-connector-java-5.1.28.jar:bin/../lib/parseq-1.3.7.jar:bin/../lib/pegasus-common-1.15.7.jar:bin/../lib/r2-1.15.7.jar:bin/../lib/restli-common-1.15.7.jar:bin/../lib/restli-server-1.15.7.jar:bin/../lib/servlet-api-2.5.jar:bin/../lib/slf4j-api-1.6.1.jar:bin/../lib/velocity-1.7.jar:bin/../lib/velocity-tools-2.0.jar:bin/../extlib/azkaban-common-2.6.4.jar:bin/../extlib/azkaban-execserver-2.6.4.jar:bin/../extlib/azkaban-webserver-2.6.4.jar:bin/../extlib/commons-cli-1.2.jar:bin/../extlib/hadoop-auth-2.5.1.jar:bin/../extlib/hadoop-common-2.5.1.jar:bin/../extlib/hadoop-hdfs-2.5.1.jar:bin/../extlib/hive-cli-0.13.1.jar:bin/../extlib/hive-common-0.13.1.jar:bin/../extlib/hive-exec-0.13.1.jar:bin/../extlib/jackson-core-asl-1.9.5.jar:bin/../extlib/jackson-mapper-asl-1.9.5.jar:bin/../extlib/log4j-1.2.16.jar:bin/../extlib/protobuf-java-2.5.0.jar:bin/../extlib/servlet-api-2.5.jar:bin/../extlib/slf4j-api-1.6.1.jar:bin/../extlib/slf4j-log4j12-1.6.4.jar:bin/../extlib/velocity-1.7.jar:bin/../extlib/velocity-tools-2.0.jar:bin/../plugins/*/*.jar:/usr/local/hadoop-suite/hadoop/conf:/usr/local/hadoop-suite/hadoop/*:/usr/local/hadoop-suite/hive/conf:/usr/local/hadoop-suite/hive/lib/* 2015/01/21 16:02:33.518 +0800 ERROR [AzkabanWebServer] [Azkaban] Starting Jetty Azkaban Executor... 2015/01/21 16:02:33.937 +0800 ERROR [AzkabanWebServer] [Azkaban] Plugin class azkaban.viewer.hdfs.HdfsBrowserServlet 2015/01/21 16:02:33.941 +0800 INFO [AzkabanWebServer] [Azkaban] Source jar /usr/local/hadoop-suite/azkaban-web-2.6.4-old/plugins/viewer/hdfs/lib/azkaban-hdfs-viewer-2.6.4.jar 2015/01/21 16:02:33.945 +0800 ERROR [AzkabanWebServer] [Azkaban] Plugin class azkaban.viewer.javaviewer.JavaViewerServlet 2015/01/21 16:02:33.946 +0800 INFO [AzkabanWebServer] [Azkaban] Source jar /usr/local/hadoop-suite/azkaban-web-2.6.4-old/plugins/viewer/javaviewer/lib/azkaban-javaviewer-2.6.3.jar 2015/01/21 16:02:33.947 +0800 ERROR [AzkabanWebServer] [Azkaban] Plugin class azkaban.viewer.reportal.ReportalServlet 2015/01/21 16:02:33.947 +0800 ERROR [AzkabanWebServer] [Azkaban] External library path /usr/local/hadoop-suite/azkaban-web-2.6.4-old/plugins/viewer/reportal/extlib not found. 2015/01/21 16:02:33.950 +0800 INFO [AzkabanWebServer] [Azkaban] Source jar /usr/local/hadoop-suite/azkaban-web-2.6.4-old/plugins/viewer/reportal/lib/azkaban-reportal-$%7Bgit.tag%7D.jar Reportal web resources: /usr/local/hadoop-suite/azkaban-web-2.6.4-old/plugins/viewer/reportal/web 2015/01/21 16:02:33.953 +0800 ERROR [AzkabanWebServer] [Azkaban] Plugin class azkaban.viewer.jobsummary.JobSummaryServlet 2015/01/21 16:02:33.953 +0800 ERROR [AzkabanWebServer] [Azkaban] External library path /usr/local/hadoop-suite/azkaban-web-2.6.4-old/plugins/viewer/jobsummary/extlib/* not found.

DataNode未报错直接关闭: 启动得好好的，过一段时间就掉了。而且没有任何异常信息，这是为什么呢？这是后面的一些日志，完全没有抛出异常。 ``` 2019-11-02 16:13:13,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742045_1221 to 172.31.19.252:50010 2019-11-02 16:13:14,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742045_1221 (numBytes=109043) to /172.31.19.252:50010 2019-11-02 16:13:14,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742042_1218 (numBytes=197986) to /172.31.19.252:50010 2019-11-02 16:13:16,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742051_1227 src: /172.31.19.252:46170 dest: /172.31.23.3:50010 2019-11-02 16:13:16,981 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742050_1226 src: /172.31.19.252:46171 dest: /172.31.23.3:50010 2019-11-02 16:13:16,981 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742051_1227 src: /172.31.19.252:46170 dest: /172.31.23.3:50010 of size 58160 2019-11-02 16:13:16,985 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742050_1226 src: /172.31.19.252:46171 dest: /172.31.23.3:50010 of size 2178774 2019-11-02 16:13:16,998 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742047_1223 to 172.31.19.252:50010 2019-11-02 16:13:16,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742048_1224 to 172.31.19.252:50010 2019-11-02 16:13:17,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742048_1224 (numBytes=34604) to /172.31.19.252:50010 2019-11-02 16:13:17,001 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742047_1223 (numBytes=780664) to /172.31.19.252:50010 2019-11-02 16:13:19,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742049_1225 to 172.31.19.252:50010 2019-11-02 16:13:19,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742052_1228 to 172.31.19.252:50010 2019-11-02 16:13:20,003 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742052_1228 (numBytes=6052) to /172.31.19.252:50010 2019-11-02 16:13:20,003 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742049_1225 (numBytes=592319) to /172.31.19.252:50010 2019-11-02 16:13:44,183 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742056_1232 src: /172.31.20.57:51732 dest: /172.31.23.3:50010 2019-11-02 16:13:44,193 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51732, dest: /172.31.23.3:50010, bytes: 1108073, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742056_1232, duration(ns): 9331035 2019-11-02 16:13:44,193 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742056_1232, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:44,223 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742058_1234 src: /172.31.20.57:51736 dest: /172.31.23.3:50010 2019-11-02 16:13:44,225 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51736, dest: /172.31.23.3:50010, bytes: 20744, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742058_1234, duration(ns): 822959 2019-11-02 16:13:44,225 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742058_1234, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:44,240 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742059_1235 src: /172.31.20.57:51738 dest: /172.31.23.3:50010 2019-11-02 16:13:44,241 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51738, dest: /172.31.23.3:50010, bytes: 53464, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742059_1235, duration(ns): 834208 2019-11-02 16:13:44,241 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742059_1235, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:44,250 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742060_1236 src: /172.31.20.57:51740 dest: /172.31.23.3:50010 2019-11-02 16:13:44,252 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51740, dest: /172.31.23.3:50010, bytes: 60686, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742060_1236, duration(ns): 836219 2019-11-02 16:13:44,252 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742060_1236, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:45,139 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742064_1240 src: /172.31.20.57:51748 dest: /172.31.23.3:50010 2019-11-02 16:13:45,147 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51748, dest: /172.31.23.3:50010, bytes: 914311, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742064_1240, duration(ns): 7451340 2019-11-02 16:13:45,147 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742064_1240, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:45,179 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742066_1242 src: /172.31.20.57:51752 dest: /172.31.23.3:50010 2019-11-02 16:13:45,182 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51752, dest: /172.31.23.3:50010, bytes: 706710, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742066_1242, duration(ns): 2666689 2019-11-02 16:13:45,182 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742066_1242, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:45,192 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742067_1243 src: /172.31.20.57:51754 dest: /172.31.23.3:50010 2019-11-02 16:13:45,194 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51754, dest: /172.31.23.3:50010, bytes: 186260, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742067_1243, duration(ns): 1335836 2019-11-02 16:13:45,194 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742067_1243, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:45,617 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742068_1244 src: /172.31.20.57:51756 dest: /172.31.23.3:50010 2019-11-02 16:13:45,627 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51756, dest: /172.31.23.3:50010, bytes: 1768012, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742068_1244, duration(ns): 8602898 2019-11-02 16:13:45,627 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742068_1244, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:46,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742057_1233 src: /172.31.19.252:46174 dest: /172.31.23.3:50010 2019-11-02 16:13:46,981 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742057_1233 src: /172.31.19.252:46174 dest: /172.31.23.3:50010 of size 205389 2019-11-02 16:13:46,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742056_1232 to 172.31.19.252:50010 2019-11-02 16:13:46,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742058_1234 to 172.31.19.252:50010 2019-11-02 16:13:47,001 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742058_1234 (numBytes=20744) to /172.31.19.252:50010 2019-11-02 16:13:47,006 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742056_1232 (numBytes=1108073) to /172.31.19.252:50010 2019-11-02 16:13:47,315 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742073_1249 src: /172.31.20.57:51766 dest: /172.31.23.3:50010 2019-11-02 16:13:47,320 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51766, dest: /172.31.23.3:50010, bytes: 990927, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742073_1249, duration(ns): 3408777 2019-11-02 16:13:47,320 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742073_1249, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:47,329 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742074_1250 src: /172.31.20.57:51768 dest: /172.31.23.3:50010 2019-11-02 16:13:47,331 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51768, dest: /172.31.23.3:50010, bytes: 36519, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742074_1250, duration(ns): 1284246 2019-11-02 16:13:47,331 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742074_1250, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:47,789 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742078_1254 src: /172.31.20.57:51776 dest: /172.31.23.3:50010 2019-11-02 16:13:47,794 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51776, dest: /172.31.23.3:50010, bytes: 279012, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742078_1254, duration(ns): 2573122 2019-11-02 16:13:47,794 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742078_1254, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:47,808 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742079_1255 src: /172.31.20.57:51778 dest: /172.31.23.3:50010 2019-11-02 16:13:47,812 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51778, dest: /172.31.23.3:50010, bytes: 1344870, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742079_1255, duration(ns): 3770082 2019-11-02 16:13:47,812 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742079_1255, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:48,225 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742080_1256 src: /172.31.20.57:51780 dest: /172.31.23.3:50010 2019-11-02 16:13:48,228 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51780, dest: /172.31.23.3:50010, bytes: 990927, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742080_1256, duration(ns): 2365213 2019-11-02 16:13:48,228 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742080_1256, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:48,638 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742081_1257 src: /172.31.20.57:51782 dest: /172.31.23.3:50010 2019-11-02 16:13:48,640 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51782, dest: /172.31.23.3:50010, bytes: 99555, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742081_1257, duration(ns): 1140563 2019-11-02 16:13:48,640 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742081_1257, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:49,062 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742083_1259 src: /172.31.20.57:51786 dest: /172.31.23.3:50010 2019-11-02 16:13:49,064 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51786, dest: /172.31.23.3:50010, bytes: 20998, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742083_1259, duration(ns): 823110 2019-11-02 16:13:49,064 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742083_1259, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:49,500 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742086_1262 src: /172.31.20.57:51792 dest: /172.31.23.3:50010 2019-11-02 16:13:49,502 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51792, dest: /172.31.23.3:50010, bytes: 224277, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742086_1262, duration(ns): 1129868 2019-11-02 16:13:49,502 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742086_1262, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:49,511 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742087_1263 src: /172.31.20.57:51794 dest: /172.31.23.3:50010 2019-11-02 16:13:49,514 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.20.57:51794, dest: /172.31.23.3:50010, bytes: 780664, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742087_1263, duration(ns): 2377601 2019-11-02 16:13:49,514 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742087_1263, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:49,980 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742061_1237 src: /172.31.19.252:46176 dest: /172.31.23.3:50010 2019-11-02 16:13:49,981 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742062_1238 src: /172.31.19.252:46177 dest: /172.31.23.3:50010 2019-11-02 16:13:49,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742062_1238 src: /172.31.19.252:46177 dest: /172.31.23.3:50010 of size 232248 2019-11-02 16:13:49,983 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742061_1237 src: /172.31.19.252:46176 dest: /172.31.23.3:50010 of size 434678 2019-11-02 16:13:49,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742059_1235 to 172.31.19.252:50010 2019-11-02 16:13:49,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742060_1236 to 172.31.19.252:50010 2019-11-02 16:13:49,999 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742073_1249 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742073 for deletion 2019-11-02 16:13:50,000 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742059_1235 (numBytes=53464) to /172.31.19.252:50010 2019-11-02 16:13:50,000 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742073_1249 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742073 2019-11-02 16:13:50,001 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742060_1236 (numBytes=60686) to /172.31.19.252:50010 2019-11-02 16:13:51,310 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742093_1269 src: /172.31.19.252:46180 dest: /172.31.23.3:50010 2019-11-02 16:13:51,313 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.19.252:46180, dest: /172.31.23.3:50010, bytes: 94, op: HDFS_WRITE, cliID: DFSClient_NONMAPREDUCE_803773611_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742093_1269, duration(ns): 2826729 2019-11-02 16:13:51,313 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742093_1269, type=LAST_IN_PIPELINE terminating 2019-11-02 16:13:52,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742063_1239 src: /172.31.19.252:46190 dest: /172.31.23.3:50010 2019-11-02 16:13:52,983 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742065_1241 src: /172.31.19.252:46192 dest: /172.31.23.3:50010 2019-11-02 16:13:52,986 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742063_1239 src: /172.31.19.252:46190 dest: /172.31.23.3:50010 of size 1033299 2019-11-02 16:13:52,987 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742065_1241 src: /172.31.19.252:46192 dest: /172.31.23.3:50010 of size 892808 2019-11-02 16:13:52,998 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742064_1240 to 172.31.19.252:50010 2019-11-02 16:13:52,998 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742066_1242 to 172.31.19.252:50010 2019-11-02 16:13:53,002 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742064_1240 (numBytes=914311) to /172.31.19.252:50010 2019-11-02 16:13:53,005 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742066_1242 (numBytes=706710) to /172.31.19.252:50010 2019-11-02 16:13:55,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742067_1243 to 172.31.19.252:50010 2019-11-02 16:13:55,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742068_1244 to 172.31.19.252:50010 2019-11-02 16:13:56,002 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742067_1243 (numBytes=186260) to /172.31.19.252:50010 2019-11-02 16:13:56,025 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742069_1245 src: /172.31.19.252:46198 dest: /172.31.23.3:50010 2019-11-02 16:13:56,026 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742070_1246 src: /172.31.19.252:46200 dest: /172.31.23.3:50010 2019-11-02 16:13:56,027 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742070_1246 src: /172.31.19.252:46200 dest: /172.31.23.3:50010 of size 36455 2019-11-02 16:13:56,040 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742069_1245 src: /172.31.19.252:46198 dest: /172.31.23.3:50010 of size 1801469 2019-11-02 16:13:56,068 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742068_1244 (numBytes=1768012) to /172.31.19.252:50010 2019-11-02 16:13:58,995 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742071_1247 src: /172.31.19.252:46206 dest: /172.31.23.3:50010 2019-11-02 16:13:58,996 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742072_1248 src: /172.31.19.252:46208 dest: /172.31.23.3:50010 2019-11-02 16:13:58,996 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742072_1248 src: /172.31.19.252:46208 dest: /172.31.23.3:50010 of size 19827 2019-11-02 16:13:58,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742074_1250 to 172.31.19.252:50010 2019-11-02 16:13:59,003 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742074_1250 (numBytes=36519) to /172.31.19.252:50010 2019-11-02 16:13:59,006 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742071_1247 src: /172.31.19.252:46206 dest: /172.31.23.3:50010 of size 267634 2019-11-02 16:14:01,833 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742097_1273 src: /172.31.23.3:50512 dest: /172.31.23.3:50010 2019-11-02 16:14:01,837 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /172.31.23.3:50512, dest: /172.31.23.3:50010, bytes: 1029, op: HDFS_WRITE, cliID: DFSClient_attempt_1572710114754_0009_m_000000_0_-2142389405_1, offset: 0, srvID: 77d8a096-eb31-4724-8d4a-187b54f9fe8d, blockid: BP-793432708-172.31.20.57-1572709584342:blk_1073742097_1273, duration(ns): 3798130 2019-11-02 16:14:01,837 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder: BP-793432708-172.31.20.57-1572709584342:blk_1073742097_1273, type=LAST_IN_PIPELINE terminating 2019-11-02 16:14:01,993 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742076_1252 src: /172.31.19.252:46218 dest: /172.31.23.3:50010 2019-11-02 16:14:01,994 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742076_1252 src: /172.31.19.252:46218 dest: /172.31.23.3:50010 of size 375618 2019-11-02 16:14:01,995 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving BP-793432708-172.31.20.57-1572709584342:blk_1073742075_1251 src: /172.31.19.252:46216 dest: /172.31.23.3:50010 2019-11-02 16:14:01,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742078_1254 to 172.31.19.252:50010 2019-11-02 16:14:01,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received BP-793432708-172.31.20.57-1572709584342:blk_1073742075_1251 src: /172.31.19.252:46216 dest: /172.31.23.3:50010 of size 1765905 2019-11-02 16:14:01,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742079_1255 to 172.31.19.252:50010 2019-11-02 16:14:02,002 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742078_1254 (numBytes=279012) to /172.31.19.252:50010 2019-11-02 16:14:02,008 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742079_1255 (numBytes=1344870) to /172.31.19.252:50010 2019-11-02 16:14:04,999 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Can't replicate block BP-793432708-172.31.20.57-1572709584342:blk_1073742080_1256 because on-disk length 990927 is shorter than NameNode recorded length 9223372036854775807 2019-11-02 16:14:08,005 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073741990_1166 to 172.31.19.252:50010 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073741996_1172 to 172.31.19.252:50010 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742080_1256 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742080 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742081_1257 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742081 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742083_1259 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742083 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742086_1262 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742086 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742087_1263 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742087 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742093_1269 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742093 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742056_1232 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742056 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742057_1233 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742057 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742058_1234 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742058 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742059_1235 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742059 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742060_1236 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742060 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742061_1237 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742061 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742062_1238 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742062 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742063_1239 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742063 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742080_1256 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742080 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742064_1240 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742064 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742065_1241 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742065 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742066_1242 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742066 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742067_1243 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742067 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742068_1244 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742068 for deletion 2019-11-02 16:14:08,006 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742069_1245 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742069 for deletion 2019-11-02 16:14:08,007 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742070_1246 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742070 for deletion 2019-11-02 16:14:08,007 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742081_1257 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742081 2019-11-02 16:14:08,007 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742071_1247 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742071 for deletion 2019-11-02 16:14:08,007 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742072_1248 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742072 for deletion 2019-11-02 16:14:08,007 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742083_1259 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742083 2019-11-02 16:14:08,007 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742074_1250 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742074 for deletion 2019-11-02 16:14:08,008 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742075_1251 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742075 for deletion 2019-11-02 16:14:08,008 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742076_1252 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742076 for deletion 2019-11-02 16:14:08,008 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742086_1262 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742086 2019-11-02 16:14:08,008 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742078_1254 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742078 for deletion 2019-11-02 16:14:08,008 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Scheduling blk_1073742079_1255 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742079 for deletion 2019-11-02 16:14:08,009 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742087_1263 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742087 2019-11-02 16:14:08,009 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742093_1269 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir1/blk_1073742093 2019-11-02 16:14:08,009 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073741996_1172 (numBytes=375618) to /172.31.19.252:50010 2019-11-02 16:14:08,009 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742056_1232 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742056 2019-11-02 16:14:08,009 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742057_1233 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742057 2019-11-02 16:14:08,009 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742058_1234 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742058 2019-11-02 16:14:08,009 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742059_1235 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742059 2019-11-02 16:14:08,009 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073741990_1166 (numBytes=36455) to /172.31.19.252:50010 2019-11-02 16:14:08,009 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742060_1236 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742060 2019-11-02 16:14:08,010 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742061_1237 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742061 2019-11-02 16:14:08,010 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742062_1238 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742062 2019-11-02 16:14:08,010 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742063_1239 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742063 2019-11-02 16:14:08,010 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742064_1240 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742064 2019-11-02 16:14:08,011 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742065_1241 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742065 2019-11-02 16:14:08,011 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742066_1242 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742066 2019-11-02 16:14:08,011 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742067_1243 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742067 2019-11-02 16:14:08,011 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742068_1244 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742068 2019-11-02 16:14:08,012 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742069_1245 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742069 2019-11-02 16:14:08,012 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742070_1246 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742070 2019-11-02 16:14:08,012 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742071_1247 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742071 2019-11-02 16:14:08,012 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742072_1248 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742072 2019-11-02 16:14:08,012 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742074_1250 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742074 2019-11-02 16:14:08,013 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742075_1251 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742075 2019-11-02 16:14:08,013 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742076_1252 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742076 2019-11-02 16:14:08,013 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742078_1254 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742078 2019-11-02 16:14:08,013 INFO org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetAsyncDiskService: Deleted BP-793432708-172.31.20.57-1572709584342 blk_1073742079_1255 file /data/dn/current/BP-793432708-172.31.20.57-1572709584342/current/finalized/subdir0/subdir0/blk_1073742079 2019-11-02 16:14:11,006 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(172.31.23.3:50010, datanodeUuid=77d8a096-eb31-4724-8d4a-187b54f9fe8d, infoPort=50075, infoSecurePort=0, ipcPort=50020, storageInfo=lv=-57;cid=CID-d359683c-0d1c-450d-81d1-67712499ef0b;nsid=696889141;c=1572709584342) Starting thread to transfer BP-793432708-172.31.20.57-1572709584342:blk_1073742004_1180 to 172.31.19.252:50010 2019-11-02 16:14:11,007 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: DataTransfer, at hadoop2:50010: Transmitted BP-793432708-172.31.20.57-1572709584342:blk_1073742004_1180 (numBytes=25496) to /172.31.19.252:50010 2019-11-02 17:01:35,904 INFO org.apache.hadoop.hdfs.server.datanode.DirectoryScanner: BlockPool BP-793432708-172.31.20.57-1572709584342 Total blocks: 88, missing metadata files:0, missing block files:0, missing blocks in memory:0, mismatched blocks:0 2019-11-02 19:52:23,330 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Successfully sent block report 0x15733bb21ccd9a44, containing 1 storage report(s), of which we sent 1. The reports had 88 total blocks and used 1 RPC(s). This took 1 msec to generate and 2 msecs for RPC and NN processing. Got back one command: FinalizeCommand/5. 2019-11-02 19:52:23,330 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Got finalize command for block pool BP-793432708-172.31.20.57-1572709584342 ```

当jar在hdfs的时候提交spark job报错: (一)jar不在hdfs上的时候提交spark任务成功，使用的命令： spark-submit --master spark://192.168.244.130:7077 --class cn.com.cnpc.klmy.common.WordCount2 --executor-memory 1G --total-executor-cores 2 /root/modelcall-2.0.jar (二)而当jar在hdfs上的时候提交spark任务报错：classNotFoundException呢？，命令如下： spark-submit --master spark://192.168.244.130:7077 --class cn.com.cnpc.klmy.common.WordCount2 --executor-memory 1G --total-executor-cores 2 hdfs://192.168.244.130:9000/mdjar/modelcall-2.0.jar 请教各位大咖这到底是什么原因造成的？望各位大咖不吝赐教！跪谢！！！注：hdfs能够正常访问，代码里面产生的结果存在hdfs上（第一情况正常运行，在hdfs上能够查看到结果）

Hive on spark查询报错。: 求助！！！在hadoop使用Hive on spark执行Bigbench测试时，一直会有报错，log信息： FAILED: SemanticException Failed to get a spark session: org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create spark client. WARN: The method class org.apache.commons.logging.impl.SLF4JLogFactory#release() was invoked. WARN: Please see http://www.slf4j.org/codes.html#release for an explanation. An error occured while running command: ========== runEngineCmd -f /var/lib/hadoop-hdfs/Big-Bench/engines/hive/queries/q04/q04.sql ========== 在网上查了很多资料，有说版本不匹配的，有说是概率性问题，有没有大佬来瞅一眼啊。。哭了

esxi5客户端访问映射到本地的hdfs问题，求解答~~: 我在VMware workstation装了3台ubuntu12.04的虚拟机，搭建hadoop。前面一切顺利，然后通过hdfs-webdav.war　将hdfs部署到webdav上浏览器访问http://namenode:8080/hdfs-webdav 能够成功访问hdfs 。然后通过davfs2 将hdfs-webdav映射到本地。。语句：mount -t davfs http://namenode:8080/hdfs-webdav /data/hdfs -o uid=hadoop,gid=root,dir_mode=775 成功映射到本地接下来就是问题了想通过pc机的VMware vSphere client作为客户端，利用iscsi访问挂载到本地的/data/hdfs 方法如下：dd if=/dev/zero of=/data/hdfs/hdfs.img count=0 obs=1 seek=30G 创建30G的虚拟镜像文件。然后启动 iscsitarget /etc/iet/ietd.conf 文件里面设置lun0 path=/data/hdfs/hdfs.img Type=fileio 然后VMware vSphere Client客户可以找到iscsitarget 也成功加载了30G的存储器。。但是这个存储器始终无法和namenode里面的hdfs同步数据通过http://namenode:8080/hdfs-webdav也无法访问到hdfs.img里面的数据，并且显示hdfs.img的大小是0k。。 pc机的网上邻居，添加网络邻居网络地址写http://namenode:8080/hdfs-webdav却可以实现和hdfs同步。。求问怎么让VMware vSphere Client的可以访问到映射到本地磁盘的hadoop集群数据。还有帮忙看看我操作中间有没有出现什么错误呢

flume 的hdfs sink效率低的问题: 哈喽，大家好，我现在遇到了一个问题。我的flume向hdfs中写文件时，效率比较低大约1G/3分钟我单独测试时用put方式 1分钟能达到8G 如果用file sink也能达到1分钟1G 日志没有任何异常只是DEBUG的时候发现每次提交一个块用时将近20秒有高手能帮忙分析下是什么原因么 client.sources = r1 client.channels = c1 client.sinks = k1 client.sources.r1.type = spooldir client.sources.r1.spoolDir = /var/data/tmpdata client.sources.r1.fileSuffix = .COMPLETED client.sources.r1.deletePolicy = never client.sources.r1.batchSize = 500 client.sources.r1.channels = c1 client.channels.c1.type = memory client.channels.c1.capacity = 1000000 client.channels.c1.transactionCapacity = 50000 client.channels.c1.keep-alive = 3 client.sinks.k1.type = hdfs client.sinks.k1.hdfs.path = /flume/events/%Y%m%d/%H client.sinks.k1.hdfs.useLocalTimeStamp = true client.sinks.k1.hdfs.rollInterval = 3600 client.sinks.k1.hdfs.rollSize = 1000000000 client.sinks.k1.hdfs.rollCount = 0 client.sinks.k1.hdfs.batchSize = 500 client.sinks.k1.hdfs.callTimeout = 30000 client.sinks.k1.hdfs.fileType = DataStream client.sinks.k1.channel = c1 12 Aug 2015 16:14:24,739 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 12 Aug 2015 16:14:54,740 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 12 Aug 2015 16:15:24,740 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 12 Aug 2015 16:15:54,741 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 12 Aug 2015 16:16:24,742 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 12 Aug 2015 16:16:54,742 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 12 Aug 2015 16:17:24,743 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 12 Aug 2015 16:17:54,744 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 12 Aug 2015 16:18:24,745 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 12 Aug 2015 16:18:54,746 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 12 Aug 2015 16:19:24,746 DEBUG [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:126) - Checking file:../conf/flume-client.conf for changes 日志没有问题就是慢

python操作hdfs时抛出hdfs.util.HdfsError: None的异常？: python操作hdfs向hdfs上传文件时抛出异常 File "E:/代码/2019-6/6-10/myhdfs.py", line 7, in <module> client.upload('/foo','E:\\资料\\py.txt') File "E:\python-01\bin\lib\site-packages\hdfs\client.py", line 605, in upload raise err File "E:\python-01\bin\lib\site-packages\hdfs\client.py", line 594, in upload _upload(path_tuple) File "E:\python-01\bin\lib\site-packages\hdfs\client.py", line 524, in _upload self.write(_temp_path, wrap(reader, chunk_size, progress), **kwargs) File "E:\python-01\bin\lib\site-packages\hdfs\client.py", line 456, in write buffersize=buffersize, File "E:\python-01\bin\lib\site-packages\hdfs\client.py", line 112, in api_handler raise err File "E:\python-01\bin\lib\site-packages\hdfs\client.py", line 107, in api_handler **self.kwargs File "E:\python-01\bin\lib\site-packages\hdfs\client.py", line 210, in _request _on_error(response) File "E:\python-01\bin\lib\site-packages\hdfs\client.py", line 50, in _on_error raise HdfsError(message, exception=exception) hdfs.util.HdfsError: None

osx下编译hadoop-2.5.2-src出错: [exec] /pein/hadoop/hadoop-2.5.2-src/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:61:9: warning: implicit declaration of function 'clock_gettime' is invalid in C99 [-Wimplicit-function-declaration] [exec] if (clock_gettime(CLOCK_MONOTONIC, &watch->start)) { [exec] ^ [exec] /pein/hadoop/hadoop-2.5.2-src/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:61:23: error: use of undeclared identifier 'CLOCK_MONOTONIC' [exec] if (clock_gettime(CLOCK_MONOTONIC, &watch->start)) { [exec] ^ [exec] /pein/hadoop/hadoop-2.5.2-src/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/test/vecsum.c:79:23: error: use of undeclared identifier 'CLOCK_MONOTONIC' [exec] if (clock_gettime(CLOCK_MONOTONIC, &watch->stop)) { [exec] ^ [exec] 1 warning and 2 errors generated. [exec] make[2]: *** [CMakeFiles/test_libhdfs_vecsum.dir/main/native/libhdfs/test/vecsum.c.o] Error 1 [exec] make[1]: *** [CMakeFiles/test_libhdfs_vecsum.dir/all] Error 2 [exec] make: *** [all] Error 2

datax从hive导出数据到mysql: 从MySQL导入数据到hive上是没问题的，从hive上导出数据时，提示： [您的配置错误.]. - 列配置信息有错误. 因为您配置的任务中，源头读取字段数:1 与目的表要写入的字段数:4 不相等. 请检查您的配置并作出修改. 下面是我的json文件： ```{ "job": { "content": [{ "reader": { "parameter": { "path": "/apps/hive/warehouse/test.db/job01", "column": ["*"], "defaultFS": "hdfs://xxxx.xx.xx:8020", "encoding": "utf-8", "fieldDelimiter": "\u0001", "fileType": "text" }, "name": "hdfsreader" }, "writer": { "parameter": { "password": "*****", "column": ["*"], "connection": [{ "jdbcUrl": "jdbc:mysql://xxxxx:3308/groundcherry", "table": ["scoop_test"] }], "writeMode": "insert", "username": "****" }, "name": "mysqlwriter" } }], "setting": { "speed": { "channel": 1 } } } } ``` 网上说是分割符的问题，使用默认的 \u0001，导入的时候可以导入进去，导出的时候就报错了，使用","号也是一样，hive上建表时指定的字段分隔符与这个是一致的，有点怀疑是不是字段分割的原因，还会有什么原因导致这种错误呢，请大家指点指点谢谢

爬虫福利二之妹子图网MM批量下载: 爬虫福利一：27报网MM批量下载点击看了本文，相信大家对爬虫一定会产生强烈的兴趣，激励自己去学习爬虫，在这里提前祝：大家学有所成！目标网站：妹子图网环境：Python3.x 相关第三方模块：requests、beautifulsoup4 Re：各位在测试时只需要将代码里的变量 path 指定为你当前系统要保存的路径，使用 python xxx.py 或IDE运行即可。

Java学习的正确打开方式: 在博主认为，对于入门级学习java的最佳学习方法莫过于视频+博客+书籍+总结，前三者博主将淋漓尽致地挥毫于这篇博客文章中，至于总结在于个人，实际上越到后面你会发现学习的最好方式就是阅读参考官方文档其次就是国内的书籍，博客次之，这又是一个层次了，这里暂时不提后面再谈。博主将为各位入门java保驾护航，各位只管冲鸭！！！上天是公平的，只要不辜负时间，时间自然不会辜负你。何谓学习？博主所理解的学习，它

大学四年自学走来，这些私藏的实用工具/学习网站我贡献出来了: 大学四年，看课本是不可能一直看课本的了，对于学习，特别是自学，善于搜索网上的一些资源来辅助，还是非常有必要的，下面我就把这几年私藏的各种资源，网站贡献出来给你们。主要有：电子书搜索、实用工具、在线视频学习网站、非视频学习网站、软件下载、面试/求职必备网站。注意：文中提到的所有资源，文末我都给你整理好了，你们只管拿去，如果觉得不错，转发、分享就是最大的支持了。一、PDF搜索网站推荐对于大部

linux系列之常用运维命令整理笔录: 本博客记录工作中需要的linux运维命令，大学时候开始接触linux，会一些基本操作，可是都没有整理起来，加上是做开发，不做运维，有些命令忘记了，所以现在整理成博客，当然vi，文件操作等就不介绍了，慢慢积累一些其它拓展的命令，博客不定时更新顺便拉下票，我在参加csdn博客之星竞选，欢迎投票支持，每个QQ或者微信每天都可以投5票，扫二维码即可，http://m234140.nofollow.ax.

比特币原理详解: 一、什么是比特币比特币是一种电子货币，是一种基于密码学的货币，在2008年11月1日由中本聪发表比特币白皮书，文中提出了一种去中心化的电子记账系统，我们平时的电子现金是银行来记账，因为银行的背后是国家信用。去中心化电子记账系统是参与者共同记账。比特币可以防止主权危机、信用风险。其好处不多做赘述，这一层面介绍的文章很多，本文主要从更深层的技术原理角度进行介绍。二、问题引入假设现有4个人

程序员接私活怎样防止做完了不给钱？: 首先跟大家说明一点，我们做 IT 类的外包开发，是非标品开发，所以很有可能在开发过程中会有这样那样的需求修改，而这种需求修改很容易造成扯皮，进而影响到费用支付，甚至出现做完了项目收不到钱的情况。那么，怎么保证自己的薪酬安全呢？我们在开工前，一定要做好一些证据方面的准备（也就是“讨薪”的理论依据），这其中最重要的就是需求文档和验收标准。一定要让需求方提供这两个文档资料作为开发的基础。之后开发

网页实现一个简单的音乐播放器（大佬别看。(⊙﹏⊙)）: 今天闲着无事，就想写点东西。然后听了下歌，就打算写个播放器。于是乎用h5 audio的加上js简单的播放器完工了。欢迎改进留言。演示地点跳到演示地点 html代码如下`<!DOCTYPE html> <html> <head> <title>music</title> <meta charset="utf-8"&gt

Python十大装B语法: Python 是一种代表简单思想的语言，其语法相对简单，很容易上手。不过，如果就此小视 Python 语法的精妙和深邃，那就大错特错了。本文精心筛选了最能展现 Python 语法之精妙的十个知识点，并附上详细的实例代码。如能在实战中融会贯通、灵活使用，必将使代码更为精炼、高效，同时也会极大提升代码B格，使之看上去更老练，读起来更优雅。 1. for - else 什么？不是 if 和 else 才

数据库优化 - SQL优化: 前面一篇文章从实例的角度进行数据库优化，通过配置一些参数让数据库性能达到最优。但是一些“不好”的SQL也会导致数据库查询变慢，影响业务流程。本文从SQL角度进行数据库优化，提升SQL运行效率。判断问题SQL 判断SQL是否有问题时可以通过两个表象进行判断：系统级别表象 CPU消耗严重 IO等待严重页面响应时间过长

2019年11月中国大陆编程语言排行榜: 2019年11月2日，我统计了某招聘网站，获得有效程序员招聘数据9万条。针对招聘信息，提取编程语言关键字，并统计如下：编程语言比例 rank pl_ percentage 1 java 33.62% 2 c/c++ 16.42% 3 c_sharp 12.82% 4 javascript 12.31% 5 python 7.93% 6 go 7.25% 7

通俗易懂地给女朋友讲：线程池的内部原理: 餐厅的约会餐盘在灯光的照耀下格外晶莹洁白，女朋友拿起红酒杯轻轻地抿了一小口，对我说：“经常听你说线程池，到底线程池到底是个什么原理？”我楞了一下，心里想女朋友今天是怎么了，怎么突然问出这么专业的问题，但做为一个专业人士在女朋友面前也不能露怯啊，想了一下便说：“我先给你讲讲我前同事老王的故事吧！” 大龄程序员老王老王是一个已经北漂十多年的程序员，岁数大了，加班加不动了，升迁也无望，于是拿着手里

经典算法（5）杨辉三角: 写在前面：我是扬帆向海，这个昵称来源于我的名字以及女朋友的名字。我热爱技术、热爱开源、热爱编程。技术是开源的、知识是共享的。这博客是对自己学习的一点点总结及记录，如果您对 Java、算法感兴趣，可以关注我的动态，我们一起学习。用知识改变命运，让我们的家人过上更好的生活。目录一、杨辉三角的介绍二、杨辉三角的算法思想三、代码实现1.第一种写法2.第二种写法一、杨辉三角的介绍百度

腾讯算法面试题：64匹马8个跑道需要多少轮才能选出最快的四匹？: 昨天，有网友私信我，说去阿里面试，彻底的被打击到了。问了为什么网上大量使用ThreadLocal的源码都会加上private static？他被难住了，因为他从来都没有考虑过这个问题。无独有偶，今天笔者又发现有网友吐槽了一道腾讯的面试题，我们一起来看看。腾讯算法面试题：64匹马8个跑道需要多少轮才能选出最快的四匹？在互联网职场论坛，一名程序员发帖求助到。二面腾讯，其中一个算法题：64匹

面试官：你连RESTful都不知道我怎么敢要你？: 面试官：了解RESTful吗？我：听说过。面试官：那什么是RESTful？我：就是用起来很规范，挺好的面试官：是RESTful挺好的，还是自我感觉挺好的我：都挺好的。面试官：… 把门关上。我：… 要干嘛？先关上再说。面试官：我说出去把门关上。我：what ？，夺门而去文章目录01 前言02 RESTful的来源03 RESTful6大原则1. C-S架构2. 无状态3.统一的接

JDK12 Collectors.teeing 你真的需要了解一下: 前言在 Java 12 里面有个非常好用但在官方 JEP 没有公布的功能，因为它只是 Collector 中的一个小改动，它的作用是 merge 两个 collector 的结果，这句话

为啥国人偏爱Mybatis，而老外喜欢Hibernate/JPA呢？: 关于SQL和ORM的争论，永远都不会终止，我也一直在思考这个问题。昨天又跟群里的小伙伴进行了一番讨论，感触还是有一些，于是就有了今天这篇文。声明：本文不会下关于Mybatis和JPA两个持久层框架哪个更好这样的结论。只是摆事实，讲道理，所以，请各位看官勿喷。一、事件起因关于Mybatis和JPA孰优孰劣的问题，争论已经很多年了。一直也没有结论，毕竟每个人的喜好和习惯是大不相同的。我也看

SQL-小白最佳入门sql查询一: 一说明如果是初学者，建议去网上寻找安装Mysql的文章安装，以及使用navicat连接数据库，以后的示例基本是使用mysql数据库管理系统；二准备前提需要建立一张学生表，列分别是id，名称，年龄，学生信息；本示例中文章篇幅原因SQL注释略；建表语句： CREATE TABLE `student` ( `id` int(11) NOT NULL AUTO_INCREMENT, `

项目中的if else太多了，该怎么重构？: 介绍最近跟着公司的大佬开发了一款IM系统，类似QQ和微信哈，就是聊天软件。我们有一部分业务逻辑是这样的 if (msgType = "文本") { // dosomething } else if(msgType = "图片") { // doshomething } else if(msgType = "视频") { // doshomething } else { // dosho

【图解经典算法题】如何用一行代码解决约瑟夫环问题: 约瑟夫环问题算是很经典的题了，估计大家都听说过，然后我就在一次笔试中遇到了，下面我就用 3 种方法来详细讲解一下这道题，最后一种方法学了之后保证让你可以让你装逼。问题描述：编号为 1-N 的 N 个士兵围坐在一起形成一个圆圈，从编号为 1 的士兵开始依次报数（1，2，3…这样依次报），数到 m 的士兵会被杀死出列，之后的士兵再从 1 开始报数。直到最后剩下一士兵，求这个士兵的编号。 1、方

致 Python 初学者: 文章目录1. 前言2. 明确学习目标，不急于求成，不好高骛远3. 在开始学习 Python 之前，你需要做一些准备2.1 Python 的各种发行版2.2 安装 Python2.3 选择一款趁手的开发工具3. 习惯使用IDLE，这是学习python最好的方式4. 严格遵从编码规范5. 代码的运行、调试5. 模块管理5.1 同时安装了py2/py35.2 使用Anaconda，或者通过IDE来安装模

“狗屁不通文章生成器”登顶GitHub热榜，分分钟写出万字形式主义大作: 一、垃圾文字生成器介绍最近在浏览GitHub的时候，发现了这样一个骨骼清奇的雷人项目，而且热度还特别高。项目中文名：狗屁不通文章生成器项目英文名：BullshitGenerator 根据作者的介绍，他是偶尔需要一些中文文字用于GUI开发时测试文本渲染，因此开发了这个废话生成器。但由于生成的废话实在是太过富于哲理，所以最近已经被小伙伴们给玩坏了。他的文风可能是这样的：你发现，

程序员：我终于知道post和get的区别: IT界知名的程序员曾说：对于那些月薪三万以下，自称IT工程师的码农们，其实我们从来没有把他们归为我们IT工程师的队伍。他们虽然总是以IT工程师自居，但只是他们一厢情愿罢了。此话一出，不知激起了多少(码农)程序员的愤怒，却又无可奈何，于是码农问程序员。码农：你知道get和post请求到底有什么区别？程序员：你看这篇就知道了。码农：你月薪三万了？程序员：嗯。码农：你是怎么做到的? 程序员：

GitHub标星近1万：只需5秒音源，这个网络就能实时“克隆”你的声音: 作者 | Google团队译者 | 凯隐编辑 | Jane出品 | AI科技大本营（ID：rgznai100）本文中，Google 团队提出了一种文本语音合成（text to speech）神经系统，能通过少量样本学习到多个不同说话者（speaker）的语音特征，并合成他们的讲话音频。此外，对于训练时网络没有接触过的说话者，也能在不重新训练的情况下，仅通过未知说话者数秒的音频来合成其讲话音频，即网

《程序人生》系列-这个程序员只用了20行代码就拿了冠军: 你知道的越多，你不知道的越多点赞再看，养成习惯GitHub上已经开源https://github.com/JavaFamily，有一线大厂面试点脑图，欢迎Star和完善前言这一期不算《吊打面试官》系列的，所有没前言我直接开始。絮叨本来应该是没有这期的，看过我上期的小伙伴应该是知道的嘛，双十一比较忙嘛，要值班又要去帮忙拍摄年会的视频素材，还得搞个程序员一天的Vlog，还要写BU

加快推动区块链技术和产业创新发展，2019可信区块链峰会在京召开: 11月8日，由中国信息通信研究院、中国通信标准化协会、中国互联网协会、可信区块链推进计划联合主办，科技行者协办的2019可信区块链峰会将在北京悠唐皇冠假日酒店开幕。　　区块链技术被认为是继蒸汽机、电力、互联网之后，下一代颠覆性的核心技术。如果说蒸汽机释放了人类的生产力，电力解决了人类基本的生活需求，互联网彻底改变了信息传递的方式，区块链作为构造信任的技术有重要的价值。　　1

程序员把地府后台管理系统做出来了，还有3.0版本！12月7号最新消息：已在开发中有github地址: 第一幕：缘起听说阎王爷要做个生死簿后台管理系统，我们派去了一个程序员…… 996程序员做的梦：第一场：团队招募为了应对地府管理危机，阎王打算找“人”开发一套地府后台管理系统，于是就在地府总经办群中发了项目需求。话说还是中国电信的信号好，地府都是满格，哈哈！！！经常会有外行朋友问：看某网站做的不错，功能也简单，你帮忙做一下？而这次，面对这样的需求，这个程序员

网易云6亿用户音乐推荐算法: 网易云音乐是音乐爱好者的集聚地，云音乐推荐系统致力于通过 AI 算法的落地，实现用户千人千面的个性化推荐，为用户带来不一样的听歌体验。本次分享重点介绍 AI 算法在音乐推荐中的应用实践，以及在算法落地过程中遇到的挑战和解决方案。将从如下两个部分展开： AI 算法在音乐推荐中的应用音乐场景下的 AI 思考从 2013 年 4 月正式上线至今，网易云音乐平台持续提供着：乐屏社区、UGC

【技巧总结】位运算装逼指南: 位算法的效率有多快我就不说，不信你可以去用 10 亿个数据模拟一下，今天给大家讲一讲位运算的一些经典例子。不过，最重要的不是看懂了这些例子就好，而是要在以后多去运用位运算这些技巧，当然，采用位运算，也是可以装逼的，不信，你往下看。我会从最简单的讲起，一道比一道难度递增，不过居然是讲技巧，那么也不会太难，相信你分分钟看懂。判断奇偶数判断一个数是基于还是偶数，相信很多人都做过，一般的做法的代码如下

【管理系统课程设计】美少女手把手教你后台管理: 【后台管理系统】URL设计与建模分析+项目源码+运行界面栏目管理、文章列表、用户管理、角色管理、权限管理模块（文章最后附有源码）一、这是一个什么系统? 1.1 学习后台管理系统的原因随着时代的变迁，现如今各大云服务平台横空出世，市面上有许多如学生信息系统、图书阅读系统、停车场管理系统等的管理系统，而本人家里就有人在用烟草销售系统，直接在网上完成挑选、购买与提交收货点，方便又快捷。试想，

4G EPS 第四代移动通信系统: 目录文章目录目录4G EPSEPS 的架构EPS 的参考模型E-UTRANUEeNodeBEPCMME（移动性控制处理单元）S-GW（E-RAB 无线访问承载接入点）P-GW（PDN 接入点）HSS（用户认证中心）PCRF（计费规则与策略）SPR（用户档案）OCS（在线计费）OFCS（离线计费）接口类型Uu 接口（空中接口，UE 和 AN 之间）S1 接口（AN 和 CN 之间）S1-U 接口和