IPC和8020端口通信出现问题 hadoop

24347 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient writeChunk allocating new packet seqno=0, src=/tmp/hadoop-yarn/staging/root/.staging/job_1508918621838_0014/job.xml, packetSize=65532, chunksPerPacket=127, bytesCurBlock=0
24482 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient writeChunk packet full seqno=0, src=/tmp/hadoop-yarn/staging/root/.staging/job_1508918621838_0014/job.xml, bytesCurBlock=65024, blockSize=134217728, appendChunk=false
24482 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - Queued packet 0
24486 [Thread-14] DEBUG org.apache.hadoop.hdfs.DFSClient - Allocating new block
24489 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client - IPC Client (453523494) connection to master/192.168.3.60:8020 from root sending #32
24489 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - computePacketChunkSize: src=/tmp/hadoop-yarn/staging/root/.staging/job_1508918621838_0014/job.xml, chunkSize=516, chunksPerPacket=127, packetSize=65532
24489 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient writeChunk allocating new packet seqno=1, src=/tmp/hadoop-yarn/staging/root/.staging/job_1508918621838_0014/job.xml, packetSize=65532, chunksPerPacket=127, bytesCurBlock=65024
24495 [IPC Client (453523494) connection to master/192.168.3.60:8020 from root] DEBUG org.apache.hadoop.ipc.Client - IPC Client (453523494) connection to master/192.168.3.60:8020 from root got value #32
24495 [Thread-14] DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine - Call: addBlock took 7ms
24495 [Thread-14] DEBUG org.apache.hadoop.hdfs.DFSClient - pipeline = DatanodeInfoWithStorage[192.168.3.60:50010,DS-c7341036-19f0-49e8-a3de-980de9c85471,DISK]
24495 [Thread-14] DEBUG org.apache.hadoop.hdfs.DFSClient - pipeline = DatanodeInfoWithStorage[192.168.0.61:50010,DS-9afe2d04-57db-4569-863a-2eaef5b48790,DISK]
24495 [Thread-14] DEBUG org.apache.hadoop.hdfs.DFSClient - pipeline = DatanodeInfoWithStorage[192.168.0.62:50010,DS-51bb34a6-f71b-40f1-a2b6-0eae0828e3e6,DISK]
24495 [Thread-14] DEBUG org.apache.hadoop.hdfs.DFSClient - Connecting to datanode 192.168.3.60:50010
24496 [Thread-14] DEBUG org.apache.hadoop.hdfs.DFSClient - Send buf size 124928
24496 [Thread-14] DEBUG org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient - SASL client skipping handshake in unsecured configuration for addr = /192.168.3.60, datanodeId = DatanodeInfoWithStorage[192.168.3.60:50010,DS-c7341036-19f0-49e8-a3de-980de9c85471,DISK]
24511 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - Queued packet 1
24511 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - Queued packet 2
24511 [main] DEBUG org.apache.hadoop.hdfs.DFSClient - Waiting for ack for: 2
24515 [DataStreamer for file /tmp/hadoop-yarn/staging/root/.staging/job_1508918621838_0014/job.xml block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272] DEBUG org.apache.hadoop.hdfs.DFSClient - DataStreamer block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272 sending packet packet seqno: 0 offsetInBlock: 0 lastPacketInBlock: false lastByteOffsetInBlock: 65024
24516 [DataStreamer for file /tmp/hadoop-yarn/staging/root/.staging/job_1508918621838_0014/job.xml block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272] DEBUG org.apache.hadoop.hdfs.DFSClient - DataStreamer block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272 sending packet packet seqno: 1 offsetInBlock: 65024 lastPacketInBlock: false lastByteOffsetInBlock: 96423
24523 [ResponseProcessor for block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient seqno: 0 reply: 0 reply: 0 reply: 0 downstreamAckTimeNanos: 5654125
24523 [ResponseProcessor for block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient seqno: 1 reply: 0 reply: 0 reply: 0 downstreamAckTimeNanos: 5760521
24524 [DataStreamer for file /tmp/hadoop-yarn/staging/root/.staging/job_1508918621838_0014/job.xml block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272] DEBUG org.apache.hadoop.hdfs.DFSClient - DataStreamer block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272 sending packet packet seqno: 2 offsetInBlock: 96423 lastPacketInBlock: true lastByteOffsetInBlock: 96423
24529 [ResponseProcessor for block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272] DEBUG org.apache.hadoop.hdfs.DFSClient - DFSClient seqno: 2 reply: 0 reply: 0 reply: 0 downstreamAckTimeNanos: 2267573
24530 [DataStreamer for file /tmp/hadoop-yarn/staging/root/.staging/job_1508918621838_0014/job.xml block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272] DEBUG org.apache.hadoop.hdfs.DFSClient - Closing old block BP-60132808-192.168.3.60-1504150527865:blk_1073811055_70272
24530 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client - IPC Client (453523494) connection to master/192.168.3.60:8020 from root sending #33
43037 [main] DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 1ms
43037 [main] INFO org.apache.hadoop.mapreduce.Job - Job job_1508918621838_0014 failed with state FAILED due to: Application application_1508918621838_0014 failed 2 times due to AM Container for appattempt_1508918621838_0014_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://master:8088/proxy/application_1508918621838_0014/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1508918621838_0014_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:578)
at org.apache.hadoop.util.Shell.run(Shell.java:481)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:763)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)

Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
43038 [main] DEBUG org.apache.hadoop.security.UserGroupInformation - PrivilegedAction as:root (auth:SIMPLE) from:org.apache.hadoop.mapreduce.Job.getCounters(Job.java:775)
43042 [IPC Parameter Sending Thread #0] DEBUG org.apache.hadoop.ipc.Client - IPC Client (453523494) connection to master/192.168.3.60:8032 from root sending #95
43045 [IPC Client (453523494) connection to master/192.168.3.60:8032 from root] DEBUG org.apache.hadoop.ipc.Client - IPC Client (453523494) connection to master/192.168.3.60:8032 from root got value #95
43047 [main] DEBUG org.apache.hadoop.ipc.ProtobufRpcEngine - Call: getApplicationReport took 6ms
43088 [main] INFO org.apache.hadoop.mapreduce.Job - Counters: 0
14
43096 [Thread-3] DEBUG org.apache.hadoop.ipc.Client - stopping client from cache: org.apache.hadoop.ipc.Client@4690b489

Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
其他相关推荐
hadoop的50010端口出现大量的close_wait

hadoop的50010端口出现大量的close_wait hadoop的50010端口出现大量的close_wait

hadoop启动后端口均未开启

所用的系统是ubuntu18.04 hadoop版本为3.1.3 格式化namenode正常,启动正常,jps后namenode datanode sm都正常,就是端口没启动![图片说明](https://img-ask.csdn.net/upload/202004/03/1585880272_37821.png) 几乎所有网上的方法都试过了,就是端口启动不了

无法打开HADOOP的网页端口50070 8088

无法打开HADOOP的网页端口50070 8088 这是端口情况 希望能解释一下问题在哪 谢谢 有需要别的信息的可以问我 我在线回复 多谢了 ![图片说明](https://img-ask.csdn.net/upload/202003/31/1585636954_638642.png)

关于HADOOP相关服务器的4433端口问题

处于信息安全考虑,公司近期对违禁端口进行整改,发现 HADOOP服务有 4433 违禁端口,公司要求对4433端口整改。该端口被一个PYTHON进程占用。将其杀掉后,过一会又自动启动了,应该是存在守护进程。 请教各位老师,此端口能否禁用或修改?

hadoop2.4.1 ResourceManager的8088 web端口无法访问

hadoop ResourceManager的8088端口无法访问。我们的集群是部署在阿里云服务器,版本为hadoop2.4.1。希望大神门能指导一下,谢谢!

Hadoop界面访问问题,局域网其他电脑无法访问

# 服务器虚拟机上安装了Hadoop(Centos 7 系统),想通过局域网的电脑访问其Web接口 ## 虚拟机上可以,别的机器上不行,自己也尝试了很多方法

hadoop启动dfs时出现问题

刚刚接触hadoop,namenode格式化后,启动hadoop sudo sbin/start-dfs.sh 出现错误: hadoop@qiaoyu-Lenovo-G460:/usr/local/hadoop-2.4.1$ sudo sbin/start-dfs.sh [sudo] password for hadoop: Starting namenodes on [localhost] root@localhost's password: localhost: Permission denied, please try again. 上网找了好久,最多的就是说sudo passwd改变密码 试了之后仍然出现上面的情况,没有变化 急着进行下去,求助啊

hadoop集群添加kerberos认证后namenode启动报ipc认证失败?

问题描述: namenode连接journalnode报错,zkfc连接namenode也报错,都是同样的错。 namenode错误日志: 2019-07-16 18:55:52,617 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hostname/ip:8485. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2019-07-16 18:55:52,616 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hostname/ip:8485. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2019-07-16 18:55:53,438 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 6001 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. 2019-07-16 18:55:53,618 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hostname/ip:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2019-07-16 18:55:53,618 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hostname/ip:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2019-07-16 18:55:53,619 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: hostname/ip:8485. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) 2019-07-16 18:55:54,439 INFO org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager: Waited 7003 ms (timeout=20000 ms) for a response for selectInputStreams. No responses yet. journalnode错误日志: 2019-07-16 18:56:10,836 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for ip:port:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-07-16 18:56:11,939 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for ip:port:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-07-16 18:56:12,391 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for ip:port:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-07-16 18:56:13,341 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for ip:port:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-07-16 18:56:16,212 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for ip:port:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-07-16 18:56:17,871 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for ip:port:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-07-16 18:56:20,902 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for ip:port:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-07-16 18:56:21,081 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for ip:port:null (GSS initiate failed) with true cause: (GSS initiate failed) 查看了一下kdc的日志:可能问题在这里 Jul 16 17:03:50 hadoop01 krb5kdc[47](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.10.10.40: LOOKING_UP_SERVER: authtime 0, root/hadoop00@HADOOP.COM for host/hadoop01@HADOOP.COM, Server not found in Kerberos database Jul 16 17:03:50 hadoop01 krb5kdc[47](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.10.10.40: LOOKING_UP_SERVER: authtime 0, root/hadoop00@HADOOP.COM for host/hadoop00@HADOOP.COM, Server not found in Kerberos database Jul 16 17:03:52 hadoop01 krb5kdc[47](info): AS_REQ (3 etypes {17 16 23}) 10.10.10.40: ISSUE: authtime 1563267832, etypes {rep=17 tkt=18 ses=17}, root/hadoop00@HADOOP.COM for krbtgt/HADOOP.COM@HADOOP.COM Jul 16 17:03:53 hadoop01 krb5kdc[47](info): TGS_REQ (3 etypes {17 16 23}) 10.10.10.40: ISSUE: authtime 1563267832 , etypes {rep=17 tkt=18 ses=17}, root/hadoop00@HADOOP.COM for root/hadoop01@HADOOP.COM Jul 16 17:03:54 hadoop01 krb5kdc[47](info): TGS_REQ (8 etypes {18 17 20 19 16 23 25 26}) 10.10.10.40: LOOKING_UP_SERVER: authtime 0, root/hadoop00@HADOOP.COM for host/hadoop10@HADOOP.COM, Server not found in Kerberos database 所以怀疑问题处在这里,本地kinit root 和HTTP用户都是可以的,正常情况下应该是访问HTTP/hadoop01@HADOOP.COM 而不是host/hadoop01@HADOOP.COM 不知道这里为什么会出现host,请kerberos的大神指导一下

hadoop2.2.0集群rm配置了HA,但nodemanager无法与resourcemanager通信

yarn-site.xml: <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <configuration> <property> <name>yarn.resourcemanager.ha.enabled</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.cluster-id</name> <value>yrc</value> </property> <property> <name>yarn.resourcemanager.ha.rm-ids</name> <value>rm1,rm2</value> </property> <property> <name>yarn.resourcemanager.hostname.rm1</name> <value>11.24.88.242</value> </property> <property> <name>yarn.resourcemanager.hostname.rm2</name> <value>11.24.88.244</value> </property> <property> <name>yarn.resourcemanager.zk-address</name> <value>11.20.26.6:2181,11.20.26.2:2181,11.20.26.3:2181</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> 在yarn.site.xml中MR配置了HA,但一直报错,datanode一直与 0.0.0.0:8031通信,却不与MRtong'x: 2019-08-13 13:33:26,799 FATAL org.apache.hadoop.yarn.server.nodemanager.NodeManager: Error starting NodeManager org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.net.ConnectException: Call From hadoop7/11.20.200.197 to 0.0.0.0:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:181) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:199) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:339) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:386) Caused by: java.net.ConnectException: Call From hadoop7/11.20.200.197 to 0.0.0.0:8031 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.GeneratedConstructorAccessor9.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730) at org.apache.hadoop.ipc.Client.call(Client.java:1351) at org.apache.hadoop.ipc.Client.call(Client.java:1300) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at $Proxy23.registerNodeManager(Unknown Source) at org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:68) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at $Proxy24.registerNodeManager(Unknown Source) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:238) at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:175) ... 6 more Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642) at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399) at org.apache.hadoop.ipc.Client.call(Client.java:1318)

hadoop集群启动后namenode自动关闭

2017-09-05 10:14:17,973 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode ON, in safe mode extension. The reported blocks 189 has reached the threshold 0.9990 of total blocks 189. The number of live datanodes 2 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 9 seconds. 2017-09-05 10:14:23,736 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.updateBlockForPipeline from 172.28.14.61:41497 Call#164039 Retry#12 org.apache.hadoop.ipc.RetriableException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot get a new generation stamp and an access token for block BP-1552766309-172.28.41.193-1503397713205:blk_1073742745_1926. Name node is in safe mode. The reported blocks 189 has reached the threshold 0.9990 of total blocks 189. The number of live datanodes 2 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 4 seconds. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1331) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkUCBlock(FSNamesystem.java:6234) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updateBlockForPipeline(FSNamesystem.java:6309) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updateBlockForPipeline(NameNodeRpcServer.java:806) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updateBlockForPipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:955) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043) Caused by: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot get a new generation stamp and an access token for block BP-1552766309-172.28.41.193-1503397713205:blk_1073742745_1926. Name node is in safe mode. The reported blocks 189 has reached the threshold 0.9990 of total blocks 189. The number of live datanodes 2 has reached the minimum number 0. In safe mode extension. Safe mode will be turned off automatically in 4 seconds. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkNameNodeSafeMode(FSNamesystem.java:1327) ... 13 more 2017-09-05 10:14:27,976 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: initializing replication queues 2017-09-05 10:14:27,977 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 55 secs 2017-09-05 10:14:27,977 INFO org.apache.hadoop.hdfs.StateChange: STATE* Safe mode is OFF 2017-09-05 10:14:27,977 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 1 racks and 2 datanodes 2017-09-05 10:14:27,977 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks 2017-09-05 10:14:28,013 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Total number of blocks = 190 2017-09-05 10:14:28,013 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of invalid blocks = 0 2017-09-05 10:14:28,013 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of under-replicated blocks = 3 2017-09-05 10:14:28,013 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of over-replicated blocks = 0 2017-09-05 10:14:28,013 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: Number of blocks being written = 1 2017-09-05 10:14:28,013 INFO org.apache.hadoop.hdfs.StateChange: STATE* Replication Queue initialization scan for invalid, over- and under-replicated blocks completed in 29 msec 2017-09-05 10:14:59,141 INFO org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager: topN size for command listStatus is: 0 2017-09-05 10:14:59,141 INFO org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager: topN size for command * is: 0 2017-09-05 10:14:59,143 INFO org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager: topN size for command listStatus is: 1 2017-09-05 10:14:59,145 INFO org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager: topN size for command * is: 1 2017-09-05 10:14:59,185 INFO org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager: topN size for command listStatus is: 1 2017-09-05 10:14:59,186 INFO org.apache.hadoop.hdfs.server.namenode.top.window.RollingWindowManager: topN size for command * is: 1 2017-09-05 10:16:50,848 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 172.28.41.196 2017-09-05 10:16:50,849 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs 2017-09-05 10:16:50,849 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 15839 2017-09-05 10:16:50,849 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 3 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 88 18 2017-09-05 10:16:50,883 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 3 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 120 20 2017-09-05 10:16:50,910 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /home/hadoop/hadoop_name/current/edits_inprogress_0000000000000015839 -> /home/hadoop/hadoop_name/current/edits_0000000000000015839-0000000000000015841 2017-09-05 10:16:50,915 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 15842 2017-09-05 10:18:51,193 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 172.28.41.196 2017-09-05 10:18:51,193 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs 2017-09-05 10:18:51,193 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 15842 2017-09-05 10:18:51,194 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 19 8 2017-09-05 10:18:51,372 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 129 76 2017-09-05 10:18:51,405 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /home/hadoop/hadoop_name/current/edits_inprogress_0000000000000015842 -> /home/hadoop/hadoop_name/current/edits_0000000000000015842-0000000000000015843 2017-09-05 10:18:51,406 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 15844 2017-09-05 10:20:52,122 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 172.28.41.196 2017-09-05 10:20:52,122 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs 2017-09-05 10:20:52,122 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 15844 2017-09-05 10:20:52,122 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 1 SyncTimes(ms): 39 341 2017-09-05 10:20:52,258 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 1 Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 103 413 2017-09-05 10:20:52,284 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /home/hadoop/hadoop_name/current/edits_inprogress_0000000000000015844 -> /home/hadoop/hadoop_name/current/edits_0000000000000015844-0000000000000015845 2017-09-05 10:20:52,284 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 15846 报这样的错误是不是有问题

hadoop执行wordcount报错

WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 17/05/03 02:35:31 WARN hdfs.DFSClient: DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989) 17/05/03 02:35:31 WARN hdfs.DFSClient: Error Recovery for block null bad datanode[0] nodes == null 17/05/03 02:35:31 WARN hdfs.DFSClient: Could not get block locations. Source file "/zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar" - Aborting... 17/05/03 02:35:31 INFO mapred.JobClient: Cleaning up the staging area hdfs://192.168.136.131:9000/zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001 17/05/03 02:35:31 ERROR security.UserGroupInformation: PriviledgedActionException as:root cause:org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989) 17/05/03 02:35:31 ERROR hdfs.DFSClient: Failed to close file /zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar org.apache.hadoop.ipc.RemoteException: java.io.IOException: File /zxc/hdfs/tmp/mapred/staging/root/.staging/job_201705030234_0001/job.jar could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1639) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at com.sun.proxy.$Proxy1.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989) 报错内容是没有节点,但是用jps查看都已经正常启动,而且防火墙也关闭了,节点间通信也都正常

hadoop jobtracker 和 tasktracker 没有启动

hadoop版本:2.6 hosts已经修改; java 环境变量:配置的/etc/environment中,如下图: ![图片说明](https://img-ask.csdn.net/upload/201504/03/1428033206_312570.png) core-site.xml配置如下图: ![图片说明](https://img-ask.csdn.net/upload/201504/03/1428033234_981229.png) mapred-site.xml配置如下图 ![图片说明](https://img-ask.csdn.net/upload/201504/03/1428033257_103800.png) hdfs-site.xml配置如下图: ![图片说明](https://img-ask.csdn.net/upload/201504/03/1428033251_620374.png) 启动脚本: 使用 start-all.sh启动hadoop 但是jps查看进程没有jobtracker和 tasktracker ![图片说明](https://img-ask.csdn.net/upload/201504/03/1428033376_905115.png) 大神指点啊,小弟初学找了很久了没解决问题。 就只有5c币,全部奉上了

关于mysql和hadoop数据交互的问题,和hadoop的文件夹设计

关于mysql和hadoop数据交互的问题,和hadoop的文件夹设计 关于mysql和hadoop数据交互的问题,和hadoop的文件夹设计 目前mysql按地区,商业区区分,假设读取mysql数据库的地区,根据地区划分 我昨天和领导沟通了,领导说点击率不是必要条件,地区划分才是重点,后面就是各方面劝导,只好以地区区分,关键是这个镇区区分数据和产品的话,全国有6k多个地区, 这样的hdfs文件夹数量,岂不是很崩溃, 后面感觉还有很多纬度和条件查询啥的,上头又喊了一句不一定要用hadoop,发挥hadoop啥啥优势,mysql多条件查询方便,好好做这个方案,然后我就一头雾水了,心里比较纠结和郁闷。最近搞mahout出了点成绩,发现单机版也挺好使(数据量少的前提下),搜了搜房子,北京也就519,059这么多套房子在待售,没啥必要用hadoop,自我感觉,如果只是单独分析这些条数据519,059条,感觉不大, 如果你有好的博客和资源提供,麻烦提供url连接,谢谢

hadoop wordcount报错192.168.79.172 to  :54895 拒绝连接

各位大神,我是hadoop新手,在自己电脑安装好hadoop后,运行wordcount报错,报错已困扰两天,希望有大神可以帮解决。 报错日志如下: [hadoop@hadoop0 mapreduce]$ hadoop jar hadoop-mapreduce-examples-2.7.1.jar wordcount /input /output 17/04/14 13:49:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/04/14 13:49:48 INFO client.RMProxy: Connecting to ResourceManager at hadoop0/192.168.79.172:8032 17/04/14 13:49:50 INFO input.FileInputFormat: Total input paths to process : 2 17/04/14 13:49:50 INFO mapreduce.JobSubmitter: number of splits:2 17/04/14 13:49:50 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1492077890345_0003 17/04/14 13:49:51 INFO impl.YarnClientImpl: Submitted application application_1492077890345_0003 17/04/14 13:49:51 INFO mapreduce.Job: The url to track the job: http://hadoop0:8088/proxy/application_1492077890345_0003/ 17/04/14 13:49:51 INFO mapreduce.Job: Running job: job_1492077890345_0003 17/04/14 13:50:12 INFO mapreduce.Job: Job job_1492077890345_0003 running in uber mode : false 17/04/14 13:50:12 INFO mapreduce.Job: map 0% reduce 0% 17/04/14 13:50:12 INFO mapreduce.Job: Job job_1492077890345_0003 failed with state FAILED due to: Application application_1492077890345_0003 failed 2 times due to Error launching appattempt_1492077890345_0003_000002. Got exception: java.net.ConnectException: Call From hadoop0/192.168.79.172 to  :54895 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1480) at org.apache.hadoop.ipc.Client.call(Client.java:1407) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy83.startContainers(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:119) at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:254) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.net.ConnectException: 拒绝连接 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:609) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:707) at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1529) at org.apache.hadoop.ipc.Client.call(Client.java:1446) ... 9 more . Failing the application. 17/04/14 13:50:12 INFO mapreduce.Job: Counters: 0 metrics文件内容如下: ![图片说明](https://img-ask.csdn.net/upload/201704/14/1492150334_860585.png) 另外,在报错日志中发下这么句话Call From hadoop0/192.168.79.172 to  :54895 failed ,为什么54895只有端口前面没有IP,是不是环境没有配置正确?

hadoop使用kerberos认证后,hadoop fs -ls命令行无法使用,求大神帮忙

hadoop版本apache hadoop 2.7.3,jdk-1.7 输入hadoop fs -ls,错误信息如下: hadoop@hadoop01 native]$ hadoop fs -ls 17/08/01 01:33:36 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)] ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "hadoop01/192.168.148.129"; destination host is: "hadoop01":9000; klist查看凭据缓存,是存在的: [hadoop@hadoop01 native]$ klist Ticket cache: KEYRING:persistent:1001:1001 Default principal: hadoop/hadoop01@HADOOP.COM Valid starting Expires Service principal 08/01/2017 01:12:54 08/02/2017 01:12:54 krbtgt/HADOOP.COM@HADOOP.COM 通过http://192.168.148.129:50070/dfshealth.html#tab-overview访问界面也是OK的: Configured Capacity: 55.38 GB DFS Used: 16 KB (0%) Non DFS Used: 11.4 GB DFS Remaining: 43.99 GB (79.42%) Block Pool Used: 16 KB (0%) DataNodes usages% (Min/Median/Max/stdDev): 0.00% / 0.00% / 0.00% / 0.00% Live Nodes 2 (Decommissioned: 0) Dead Nodes 0 (Decommissioned: 0) Decommissioning Nodes 0 Total Datanode Volume Failures 0 (0 B) Number of Under-Replicated Blocks 0 Number of Blocks Pending Deletion 0 Block Deletion Start Time 2017/8/1 上午10:12:21

hadoop 安装datanode无法链接namenode,,但是子节点 确实启动了 ?

masters日志: java.io.IOException: File /home/wangzhengqi/tmp/mapred/system/jobtracker.info could only be replicated to 0 nodes, instead of 1 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1920) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:783) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426) slaves日志:2015-04-19 11:19:45,137 INFO org.apache.hadoop.ipc.RPC: Server at ubuntu-2/192.168.128.129:9000 not available yet, Zzzzz... 查看任务: masters: root@ubuntu-2:/usr/hadoop/conf# jps 2897 TaskTracker 2386 DataNode 2177 NameNode 2603 SecondaryNameNode 2940 Jps 2684 JobTracker root@ubuntu-2:/usr/hadoop/conf# hadoop dfsadmin -report Configured Capacity: 20334034944 (18.94 GB) Present Capacity: 14065430543 (13.1 GB) DFS Remaining: 14065401856 (13.1 GB) DFS Used: 28687 (28.01 KB) DFS Used%: 0% Under replicated blocks: 1(这里应该是3台才对, 因为我还有两台子节点) Blocks with corrupt replicas: 0 Missing blocks: 0 slaves1:[root@centos-4 conf]# jps 1288 Jps 1163 DataNode 1225 TaskTracker slaves2:[root@centos-3 conf]# jps 1288 Jps 1163 DataNode 1225 TaskTracker start-all.sh后: 两台子节点的datanode都 已经启动了, 就是链接 不上namenode,, 这到底是怎么回事, 请大神解答!!!1

Hadoop 高可用HA配置出现的问题

求助, 在配置Hadoop HA高可用配置时,启动dfs过程无报错,但是启动后两台master中识别的active node不相同,不随机分享datanode,例如两台master,5台datanode。启动后会master1识别(3台datanode,随机),master2(识别3台datanode)。

网络节点向hadoop集群上传文件时,内网与外网IP出错

公司需要搭建hadoop环境收集网络上服务器节点的日志。但是hadoop搭建在微软云上,微软云的节点(namenode和datanode)之间用内网IP通信,但是网络商服务器节点上传需要用到外网IP。我本来想把hadoop的Java源码根据需求修改后,重新编译得到新包,再到微软云的虚拟机去配置。但是没法得到NAT的映射表,也就是内网IP没法在代码里映射为外网IP。有什么其他的解决方法吗?

Namenode启动时,每个端口都去认证一次,且认证失败

HDNamenode在启动时,发现每个端口都去认证了一次,使用的用户名为null,而且认证失败了。 但是整个HDFS的功能又是正常的,这是什么原因啊。 2019-01-11 00:27:08,407 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:49972:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-01-11 00:27:10,612 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:49984:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-01-11 00:27:11,651 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:49987:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-01-11 00:27:11,653 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:49988:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-01-11 00:27:15,495 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:50006:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-01-11 00:27:16,230 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:50058:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-01-11 00:27:17,101 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:50094:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-01-11 00:27:21,266 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:50118:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-01-11 00:27:25,949 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:50135:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-01-11 00:27:25,952 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:50136:null (GSS initiate failed) with true cause: (GSS initiate failed) 2019-01-11 00:27:27,639 WARN SecurityLogger.org.apache.hadoop.ipc.Server: Auth failed for 172.28.128.61:50216:null (GSS initiate failed) with true cause: (GSS initiate failed)

4小时玩转微信小程序——基础入门与微信支付实战

这是一个门针对零基础学员学习微信小程序开发的视频教学课程。课程采用腾讯官方文档作为教程的唯一技术资料来源。杜绝网络上质量良莠不齐的资料给学员学习带来的障碍。 视频课程按照开发工具的下载、安装、使用、程序结构、视图层、逻辑层、微信小程序等几个部分组织课程,详细讲解整个小程序的开发过程

Python可以这样学(第四季:数据分析与科学计算可视化)

董付国老师系列教材《Python程序设计(第2版)》(ISBN:9787302436515)、《Python可以这样学》(ISBN:9787302456469)配套视频,在教材基础上又增加了大量内容,通过实例讲解numpy、scipy、pandas、statistics、matplotlib等标准库和扩展库用法。

组成原理课程设计(实现机器数的真值还原等功能)

实现机器数的真值还原(定点小数)、定点小数的单符号位补码加减运算、定点小数的补码乘法运算和浮点数的加减运算。

javaWeb图书馆管理系统源码mysql版本

系统介绍 图书馆管理系统主要的目的是实现图书馆的信息化管理。图书馆的主要业务就是新书的借阅和归还,因此系统最核心的功能便是实现图书的借阅和归还。此外,还需要提供图书的信息查询、读者图书借阅情况的查询等

土豆浏览器

土豆浏览器可以用来看各种搞笑、电影、电视剧视频

Java面试题大全(2020版)

发现网上很多Java面试题都没有答案,所以花了很长时间搜集整理出来了这套Java面试题大全,希望对大家有帮助哈~ 本套Java面试题大全,全的不能再全,哈哈~ 一、Java 基础 1. JDK 和 JRE 有什么区别? JDK:Java Development Kit 的简称,java 开发工具包,提供了 java 的开发环境和运行环境。 JRE:Java Runtime Environ...

Java8零基础入门视频教程

这门课程基于主流的java8平台,由浅入深的详细讲解了java SE的开发技术,可以使java方向的入门学员,快速扎实的掌握java开发技术!

Java基础知识面试题(2020最新版)

文章目录Java概述何为编程什么是Javajdk1.5之后的三大版本JVM、JRE和JDK的关系什么是跨平台性?原理是什么Java语言有哪些特点什么是字节码?采用字节码的最大好处是什么什么是Java程序的主类?应用程序和小程序的主类有何不同?Java应用程序与小程序之间有那些差别?Java和C++的区别Oracle JDK 和 OpenJDK 的对比基础语法数据类型Java有哪些数据类型switc...

TTP229触摸代码以及触摸返回值处理

自己总结的ttp229触摸代码,触摸代码以及触摸按键处理

网络工程师小白入门--【思科CCNA、华为HCNA等网络工程师认证】

本课程适合CCNA或HCNA网络小白同志,高手请绕道,可以直接学习进价课程。通过本预科课程的学习,为学习网络工程师、思科CCNA、华为HCNA这些认证打下坚实的基础! 重要!思科认证2020年2月24日起,已启用新版认证和考试,包括题库都会更新,由于疫情原因,请关注官网和本地考点信息。题库网络上很容易下载到。

深度学习原理+项目实战+算法详解+主流框架(套餐)

深度学习系列课程从深度学习基础知识点开始讲解一步步进入神经网络的世界再到卷积和递归神经网络,详解各大经典网络架构。实战部分选择当下最火爆深度学习框架PyTorch与Tensorflow/Keras,全程实战演示框架核心使用与建模方法。项目实战部分选择计算机视觉与自然语言处理领域经典项目,从零开始详解算法原理,debug模式逐行代码解读。适合准备就业和转行的同学们加入学习! 建议按照下列课程顺序来进行学习 (1)掌握深度学习必备经典网络架构 (2)深度框架实战方法 (3)计算机视觉与自然语言处理项目实战。(按照课程排列顺序即可)

java jdk 8 帮助文档 中文 文档 chm 谷歌翻译

JDK1.8 API 中文谷歌翻译版 java帮助文档 JDK API java 帮助文档 谷歌翻译 JDK1.8 API 中文 谷歌翻译版 java帮助文档 Java最新帮助文档 本帮助文档是使用谷

Ubuntu18.04安装教程

Ubuntu18.04.1安装一、准备工作1.下载Ubuntu18.04.1 LTS2.制作U盘启动盘3.准备 Ubuntu18.04.1 的硬盘空间二、安装Ubuntu18.04.1三、安装后的一些工作1.安装输入法2.更换软件源四、双系统如何卸载Ubuntu18.04.1新的改变功能快捷键合理的创建标题,有助于目录的生成如何改变文本的样式插入链接与图片如何插入一段漂亮的代码片生成一个适合你的列...

快速排序---(面试碰到过好几次)

原理:    快速排序,说白了就是给基准数据找其正确索引位置的过程.    如下图所示,假设最开始的基准数据为数组第一个元素23,则首先用一个临时变量去存储基准数据,即tmp=23;然后分别从数组的两端扫描数组,设两个指示标志:low指向起始位置,high指向末尾.    首先从后半部分开始,如果扫描到的值大于基准数据就让high减1,如果发现有元素比该基准数据的值小(如上图中18&amp;lt...

手把手实现Java图书管理系统(附源码)

【超实用课程内容】 本课程演示的是一套基于Java的SSM框架实现的图书管理系统,主要针对计算机相关专业的正在做毕设的学生与需要项目实战练习的java人群。详细介绍了图书管理系统的实现,包括:环境搭建、系统业务、技术实现、项目运行、功能演示、系统扩展等,以通俗易懂的方式,手把手的带你从零开始运行本套图书管理系统,该项目附带全部源码可作为毕设使用。 【课程如何观看?】 PC端:https://edu.csdn.net/course/detail/27513 移动端:CSDN 学院APP(注意不是CSDN APP哦) 本课程为录播课,课程2年有效观看时长,大家可以抓紧时间学习后一起讨论哦~ 【学员专享增值服务】 源码开放 课件、课程案例代码完全开放给你,你可以根据所学知识,自行修改、优化 下载方式:电脑登录https://edu.csdn.net/course/detail/27513,点击右下方课程资料、代码、课件等打包下载

HTML期末大作业

这是我自己做的HTML期末大作业,花了很多时间,稍加修改就可以作为自己的作业了,而且也可以作为学习参考

Python数据挖掘简易入门

&nbsp; &nbsp; &nbsp; &nbsp; 本课程为Python数据挖掘方向的入门课程,课程主要以真实数据为基础,详细介绍数据挖掘入门的流程和使用Python实现pandas与numpy在数据挖掘方向的运用,并深入学习如何运用scikit-learn调用常用的数据挖掘算法解决数据挖掘问题,为进一步深入学习数据挖掘打下扎实的基础。

极简JAVA学习营第四期(报名以后加助教微信:eduxy-1)

想学好JAVA必须要报两万的培训班吗? Java大神勿入 如果你: 零基础想学JAVA却不知道从何入手 看了一堆书和视频却还是连JAVA的环境都搭建不起来 囊中羞涩面对两万起的JAVA培训班不忍直视 在职没有每天大块的时间专门学习JAVA 那么恭喜你找到组织了,在这里有: 1. 一群志同道合立志学好JAVA的同学一起学习讨论JAVA 2. 灵活机动的学习时间完成特定学习任务+每日编程实战练习 3. 热心助人的助教和讲师及时帮你解决问题,不按时完成作业小心助教老师的家访哦 上一张图看看前辈的感悟: &nbsp; &nbsp; 大家一定迫不及待想知道什么是极简JAVA学习营了吧,下面就来给大家说道说道: 什么是极简JAVA学习营? 1. 针对Java小白或者初级Java学习者; 2. 利用9天时间,每天1个小时时间; 3.通过 每日作业 / 组队PK / 助教答疑 / 实战编程 / 项目答辩 / 社群讨论 / 趣味知识抢答等方式让学员爱上学习编程 , 最终实现能独立开发一个基于控制台的‘库存管理系统’ 的学习模式 极简JAVA学习营是怎么学习的? &nbsp; 如何报名? 只要购买了极简JAVA一:JAVA入门就算报名成功! &nbsp;本期为第四期极简JAVA学习营,我们来看看往期学员的学习状态: 作业看这里~ &nbsp; 助教的作业报告是不是很专业 不交作业打屁屁 助教答疑是不是很用心 &nbsp; 有奖抢答大家玩的很嗨啊 &nbsp; &nbsp; 项目答辩终于开始啦 &nbsp; 优秀者的获奖感言 &nbsp; 这是答辩项目的效果 &nbsp; &nbsp; 这么细致的服务,这么好的氛围,这样的学习效果,需要多少钱呢? 不要1999,不要199,不要99,只要9.9 是的你没听错,只要9.9以上所有就都属于你了 如果你: 1、&nbsp;想学JAVA没有基础 2、&nbsp;想学JAVA没有整块的时间 3、&nbsp;想学JAVA没有足够的预算 还等什么?赶紧报名吧,抓紧抢位,本期只招300人,错过只有等时间待定的下一期了 &nbsp; 报名请加小助手微信:eduxy-1 &nbsp; &nbsp;

C++语言基础视频教程

C++语言基础视频培训课程:本课与主讲者在大学开出的程序设计课程直接对接,准确把握知识点,注重教学视频与实践体系的结合,帮助初学者有效学习。本教程详细介绍C++语言中的封装、数据隐藏、继承、多态的实现等入门知识;主要包括类的声明、对象定义、构造函数和析构函数、运算符重载、继承和派生、多态性实现等。 课程需要有C语言程序设计的基础(可以利用本人开出的《C语言与程序设计》系列课学习)。学习者能够通过实践的方式,学会利用C++语言解决问题,具备进一步学习利用C++开发应用程序的基础。

UnityLicence

UnityLicence

软件测试2小时入门

本课程内容系统、全面、简洁、通俗易懂,通过2个多小时的介绍,让大家对软件测试有个系统的理解和认识,具备基本的软件测试理论基础。 主要内容分为5个部分: 1 软件测试概述,了解测试是什么、测试的对象、原则、流程、方法、模型;&nbsp; 2.常用的黑盒测试用例设计方法及示例演示;&nbsp; 3 常用白盒测试用例设计方法及示例演示;&nbsp; 4.自动化测试优缺点、使用范围及示例‘;&nbsp; 5.测试经验谈。

YOLOv3目标检测实战:训练自己的数据集

YOLOv3是一种基于深度学习的端到端实时目标检测方法,以速度快见长。本课程将手把手地教大家使用labelImg标注和使用YOLOv3训练自己的数据集。课程分为三个小项目:足球目标检测(单目标检测)、梅西目标检测(单目标检测)、足球和梅西同时目标检测(两目标检测)。 本课程的YOLOv3使用Darknet,在Ubuntu系统上做项目演示。包括:安装Darknet、给自己的数据集打标签、整理自己的数据集、修改配置文件、训练自己的数据集、测试训练出的网络模型、性能统计(mAP计算和画出PR曲线)和先验框聚类。 Darknet是使用C语言实现的轻型开源深度学习框架,依赖少,可移植性好,值得深入探究。 除本课程《YOLOv3目标检测实战:训练自己的数据集》外,本人推出了有关YOLOv3目标检测的系列课程,请持续关注该系列的其它课程视频,包括: 《YOLOv3目标检测实战:交通标志识别》 《YOLOv3目标检测:原理与源码解析》 《YOLOv3目标检测:网络模型改进方法》 敬请关注并选择学习!

Python数据分析师-实战系列

系列课程主要包括Python数据分析必备工具包,数据分析案例实战,核心算法实战与企业级数据分析与建模解决方案实战,建议大家按照系列课程阶段顺序进行学习。所有数据集均为企业收集的真实数据集,整体风格以实战为导向,通俗讲解Python数据分析核心技巧与实战解决方案。

YOLOv3目标检测实战系列课程

《YOLOv3目标检测实战系列课程》旨在帮助大家掌握YOLOv3目标检测的训练、原理、源码与网络模型改进方法。 本课程的YOLOv3使用原作darknet(c语言编写),在Ubuntu系统上做项目演示。 本系列课程包括三门课: (1)《YOLOv3目标检测实战:训练自己的数据集》 包括:安装darknet、给自己的数据集打标签、整理自己的数据集、修改配置文件、训练自己的数据集、测试训练出的网络模型、性能统计(mAP计算和画出PR曲线)和先验框聚类。 (2)《YOLOv3目标检测:原理与源码解析》讲解YOLOv1、YOLOv2、YOLOv3的原理、程序流程并解析各层的源码。 (3)《YOLOv3目标检测:网络模型改进方法》讲解YOLOv3的改进方法,包括改进1:不显示指定类别目标的方法 (增加功能) ;改进2:合并BN层到卷积层 (加快推理速度) ; 改进3:使用GIoU指标和损失函数 (提高检测精度) ;改进4:tiny YOLOv3 (简化网络模型)并介绍 AlexeyAB/darknet项目。

超详细MySQL安装及基本使用教程

一、下载MySQL 首先,去数据库的官网http://www.mysql.com下载MySQL。 点击进入后的首页如下:  然后点击downloads,community,选择MySQL Community Server。如下图:  滑到下面,找到Recommended Download,然后点击go to download page。如下图:  点击download进入下载页面选择No...

一学即懂的计算机视觉(第一季)

图像处理和计算机视觉的课程大家已经看过很多,但总有“听不透”,“用不了”的感觉。课程致力于创建人人都能听的懂的计算机视觉,通过生动、细腻的讲解配合实战演练,让学生真正学懂、用会。 【超实用课程内容】 课程内容分为三篇,包括视觉系统构成,图像处理基础,特征提取与描述,运动跟踪,位姿估计,三维重构等内容。课程理论与实战结合,注重教学内容的可视化和工程实践,为人工智能视觉研发及算法工程师等相关高薪职位就业打下坚实基础。 【课程如何观看?】 PC端:https://edu.csdn.net/course/detail/26281 移动端:CSDN 学院APP(注意不是CSDN APP哦) 本课程为录播课,课程2年有效观看时长,但是大家可以抓紧时间学习后一起讨论哦~ 【学员专享增值服务】 源码开放 课件、课程案例代码完全开放给你,你可以根据所学知识,自行修改、优化 下载方式:电脑登录https://edu.csdn.net/course/detail/26281,点击右下方课程资料、代码、课件等打包下载

董付国老师Python全栈学习优惠套餐

购买套餐的朋友可以关注微信公众号“Python小屋”,上传付款截图,然后领取董老师任意图书1本。

爬取妹子图片(简单入门)

安装第三方请求库 requests 被网站禁止了访问 原因是我们是Python过来的 重新给一段 可能还是存在用不了,使用网页的 编写代码 上面注意看匹配内容 User-Agent:请求对象 AppleWebKit:请求内核 Chrome浏览器 //请求网页 import requests import re //正则表达式 就是去不规则的网页里面提取有规律的信息 headers = { 'User-Agent':'存放浏览器里面的' } response = requests.get

web网页制作期末大作业

分享思维,改变世界. web网页制作,期末大作业. 所用技术:html css javascript 分享所学所得

技术大佬:我去,你写的 switch 语句也太老土了吧

昨天早上通过远程的方式 review 了两名新来同事的代码,大部分代码都写得很漂亮,严谨的同时注释也很到位,这令我非常满意。但当我看到他们当中有一个人写的 switch 语句时,还是忍不住破口大骂:“我擦,小王,你丫写的 switch 语句也太老土了吧!” 来看看小王写的代码吧,看完不要骂我装逼啊。 private static String createPlayer(PlayerTypes p...

相关热词 c# cad插入影像 c#设计思想 c#正则表达式 转换 c#form复制 c#写web c# 柱形图 c# wcf 服务库 c#应用程序管理器 c#数组如何赋值给数组 c#序列化应用目的博客园
立即提问