hadoop hdfs 挂载报错

已采纳

hadoop hdfs挂载在linux上市报错fuse-dfs didn't recognize /hadoop/hdfs,-2，然后打开/hadoop/hdfs时报输入输出目录错误

点赞
写回答
关注问题
收藏
复制链接分享
邀请回答

1条回答

采纳
码龄粉丝数

被点赞

被采纳

采纳率

私信
从今以后生命中的每一秒都属于我爱的人 6年前
http://blog.csdn.net/knowledgeaaa/article/details/25896581

点赞评论复制链接分享

提交

相关推荐

hadoop hdfs 挂载报错
6年前

回答 1 已采纳 hadoop hdfs挂载在linux上市报错fuse-dfs didn't recognize /hadoop/hdfs,-2，然后打开/hadoop/hdfs时报输入输出目录错误
安装Hadoop环境的时候，输入hadoop -version报错有问必答 java
2月前

回答 2 已采纳
在eclipse运行hadoop mapreduce例子报错 eclipse
4年前

回答 1 已采纳在终端运行hadoop带的例子正常，hadoop节点正常，错误如下 17/09/05 20:20:16 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/09/05 20:20:16 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id 17/09/05 20:20:16 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= Exception in thread "main" java.net.ConnectException: Call From master/192.168.1.110 to localhost:9000 failed on connection exception: java.net.ConnectException: 拒绝连接; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:792) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:732) at org.apache.hadoop.ipc.Client.call(Client.java:1479) at org.apache.hadoop.ipc.Client.call(Client.java:1412) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy9.getFileInfo(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getFileInfo(ClientNamenodeProtocolTranslatorPB.java:707) at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:1785) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1068) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1426) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:145) at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:266) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:139) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308) at mapreduce.Temperature.main(Temperature.java:202) Caused by: java.net.ConnectException: 拒绝连接 at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531) at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495) at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528) at org.apache.hadoop.ipc.Client.call(Client.java:1451) ... 28 more
nfs挂载hdfs报错：mount.nfs3 mount system call failed
5年前

回答 2 已采纳作为一只菜鸟，为了能达到hdfs映射到linux本地文件系统中的目的，参照apache官网的文档一步一步操作下来[https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html](https://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html "") 中间都没有遇到什么问题 nfs3和portmap都启动了，防火墙也关掉了 ``` [root@Master hadoop]# rpcinfo -p Master.dev program vers proto port service 100005 3 udp 4242 mountd 100005 1 tcp 4242 mountd 100000 2 udp 111 portmapper 100000 2 tcp 111 portmapper 100005 3 tcp 4242 mountd 100005 2 tcp 4242 mountd 100003 3 tcp 2049 nfs 100005 2 udp 4242 mountd 100005 1 udp 4242 mountd ``` ``` [root@Master hadoop]# showmount -e Master.dev Export list for Master.dev: / * ``` 执行到最后一步的时候就报错mount.nfs3 mount system call failed了 ``` mount -t nfs -o nfsvers=3,vers=3,proto=tcp,nolock Master.dev:/ /data/hdfs/ ``` 已经琢磨了两天了。。还是解决不了。。。求各位指点呀T.T
新手入门，HDFS获取文件系统是报错
3年前

回答 1 已采纳 **这是代码** ``` //获取文件系统 @Test public void getFileSystem() throws Exception{ //0创建配置信息对象 Configuration configuration = new Configuration(); //1获取文件系统 FileSystem fs = FileSystem.get(new URI("hffs://hadoop102:8020"), configuration, "root"); //2打印文件系统 System.out.println(fs.toString()); } ``` **报错内容：** ``` java.io.IOException: No FileSystem for scheme: hffs at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2644) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:160) at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:157) at com.root.hdfs.HDFSClient.getFileSystem(HDFSClient.java:40) ```
Hadoop Distcp报错队列问题
4年前

回答 1 已采纳 ``` sudo -uxiaosi hadoop distcp hdfs:///user/xiaosi/tmp/data_group/histories/day=20161116 hdfs:///user/xiaosi/data_group/histories ``` 报错： ``` 17/01/17 19:18:46 ERROR security.UserGroupInformation: PriviledgedActionException as:xiaosi (auth:SIMPLE) cause:java.io.IOException: Failed to run job : User xiaosi cannot submit applications to queue root.default 17/01/17 19:18:46 ERROR tools.DistCp: Exception encountered java.io.IOException: Failed to run job : User xiaosi cannot submit applications to queue root.default at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:299) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:430) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265) at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153) at org.apache.hadoop.tools.DistCp.run(DistCp.java:118) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.tools.DistCp.main(DistCp.java:375) ```
Hadoop搭建环境报错，实在找不到方法 linux
6年前

回答 2 已采纳找了很久都不知道解决方法，大家帮帮我吧，谢谢~ STARTUP_MSG: java = 1.6.0_45 ************************************************************/ 15/11/28 08:52:23 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 15/11/28 08:52:23 INFO namenode.NameNode: createNameNode [-format] 15/11/28 08:52:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Formatting using clusterid: CID-0f5cc549-bfbb-45b2-a735-c3500a2e83cf 15/11/28 08:52:25 INFO namenode.FSNamesystem: fsLock is fair:true 15/11/28 08:52:25 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 15/11/28 08:52:25 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 15/11/28 08:52:25 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 15/11/28 08:52:25 INFO blockmanagement.BlockManager: The block deletion will start around 2015 Nov 28 08:52:25 15/11/28 08:52:25 INFO util.GSet: Computing capacity for map BlocksMap 15/11/28 08:52:25 INFO util.GSet: VM type = 32-bit 15/11/28 08:52:25 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB 15/11/28 08:52:25 INFO util.GSet: capacity = 2^22 = 4194304 entries 15/11/28 08:52:25 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 15/11/28 08:52:25 INFO blockmanagement.BlockManager: defaultReplication = 1 15/11/28 08:52:25 INFO blockmanagement.BlockManager: maxReplication = 512 15/11/28 08:52:25 INFO blockmanagement.BlockManager: minReplication = 1 15/11/28 08:52:25 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 15/11/28 08:52:25 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false 15/11/28 08:52:25 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 15/11/28 08:52:25 INFO blockmanagement.BlockManager: encryptDataTransfer = false 15/11/28 08:52:25 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 15/11/28 08:52:25 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE) 15/11/28 08:52:25 INFO namenode.FSNamesystem: supergroup = supergroup 15/11/28 08:52:25 INFO namenode.FSNamesystem: isPermissionEnabled = false 15/11/28 08:52:25 INFO namenode.FSNamesystem: HA Enabled: false 15/11/28 08:52:25 INFO namenode.FSNamesystem: Append Enabled: true 15/11/28 08:52:26 INFO util.GSet: Computing capacity for map INodeMap 15/11/28 08:52:26 INFO util.GSet: VM type = 32-bit 15/11/28 08:52:26 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB 15/11/28 08:52:26 INFO util.GSet: capacity = 2^21 = 2097152 entries 15/11/28 08:52:26 INFO namenode.NameNode: Caching file names occuring more than 10 times 15/11/28 08:52:26 INFO util.GSet: Computing capacity for map cachedBlocks 15/11/28 08:52:26 INFO util.GSet: VM type = 32-bit 15/11/28 08:52:26 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB 15/11/28 08:52:26 INFO util.GSet: capacity = 2^19 = 524288 entries 15/11/28 08:52:26 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 15/11/28 08:52:26 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 15/11/28 08:52:26 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 15/11/28 08:52:26 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 15/11/28 08:52:26 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 15/11/28 08:52:26 INFO util.GSet: Computing capacity for map NameNodeRetryCache 15/11/28 08:52:26 INFO util.GSet: VM type = 32-bit 15/11/28 08:52:26 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB 15/11/28 08:52:26 INFO util.GSet: capacity = 2^16 = 65536 entries 15/11/28 08:52:26 INFO namenode.NNConf: ACLs enabled? false 15/11/28 08:52:26 INFO namenode.NNConf: XAttrs enabled? true 15/11/28 08:52:26 INFO namenode.NNConf: Maximum size of an xattr: 16384 15/11/28 08:52:26 FATAL namenode.NameNode: Exception in namenode join java.lang.IllegalArgumentException: URI has an authority component at java.io.File.(File.java:368) at org.apache.hadoop.hdfs.server.namenode.NNStorage.getStorageDirectory(NNStorage.java:327) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:261) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournalsForWrite(FSEditLog.java:233) at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:920) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1354) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473) 15/11/28 08:52:26 INFO util.ExitUtil: Exiting with status 1 15/11/28 08:52:26 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at hadoop/192.168.131.2 ************************************************************/
window下连接hadoop集群报错
6年前

回答 2 已采纳 window下连接hadoop集群报错，已经把hadoop.dll放在window下的hadoop的bin目录了，system32也放了，还是无效，请问怎么办？？![图片说明](https://img-ask.csdn.net/upload/201508/31/1441030623_267379.png)
hadoop下的存储在hdfs中的视频文件怎么播放？
6年前

回答 2 已采纳 hadoop下的存储在hdfs中的视频文件怎么播放？可以直接拿到视频文件的http路径吗，我现在想实现的是像百度云这样的可以在线播放，但是不知道怎么得到文件的路径，而hdfs提供的API只有上传下载文件这些，对于这样的问题要怎么解决呢？？
新手，hadoop上运行wordcount程序报错
4年前

回答 1 已采纳运行的环境是：Ubuntu14.04+hadoop2.6.1 用的是virtualBox虚拟机，然后安装了一个master和三个slave节点 hadoop是可以成功启动的，没有任何问题在Ubuntu安装了eclipse，用java写了word count的程序，源码如下： ``` package wordcount; import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; /** * @author * @version 创建时间：2017年9月9日上午8:50:51 类说明 */ public class Wordcount { public static class TokenizerMapper extends Mapper { private final static IntWritable one = new IntWritable(1); private Text word = new Text(); protected void map(LongWritable key, Text value, Mapper.Context context) throws IOException, InterruptedException { StringTokenizer line = new StringTokenizer(value.toString()); while (line.hasMoreTokens()) { word.set(line.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer { private IntWritable result = new IntWritable(); protected void reduce(Text key, Iterable values, Reducer.Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable obj : values) { sum += obj.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = Job.getInstance(conf, "word count"); job.setJarByClass(Wordcount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path("hdfs://master:9000/user/hduser/demo/test.txt")); FileOutputFormat.setOutputPath(job, new Path("hdfs://master:9000/user/hduser/demo/wordcount")); //FileInputFormat.addInputPath(job, new Path(args[0])); //FileOutputFormat.setOutputPath(job, new Path(args[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } } ``` 启动hadoop后，在eclipse中直接运行上面的程序，运行成功，生成了wordcount文件夹，里面有_SUCCESS文件，也有统计的结果文件然后我想把程序打包成jar文件来运行，先把上面程序中的： ``` FileInputFormat.addInputPath(job, new Path("hdfs://master:9000/user/hduser/demo/test.txt")); FileOutputFormat.setOutputPath(job, new Path("hdfs://master:9000/user/hduser/demo/wordcount")); ``` 改成如下： ``` FileInputFormat.addInputPath(job, new Path(args[0])); FileOutputFormat.setOutputPath(job, new Path(args[1])); ``` 就是通过终端输入这两个参数用eclipse的export打包成jar文件，后在终端输入： ``` hadoop jar wordcount.jar wordcount.Wordcount hdfs://master:9000/user/hduser/demo/test.txt hdfs://master:9000/user/hduser/demo/wordcount ``` 运行就报错了，报错情况如下： ``` 17/09/09 11:18:53 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.56.100:8050 17/09/09 11:18:54 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 17/09/09 11:18:55 INFO input.FileInputFormat: Total input paths to process : 1 17/09/09 11:18:55 INFO mapreduce.JobSubmitter: number of splits:1 17/09/09 11:18:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504926710828_0001 17/09/09 11:18:56 INFO impl.YarnClientImpl: Submitted application application_1504926710828_0001 17/09/09 11:18:56 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1504926710828_0001/ 17/09/09 11:18:56 INFO mapreduce.Job: Running job: job_1504926710828_0001 17/09/09 11:19:14 INFO mapreduce.Job: Job job_1504926710828_0001 running in uber mode : false 17/09/09 11:19:14 INFO mapreduce.Job: map 0% reduce 0% 17/09/09 11:19:14 INFO mapreduce.Job: Job job_1504926710828_0001 failed with state FAILED due to: Application application_1504926710828_0001 failed 2 times due to AM Container for appattempt_1504926710828_0001_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://master:8088/proxy/application_1504926710828_0001/Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_1504926710828_0001_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) at java.lang.Thread.run(Thread.java:748) Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. 17/09/09 11:19:14 INFO mapreduce.Job: Counters: 0 ``` 去查了下日志文件， ``` 2017-09-09 11:18:55,869 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /tmp/hadoop-yarn/staging/hduser/.staging/job_1504926710828_0001/job.xml is closed by DFSClient_NONMAPREDUCE_-1306163227_1 2017-09-09 11:18:59,502 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-09-09 11:18:59,503 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 1 millisecond(s). 2017-09-09 11:19:12,241 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 192.168.56.102:53610 Call#7 Retry#0: java.io.FileNotFoundException: File does not exist: /tmp/hadoop-yarn/staging/hduser/.staging/job_1504926710828_0001/job_1504926710828_0001_1.jhist 2017-09-09 11:19:12,293 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9000, call org.apache.hadoop.hdfs.protocol.ClientProtocol.getBlockLocations from 192.168.56.102:53610 Call#8 Retry#0: java.io.FileNotFoundException: File does not exist: /tmp/hadoop-yarn/staging/hduser/.staging/job_1504926710828_0001/job_1504926710828_0001_1.jhist 2017-09-09 11:19:29,502 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30000 milliseconds 2017-09-09 11:19:29,502 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 2017-09-09 11:19:42,634 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.56.100 2017-09-09 11:19:42,634 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs 2017-09-09 11:19:42,634 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 29 2017-09-09 11:19:42,635 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 40 Total time for transactions(ms): 6 Number of transactions batched in Syncs: 0 Number of syncs: 27 SyncTimes(ms): 545 2017-09-09 11:19:42,704 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 40 Total time for transactions(ms): 6 Number of transactions batched in Syncs: 0 Number of syncs: 28 SyncTimes(ms): 613 2017-09-09 11:19:42,704 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /usr/local/hadoop/hadoop_data/hdfs/namenode/current/edits_inprogress_0000000000000000029 -> /usr/local/hadoop/hadoop_data/hdfs/namenode/current/edits_0000000000000000029-0000000000000000068 2017-09-09 11:19:42,704 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 69 2017-09-09 11:19:59,503 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-09-09 11:19:59,503 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 2017-09-09 11:20:29,504 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Rescanning after 30001 milliseconds 2017-09-09 11:20:29,504 INFO org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned 0 directive(s) and 0 block(s) in 0 millisecond(s). 2017-09-09 11:20:42,759 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from 192.168.56.100 2017-09-09 11:20:42,759 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Rolling edit logs 2017-09-09 11:20:42,759 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 69 2017-09-09 11:20:42,759 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of syncs: 2 SyncTimes(ms): 24 2017-09-09 11:20:42,791 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 0 Number of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 56 ``` 在这里面报了一个错误： ``` java.io.FileNotFoundException: File does not exist: /tmp/hadoop-yarn/staging/hduser/.staging/job_1504926710828_0001/job_1504926710828_0001_1.jhist ``` 新手，不知道怎么办了
ambari hdfs 启动报错_HDFS这些问题你会处理吗？
4月前

九子随笔的博客列举一些在维护hdfs工作中遇到的问题，有的是血的教训，有的是花了不少功夫定位，也有的是一些知识点或者技巧，其中有两个补丁已经合并到apache hadoop官方。最后根据这些问题处理经验，汇总了hadoop hdfs集群需要...
ambari hdfs 启动报错_吐血整理 | HDFS运维问题大全
4月前

今天喝到喜茶了么的博客 | 本文转载自“腾讯游戏存储与计算技术”，作者：周晓，腾讯游戏 CROS 体系高级工程师，负责多套HDFS集群的维护管理，并为Apache Hadoop社区提交过2个Patch。同时也作为游戏DBA，稳定支撑包括穿越火线、天天酷跑等...
ambari hdfs 启动报错_HDFS 运维常见问题处理
4月前

我和这个世界的博客腾讯游戏CROS体系的DBA维护着多套互娱数据平台的核心HDFS集群，积累了不少维护经验。1. 定期block全盘扫描，引起dn心跳超时而脱离集群hdfs有一个目录扫描机制，默认6小时会全盘扫描一次所有block，判断与内存里的那...
ambari hdfs 启动报错_[问题解决篇-11]因磁盘挂导致datanode启动失败
4月前

weixin_39636176的博客问题背景数据节点的datanode的配置中hdfs-site.xml 有一个dfs.data.dir 配置参数，这个配置是配置datanode的数据存储的磁盘目录,这个下面的配置使用逗号（，）来分隔每个磁盘，比如下面的配置中就说明，配置了2个...
hadoop hdfs空间满后重启不了
7年前

很青的青蛙的博客服务器检查的时候，发现存在HDFS上的文件无法同步，再发现hadoop停掉了。进行重启，重启不成功。经过一系列的操作，终于把hadoop集群重启并恢复数据。
Hadoop-HDFS学习
2年前

迷路剑客的博客 Hadoop由HDFS、MapReduce、HBase、Hive和ZooKeeper等成员组成，其中最基础最重要元素为底层用于存储集群中所有存储节点文件的文件系统HDFS（Hadoop Distributed File System）来执行MapReduce程序的MapReduce引...
hadoop新特性通过NFSv3挂载HDFS到本地目录
5年前

小小程序员1986的博客原理NFS允许用户像访问本地文件系统一样访问远程文件系统，而将NFS引入HDFS后，用户可像读写本地文件一样读写HDFS上的文件，大大简化了HDFS使用，这是通过引入一个NFS gateway服务实现的，该服务能将NFS协议转换为...
Hadoop常见错误之HDFS
6年前

邹中凡的博客下面小编整理在工作中遇到的问题...在启动Datanode时，/var/log/hadoop-hdfs/hadoop-cmf-hdfs-DATANODE-cdh-node12.grc.log.out日志文件中有如下错误：Caught exception while scanning /data/dfs/dn/current. Will thr
hadoop多硬盘挂载
4年前

weixin_33937499的博客查找发现原因：data1和data2所挂载的硬盘的根目录下有lost+found这个文件夹，而这个文件的权限是root，导致在hadoop用户下启动的hadoop在扫描配置文件中配置的data目录时，发现有文件hadoop用户没有写权限，就报错，...
HDFS多硬盘挂载
7年前

陈振阳的博客环境说明 9个节点，7个DataNode，每个DataNode有3块硬盘，两块是后来加的，挂载在系统所在文件系统。 hdfs-site.xml配置 ... /home/hadoop/develop/hadoop/hdfs/data, /home/hadoop/develop