hdfs append追加文件上传的问题

各位大神们,小弟设置了hadoop-0.20-cdh3u0版本的dfs.support.append为true,然后想测试下文件追加上传。第一次上传一个文件的前4096看,第二次上传其余部分。但是发现2次上传的文件大小之和小于总文件大小。我发现问题在于,当第二次上传时,hdfs会删除第一次的文件,然后重新建立新文件再上传,所以文件大小只有第二次上传的数据大小。
以下是hadoop的日志:
2012-07-10 15:00:04,363 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=dell ip=/172.18.9.55 cmd=[color=red]create [/color]src=/user/tmp/test.jpg dst=null perm=dell:supergroup:rw-r--r--
2012-07-10 15:00:04,373 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /user/tmp/test.jpg. blk_5234108089936612403_9027
2012-07-10 15:00:04,401 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.122:50010 is added to blk_5234108089936612403_9027 size [color=red]4096[/color]
2012-07-10 15:00:04,403 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.123:50010 is added to blk_5234108089936612403_9027 size [color=red]4096[/color]
2012-07-10 15:00:04,406 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.121:50010 is added to blk_5234108089936612403_9027 size [color=red]4096[/color]
2012-07-10 15:00:04,409 INFO org.apache.hadoop.hdfs.StateChange: Removing lease on file /user/tmp/test.jpg from client DFSClient_771894663
2012-07-10 15:00:04,409 INFO org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.completeFile: file /user/tmp/test.jpg is closed by DFSClient_771894663
2012-07-10 15:00:06,429 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addToInvalidates: blk_5234108089936612403 is added to invalidSet of 172.17.0.122:50010
2012-07-10 15:00:06,429 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addToInvalidates: blk_5234108089936612403 is added to invalidSet of 172.17.0.123:50010
2012-07-10 15:00:06,429 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addToInvalidates: blk_5234108089936612403 is added to invalidSet of 172.17.0.121:50010
2012-07-10 15:00:06,430 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=dell ip=/172.18.9.55 cmd=[color=red]delete [/color]src=/user/tmp/test.jpg dst=null perm=null
2012-07-10 15:00:06,431 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=dell ip=/172.18.9.55 cmd=[color=red]create [/color]src=/user/tmp/test.jpg dst=null perm=dell:supergroup:rw-r--r--
2012-07-10 15:00:06,435 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /user/tmp/test.jpg. blk_5499311137188998743_9028
2012-07-10 15:00:06,464 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.120:50010 is added to blk_5499311137188998743_9028 size [color=red]39455[/color]
2012-07-10 15:00:06,465 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.122:50010 is added to blk_5499311137188998743_9028 size [color=red]39455[/color]
2012-07-10 15:00:06,467 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.121:50010 is added to blk_5499311137188998743_9028 size [color=red]39455[/color]
2012-07-10 15:00:06,469 INFO org.apache.hadoop.hdfs.StateChange: Removing lease on file /user/tmp/test.jpg from client DFSClient_771894663

最终文件大小为:39455

求各位大神给小弟解决办法。

1个回答

换个版本试试呢,如果你配置hdfs-site.xml

 <property>

    <name>dfs.support.append</name>

    <value>true</value>

 </property>

通过append方法就可以实现文件追加的

weixin_42500407
快乐爸比 首先谢谢你的建议,不过我直接测试FileSystem中appeng函数,测试是成功的。 Configuration conf = new Configuration(); conf.set("fs.default.name", "hdfs://CDH:9000/"); FileSystem fs = FileSystem.get(conf); InputStream is = new BufferedInputStream(new FileInputStream("C:/a.txt")); OutputStream ot = fs.create(new Path("hdfs://CDH:9000/user/tmp/111")); IOUtils.copyBytes(is, ot, conf); InputStream is1 = new BufferedInputStream(new FileInputStream("C:/a.txt")); FSDataOutputStream out = fs.append(new Path("hdfs://CDH:9000/user/tmp/111")); IOUtils.copyBytes(is1, out1, conf); fs.copyToLocalFile(new Path("hdfs://CDH:9000/user/tmp/111"), new Path("C:/test1.txt")); fs.close(); File file = new File("C:/.111.txt.crc"); if(file.exists()){ file.delete(); } 这样是可以的,说明不是版本问题,而且由于是项目代码,量比较大,不好贴,就是想询问下可能的原因。
7 年多之前 回复
Csdn user default icon
上传中...
上传图片
插入图片
抄袭、复制答案,以达到刷声望分或其他目的的行为,在CSDN问答是严格禁止的,一经发现立刻封号。是时候展现真正的技术了!
立即提问