快乐爸比 2012-07-10 16:05
浏览 1045
已采纳

hdfs append追加文件上传的问题

各位大神们，小弟设置了hadoop-0.20-cdh3u0版本的dfs.support.append为true，然后想测试下文件追加上传。第一次上传一个文件的前4096看，第二次上传其余部分。但是发现2次上传的文件大小之和小于总文件大小。我发现问题在于，当第二次上传时，hdfs会删除第一次的文件，然后重新建立新文件再上传，所以文件大小只有第二次上传的数据大小。
以下是hadoop的日志：
2012-07-10 15:00:04,363 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=dell ip=/172.18.9.55 cmd=[color=red]create [/color]src=/user/tmp/test.jpg dst=null perm=dell:supergroup:rw-r--r--
2012-07-10 15:00:04,373 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /user/tmp/test.jpg. blk_5234108089936612403_9027
2012-07-10 15:00:04,401 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.122:50010 is added to blk_5234108089936612403_9027 size [color=red]4096[/color]
2012-07-10 15:00:04,403 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.123:50010 is added to blk_5234108089936612403_9027 size [color=red]4096[/color]
2012-07-10 15:00:04,406 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.121:50010 is added to blk_5234108089936612403_9027 size [color=red]4096[/color]
2012-07-10 15:00:04,409 INFO org.apache.hadoop.hdfs.StateChange: Removing lease on file /user/tmp/test.jpg from client DFSClient_771894663
2012-07-10 15:00:04,409 INFO org.apache.hadoop.hdfs.StateChange: DIR* NameSystem.completeFile: file /user/tmp/test.jpg is closed by DFSClient_771894663
2012-07-10 15:00:06,429 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addToInvalidates: blk_5234108089936612403 is added to invalidSet of 172.17.0.122:50010
2012-07-10 15:00:06,429 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addToInvalidates: blk_5234108089936612403 is added to invalidSet of 172.17.0.123:50010
2012-07-10 15:00:06,429 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addToInvalidates: blk_5234108089936612403 is added to invalidSet of 172.17.0.121:50010
2012-07-10 15:00:06,430 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=dell ip=/172.18.9.55 cmd=[color=red]delete [/color]src=/user/tmp/test.jpg dst=null perm=null
2012-07-10 15:00:06,431 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=dell ip=/172.18.9.55 cmd=[color=red]create [/color]src=/user/tmp/test.jpg dst=null perm=dell:supergroup:rw-r--r--
2012-07-10 15:00:06,435 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /user/tmp/test.jpg. blk_5499311137188998743_9028
2012-07-10 15:00:06,464 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.120:50010 is added to blk_5499311137188998743_9028 size [color=red]39455[/color]
2012-07-10 15:00:06,465 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.122:50010 is added to blk_5499311137188998743_9028 size [color=red]39455[/color]
2012-07-10 15:00:06,467 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: 172.17.0.121:50010 is added to blk_5499311137188998743_9028 size [color=red]39455[/color]
2012-07-10 15:00:06,469 INFO org.apache.hadoop.hdfs.StateChange: Removing lease on file /user/tmp/test.jpg from client DFSClient_771894663

最终文件大小为：39455

求各位大神给小弟解决办法。

  • 写回答

1条回答

  • wanghuanqiufly 2012-07-10 17:09
    关注

    换个版本试试呢，如果你配置hdfs-site.xml

     <property>

    <name>dfs.support.append</name>

    <value>true</value>

 </property>

    通过append方法就可以实现文件追加的

    本回答被题主选为最佳回答 , 对您是否有帮助呢?
    评论

报告相同问题？

悬赏问题

  • ¥20 ML307A在使用AT命令连接EMQX平台的MQTT时被拒绝
  • ¥20 腾讯企业邮箱邮件可以恢复么
  • ¥15 有人知道怎么将自己的迁移策略布到edgecloudsim上使用吗？
  • ¥15 错误 LNK2001 无法解析的外部符号
  • ¥50 安装pyaudiokits失败
  • ¥15 计组这些题应该咋做呀
  • ¥60 更换迈创SOL6M4AE卡的时候，驱动要重新装才能使用，怎么解决？
  • ¥15 让node服务器有自动加载文件的功能
  • ¥15 jmeter脚本回放有的是对的有的是错的
  • ¥15 r语言蛋白组学相关问题