[HADOOP-3294] distcp leaves empty blocks afte successful execution

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.16.3
• Fix Version/s: 0.16.4
• Component/s: util

Environment

0.16.3 without any patches. Dfs permissions turned off everywhere, such that HADOOP-3138 and HADOOP-3186 do not apply

• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Christian Kunz
• Created: 21/Apr/08 23:33
• Updated: 09/May/08 22:31
• Resolved: 24/Apr/08 23:02

Description

I copied around 40 TB between two hadoop clusters, with distcp running on source.

Job was successful, but one destination file was empty because of its only block being empty.
None of the distcp log files have any mentioning of this file.

There were a couple of messages in the namenode server log of the destination cluster referencing the file:

hadoop-xxxnamenode-yyy.log.2008-04-19:2008-04-19 02:19:15,666 INFO org.apache.hadoop.dfs.StateChange: BLOCK* NameSystem.allocateBlock: destinationDir/distcp_tmp_z0g93p/fileName. blk-9209890281741927376
hadoop-xxx-namenode-yyy.log.2008-04-19:2008-04-19 02:54:45,820 WARN org.apache.hadoop.dfs.StateChange: DIR* NameSystem.internalReleaseCreate: attempt to release a create lock on destinationDir/_distcp_tmp_z0g93p/fileName file does not exist.

distcp should not rely on the user to double-check.
Would it make sense to add a reducer to compare destination file sizes with source files sizes and do some appropriate action?

Issue Links

• is related to: HADOOP-2891 The dfsclient on exit deletes files that are open and not closed.