[HDFS-5685] DistCp will fail to copy with -delete switch

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.1
• Fix Version/s: 1.3.0
• Component/s: hdfs-client
• Target Version/s: 1.3.0
• Hadoop Flags: Reviewed

Release Note

Has dependency on MAPREDUCE-1285/MAPREDUCE-5698

• Assignee: Yongjun Zhang
• Reporter: Yongjun Zhang
• Created: 19/Dec/13 01:54
• Updated: 07/Jan/14 00:42
• Resolved: 07/Jan/14 00:25

Description

When using distcp command to copy files with -delete switch, running as user <xyz>,

hadoop distcp -p -i -update -delete hdfs://srchost:<port>/user hdfs://dsthost:<port>/user

It fails with the following exception:

Copy failed: java.io.FileNotFoundException: File does not exist: hdfs://dsthost:<port>/user/xyz/.stagingdistcp_urjb0g/_distcp_src_files
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:557)
at org.apache.hadoop.tools.DistCp$CopyInputFormat.getSplits(DistCp.java:266)
at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1081)
at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1073)
at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667)
at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)