Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5685

DistCp will fail to copy with -delete switch

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.1
    • Fix Version/s: 1.3.0
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      When using distcp command to copy files with -delete switch, running as user <xyz>,

      hadoop distcp -p -i -update -delete hdfs://srchost:<port>/user hdfs://dsthost:<port>/user

      It fails with the following exception:

      Copy failed: java.io.FileNotFoundException: File does not exist: hdfs://dsthost:<port>/user/xyz/.stagingdistcp_urjb0g/_distcp_src_files
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:557)
      at org.apache.hadoop.tools.DistCp$CopyInputFormat.getSplits(DistCp.java:266)
      at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1081)
      at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1073)
      at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179)
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:983)
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
      at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
      at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
      at org.apache.hadoop.tools.DistCp.copy(DistCp.java:667)
      at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)

        Attachments

        1. HDFS-5685.005.patch
          19 kB
          Yongjun Zhang
        2. HDFS-5685.004.patch
          18 kB
          Yongjun Zhang
        3. HDFS-5685.003.patch
          9 kB
          Yongjun Zhang
        4. HDFS-5685.002.patch
          9 kB
          Yongjun Zhang
        5. HDFS-5685.001.patch
          4 kB
          Yongjun Zhang

          Activity

            People

            • Assignee:
              yzhangal Yongjun Zhang
              Reporter:
              yzhangal Yongjun Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: