Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-662

distcp -update fails if source directory is empty (i.e. no files to copy) and target directory does not exists.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: distcp
    • Labels:
      None

      Description

      It should either create empty target directory or not make any changes on the target (as there is nothing to copy from source), but it should return success.

      Tested version of hadoop has HADOOP-5675.

      -bash-3.1$ bin/hadoop distcp -Dmapred.job.queue.name=xxx -i -p -update -m 10 hftp://xxx.yyy.com:50070/user/gogate/mirror_test1 hdfs://zzz.yyy.com:8020/user/gogate/mirror_test1

      09/06/24 19:46:42 INFO tools.DistCp: srcPaths=[hftp://xxx.yyy.com:50070/user/gogate/mirror_test1]
      09/06/24 19:46:42 INFO tools.DistCp: destPath=hdfs://zzz.yyy.com:8020/user/gogate/mirror_test1
      09/06/24 19:46:44 INFO tools.DistCp: hdfs://zzz.yyy.com:8020/user/gogate/mirror_test1 does not exist.
      09/06/24 19:46:44 INFO tools.DistCp: sourcePathsCount=1
      09/06/24 19:46:44 INFO tools.DistCp: filesToCopyCount=0
      09/06/24 19:46:44 INFO tools.DistCp: bytesToCopyCount=0.0
      With failures, global counters are inaccurate; consider running with -i
      Copy failed: java.io.FileNotFoundException: File does not exist: hdfs://zzz.yyy.com:8020/user/gogate/mirror_test1
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
      at org.apache.hadoop.tools.DistCp.finalize(DistCp.java:706)
      at org.apache.hadoop.tools.DistCp.copy(DistCp.java:653)
      at org.apache.hadoop.tools.DistCp.run(DistCp.java:858)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.tools.DistCp.main(DistCp.java:885)

        Activity

        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Ravi Gummadi made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 0.21.0 [ 12314045 ]
        Resolution Duplicate [ 3 ]
        Hide
        Ravi Gummadi added a comment -

        HADOOP-5762 fixed this already.

        Show
        Ravi Gummadi added a comment - HADOOP-5762 fixed this already.
        Hide
        Ravi Gummadi added a comment -

        The issue of empty directories(after HADOOP-5675) is fixed in HADOOP-5762. I verified with trunk. It passes creating the empty destination dir.

        Show
        Ravi Gummadi added a comment - The issue of empty directories(after HADOOP-5675 ) is fixed in HADOOP-5762 . I verified with trunk. It passes creating the empty destination dir.
        Suhas Gogate created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Suhas Gogate
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development