Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-662

distcp -update fails if source directory is empty (i.e. no files to copy) and target directory does not exists.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: distcp
    • Labels:
      None

      Description

      It should either create empty target directory or not make any changes on the target (as there is nothing to copy from source), but it should return success.

      Tested version of hadoop has HADOOP-5675.

      -bash-3.1$ bin/hadoop distcp -Dmapred.job.queue.name=xxx -i -p -update -m 10 hftp://xxx.yyy.com:50070/user/gogate/mirror_test1 hdfs://zzz.yyy.com:8020/user/gogate/mirror_test1

      09/06/24 19:46:42 INFO tools.DistCp: srcPaths=[hftp://xxx.yyy.com:50070/user/gogate/mirror_test1]
      09/06/24 19:46:42 INFO tools.DistCp: destPath=hdfs://zzz.yyy.com:8020/user/gogate/mirror_test1
      09/06/24 19:46:44 INFO tools.DistCp: hdfs://zzz.yyy.com:8020/user/gogate/mirror_test1 does not exist.
      09/06/24 19:46:44 INFO tools.DistCp: sourcePathsCount=1
      09/06/24 19:46:44 INFO tools.DistCp: filesToCopyCount=0
      09/06/24 19:46:44 INFO tools.DistCp: bytesToCopyCount=0.0
      With failures, global counters are inaccurate; consider running with -i
      Copy failed: java.io.FileNotFoundException: File does not exist: hdfs://zzz.yyy.com:8020/user/gogate/mirror_test1
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:457)
      at org.apache.hadoop.tools.DistCp.finalize(DistCp.java:706)
      at org.apache.hadoop.tools.DistCp.copy(DistCp.java:653)
      at org.apache.hadoop.tools.DistCp.run(DistCp.java:858)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.tools.DistCp.main(DistCp.java:885)

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Suhas Gogate
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development