Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-2129

distcp between two clusters does not work if it is run on the target cluster

    XMLWordPrintableJSON

    Details

      Description

      I am trying to copy a directory (~100k files, ~500GB) between two clusters A and B (~70 nodes), using a command like:

      hadoop distcp -log /logdir hdfs://namenode-of-A:8600/srcdir hdfs://namenode-of-B:8600/targetdir

      I tried 4 ways of doing it:

      1) Copy from A to B, by running distcp on A
      2) Copy from A to B, by running distcp on B
      3) Copy from B to A, by running distcp on B
      4) Copy from B to A, by running distcp on A

      Invocations 1 and 3 succeeded, but 2 and 4 failed.

      I got a lot of errors of the type below:

      07/10/30 20:52:11 INFO mapred.JobClient: Running job: job_200710180049_0115
      07/10/30 20:52:12 INFO mapred.JobClient: map 0% reduce 0%
      07/10/30 20:54:41 INFO mapred.JobClient: map 1% reduce 0%
      07/10/30 20:56:52 INFO mapred.JobClient: map 2% reduce 0%
      07/10/30 20:57:41 INFO mapred.JobClient: Task Id : task_200710180049_0115_m_000184_0, Status : FAILED
      java.io.IOException: Some copies could not complete. See log for details.
      at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.close(CopyFiles.java:407)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:53)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:192)
      at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:1760)

      followed by the job failing:

      07/10/30 22:07:41 INFO mapred.JobClient: map 99% reduce 100%
      Copy failed: java.io.IOException: Job failed!
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:688)
      at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:481)
      at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:555)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:54)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:67)
      at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:566)

        Attachments

        1. 2129-0.patch
          1 kB
          Christopher Douglas
        2. HADOOP-2129.patch
          2 kB
          Doug Cutting
        3. HADOOP-2129-2.patch
          1 kB
          Doug Cutting
        4. HADOOP-2129-3.patch
          1 kB
          Doug Cutting

          Activity

            People

            • Assignee:
              cutting Doug Cutting
              Reporter:
              mabasrai Murtaza A. Basrai
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: