Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-3084

distcp fails for files with zero length

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.16.0, 0.16.1, 0.16.2
    • Fix Version/s: 0.16.2
    • Component/s: util
    • Labels:
      None

      Description

      distcp fails for files with zero length. This is a regression from 0.15.3

      distcp hftp://<namenode:port>/dir1/file1 file2
      08/03/24 23:09:45 INFO util.CopyFiles: srcPaths=[hftp://<namenode:port>/dir1/file1]
      08/03/24 23:09:45 INFO util.CopyFiles: destPath=file2
      08/03/24 23:09:45 INFO util.CopyFiles: srcCount=1
      08/03/24 23:09:46 INFO mapred.JobClient: Running job: job_200803242306_0001
      08/03/24 23:09:47 INFO mapred.JobClient: map 0% reduce 0%
      08/03/24 23:10:01 INFO mapred.JobClient: Task Id : task_200803242306_0001_m_000000_0, Status : FAILED
      java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
      at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.close(CopyFiles.java:448)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:53)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
      at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2084)

      08/03/24 23:10:18 INFO mapred.JobClient: Task Id : task_200803242306_0001_m_000000_1, Status : FAILED
      java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
      at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.close(CopyFiles.java:448)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:53)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
      at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2084)

      08/03/24 23:10:33 INFO mapred.JobClient: Task Id : task_200803242306_0001_m_000000_2, Status : FAILED
      java.io.IOException: Copied: 0 Skipped: 0 Failed: 1
      at org.apache.hadoop.util.CopyFiles$FSCopyFilesMapper.close(CopyFiles.java:448)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:53)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:208)
      at org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2084)

      ^@08/03/24 23:10:48 INFO mapred.JobClient: map 100% reduce 100%
      With failures, global counters are inaccurate; consider running with -i
      Copy failed: java.io.IOException: Job failed!
      at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:894)
      at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:526)
      at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:596)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:612)

        Attachments

        1. 3084-0.patch
          4 kB
          Christopher Douglas
        2. 3084v16-0.patch
          4 kB
          Christopher Douglas

          Activity

            People

            • Assignee:
              cdouglas Christopher Douglas
              Reporter:
              mukundm Mukund Madhugiri
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: