Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When performing a massive distcp through hftp, we saw many tasks fail with

      2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
      but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
      at org.apache.hadoop.mapred.Child.main(Child.java:159)

      This means that read itself didn't fail but the resulted file was somehow smaller.

        Attachments

        1. h1085_20100716d.patch
          9 kB
          Tsz Wo Nicholas Sze
        2. h1085_20100716d_y0.20.1xx.patch
          5 kB
          Tsz Wo Nicholas Sze
        3. h1085_20100716d_y0.20.1xx_test.patch
          9 kB
          Tsz Wo Nicholas Sze
        4. h1085_20100716c.patch
          9 kB
          Tsz Wo Nicholas Sze
        5. h1085_20100716c_y0.20.1xx.patch
          5 kB
          Tsz Wo Nicholas Sze
        6. h1085_20100716b_y0.20.1xx.patch
          5 kB
          Tsz Wo Nicholas Sze
        7. h1085_20100716b_y0.20.1xx_with_test.patch
          13 kB
          Tsz Wo Nicholas Sze
        8. h1085_20100713.patch
          5 kB
          Tsz Wo Nicholas Sze

          Issue Links

            Activity

              People

              • Assignee:
                szetszwo Tsz Wo Nicholas Sze
                Reporter:
                knoguchi Koji Noguchi
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: