Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When performing a massive distcp through hftp, we saw many tasks fail with

      2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
      but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
      at org.apache.hadoop.mapred.Child.main(Child.java:159)

      This means that read itself didn't fail but the resulted file was somehow smaller.

      1. h1085_20100716d.patch
        9 kB
        Tsz Wo Nicholas Sze
      2. h1085_20100716d_y0.20.1xx_test.patch
        9 kB
        Tsz Wo Nicholas Sze
      3. h1085_20100716d_y0.20.1xx.patch
        5 kB
        Tsz Wo Nicholas Sze
      4. h1085_20100716c.patch
        9 kB
        Tsz Wo Nicholas Sze
      5. h1085_20100716c_y0.20.1xx.patch
        5 kB
        Tsz Wo Nicholas Sze
      6. h1085_20100716b_y0.20.1xx_with_test.patch
        13 kB
        Tsz Wo Nicholas Sze
      7. h1085_20100716b_y0.20.1xx.patch
        5 kB
        Tsz Wo Nicholas Sze
      8. h1085_20100713.patch
        5 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Koji Noguchi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development