Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1085

hftp read failing silently

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.22.0
    • datanode
    • None
    • Reviewed

    Description

      When performing a massive distcp through hftp, we saw many tasks fail with

      2010-04-06 17:56:43,005 INFO org.apache.hadoop.tools.DistCp: FAIL 2010/0/part-00032 : java.io.IOException: File size not matched: copied 193855488 bytes (184.9m) to tmpfile (=hdfs://omehost.com:8020/somepath/part-00032)
      but expected 1710327403 bytes (1.6g) from hftp://someotherhost/somepath/part-00032
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:435)
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:543)
      at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:310)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
      at org.apache.hadoop.mapred.Child.main(Child.java:159)

      This means that read itself didn't fail but the resulted file was somehow smaller.

      Attachments

        1. h1085_20100716d.patch
          9 kB
          Tsz-wo Sze
        2. h1085_20100716d_y0.20.1xx.patch
          5 kB
          Tsz-wo Sze
        3. h1085_20100716d_y0.20.1xx_test.patch
          9 kB
          Tsz-wo Sze
        4. h1085_20100716c.patch
          9 kB
          Tsz-wo Sze
        5. h1085_20100716c_y0.20.1xx.patch
          5 kB
          Tsz-wo Sze
        6. h1085_20100716b_y0.20.1xx.patch
          5 kB
          Tsz-wo Sze
        7. h1085_20100716b_y0.20.1xx_with_test.patch
          13 kB
          Tsz-wo Sze
        8. h1085_20100713.patch
          5 kB
          Tsz-wo Sze

        Issue Links

          Activity

            People

              szetszwo Tsz-wo Sze
              knoguchi Koji Noguchi
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: