Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6663

BlockDecompressorStream get EOF exception when decompressing the file compressed from empty file

    Details

    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fix EOF exception in BlockDecompressorStream when decompressing previous compressed empty file
    • Tags:
      hadoop io compress BlockDecompressorStream

      Description

      An empty file can be compressed using BlockDecompressorStream, which is for block-based compressiong algorithm such as LZO. However, when decompressing the compressed file, BlockDecompressorStream get EOF exception.

      Here is a typical exception stack:

      java.io.EOFException
      at org.apache.hadoop.io.compress.BlockDecompressorStream.rawReadInt(BlockDecompressorStream.java:125)
      at org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:96)
      at org.apache.hadoop.io.compress.BlockDecompressorStream.decompress(BlockDecompressorStream.java:82)
      at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
      at java.io.InputStream.read(InputStream.java:85)
      at org.apache.hadoop.util.LineReader.readLine(LineReader.java:134)
      at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:134)
      at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:39)
      at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:186)
      at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:170)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
      at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:18)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
      at org.apache.hadoop.mapred.Child.main(Child.java:196)

        Attachments

        1. BlockDecompressorStream.java.patch
          6 kB
          Kang Xiao
        2. BlockDecompressorStream.java.patch
          6 kB
          Kang Xiao
        3. BlockDecompressorStream.patch
          0.6 kB
          Kang Xiao
        4. HADOOP-6663.patch
          6 kB
          Tom White
        5. HADOOP-6663-0.20.2.patch
          0.6 kB
          Bennie Schut

          Activity

            People

            • Assignee:
              xiaokang Kang Xiao
              Reporter:
              xiaokang Kang Xiao
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: