Hadoop Common
  1. Hadoop Common
  2. HADOOP-8423

MapFile.Reader.get() crashes jvm or throws EOFException on Snappy or LZO block-compressed data

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.2
    • Fix Version/s: 1.2.0, 2.0.2-alpha
    • Component/s: io
    • Labels:
      None
    • Environment:

      Linux 2.6.32.23-0.3-default #1 SMP 2010-10-07 14:57:45 +0200 x86_64 x86_64 x86_64 GNU/Linux

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      I am using Cloudera distribution cdh3u1.

      When trying to check native codecs for better decompression
      performance such as Snappy or LZO, I ran into issues with random
      access using MapFile.Reader.get(key, value) method.
      First call of MapFile.Reader.get() works but a second call fails.

      Also I am getting different exceptions depending on number of entries
      in a map file.
      With LzoCodec and 10 record file, jvm gets aborted.

      At the same time the DefaultCodec works fine for all cases, as well as
      record compression for the native codecs.

      I created a simple test program (attached) that creates map files
      locally with sizes of 10 and 100 records for three codecs: Default,
      Snappy, and LZO.
      (The test requires corresponding native library available)

      The summary of problems are given below:

      Map Size: 100
      Compression: RECORD
      ==================
      DefaultCodec: OK
      SnappyCodec: OK
      LzoCodec: OK

      Map Size: 10
      Compression: RECORD
      ==================
      DefaultCodec: OK
      SnappyCodec: OK
      LzoCodec: OK

      Map Size: 100
      Compression: BLOCK
      ================
      DefaultCodec: OK

      SnappyCodec: java.io.EOFException at
      org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:114)

      LzoCodec: java.io.EOFException at
      org.apache.hadoop.io.compress.BlockDecompressorStream.getCompressedData(BlockDecompressorStream.java:114)

      Map Size: 10
      Compression: BLOCK
      ==================
      DefaultCodec: OK

      SnappyCodec: java.lang.NoClassDefFoundError: Ljava/lang/InternalError
      at org.apache.hadoop.io.compress.snappy.SnappyDecompressor.decompressBytesDirect(Native
      Method)

      LzoCodec:
      #

      1. A fatal error has been detected by the Java Runtime Environment:
        #
      2. SIGSEGV (0xb) at pc=0x00002b068ffcbc00, pid=6385, tid=47304763508496
        #
      3. JRE version: 6.0_21-b07
      4. Java VM: Java HotSpot(TM) 64-Bit Server VM (17.0-b17 mixed mode linux-amd64 )
      5. Problematic frame:
      6. C [liblzo2.so.2+0x13c00] lzo1x_decompress+0x1a0
        #
      1. hadoop-8423.txt
        4 kB
        Todd Lipcon
      2. HADOOP-8423-branch-1.patch
        3 kB
        Harsh J
      3. HADOOP-8423-branch-1.patch
        3 kB
        Harsh J
      4. MapFileCodecTest.java
        4 kB
        Jason B

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Todd Lipcon
            Reporter:
            Jason B
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development