Hadoop Common
  1. Hadoop Common
  2. HADOOP-8648

libhadoop: native CRC32 validation crashes when io.bytes.per.checksum=1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The native CRC32 code, found in pipelined_crc32c, crashes when chunksize is set to 1.

      12:27:14,886  INFO NativeCodeLoader:50 - Loaded the native-hadoop library
      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      #  SIGSEGV (0xb) at pc=0x00007fa00ee5a340, pid=24100, tid=140326058854144
      #
      # JRE version: 6.0_29-b11
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (20.4-b02 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # C  [libhadoop.so.1.0.0+0x8340]  pipelined_crc32c+0xa0
      #
      # An error report file with more information is saved as:
      # /h/hs_err_pid24100.log
      #
      # If you would like to submit a bug report, please visit:
      #   http://java.sun.com/webapps/bugreport/crash.jsp
      #
      Aborted
      

      The Java CRC code works fine in this case.

      Choosing blocksize=1 is a _very_ odd choice. It means that we're storing a 4-byte checksum for every byte.

      -rw-r--r--  1 cmccabe users  49398 Aug  3 11:33 blk_4702510289566780538
      -rw-r--r--  1 cmccabe users 197599 Aug  3 11:33 blk_4702510289566780538_1199.meta
      

      However, obviously crashing is never the right thing to do.

      1. HADOOP-8648.001.patch
        7 kB
        Colin Patrick McCabe
      2. HADOOP-8648.002.patch
        7 kB
        Colin Patrick McCabe
      3. HADOOP-8648.003.patch
        13 kB
        Colin Patrick McCabe
      4. HADOOP-8648.004.patch
        13 kB
        Colin Patrick McCabe
      5. HADOOP-8648.005.patch
        13 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development