[HADOOP-8648] libhadoop: native CRC32 validation crashes when io.bytes.per.checksum=1

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha
• Fix Version/s: 2.0.2-alpha
• Hadoop Flags: Reviewed
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Created: 03/Aug/12 19:41
• Updated: 11/Oct/12 17:45
• Resolved: 05/Sep/12 22:31

Description

The native CRC32 code, found in pipelined_crc32c, crashes when chunksize is set to 1.

12:27:14,886  INFO NativeCodeLoader:50 - Loaded the native-hadoop library
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fa00ee5a340, pid=24100, tid=140326058854144
#
# JRE version: 6.0_29-b11
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.4-b02 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libhadoop.so.1.0.0+0x8340]  pipelined_crc32c+0xa0
#
# An error report file with more information is saved as:
# /h/hs_err_pid24100.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#
Aborted

The Java CRC code works fine in this case.

Choosing blocksize=1 is a _very_ odd choice. It means that we're storing a 4-byte checksum for every byte.

-rw-r--r--  1 cmccabe users  49398 Aug  3 11:33 blk_4702510289566780538
-rw-r--r--  1 cmccabe users 197599 Aug  3 11:33 blk_4702510289566780538_1199.meta

However, obviously crashing is never the right thing to do.

Issue Links

• is broken by: HDFS-3759 NNBench creates files with io.bytes.per.checksum = 1
• is related to: HDFS-3753 Tests don't run with native libraries