[HADOOP-9601] Support native CRC on byte arrays

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 3.0.0-alpha1
• Component/s: performance, util
• Labels: perfomance
• Release Note: Support NativeCrc32 verification for byte[] array backed buffers
• Assignee: Gopal Vijayaraghavan
• Reporter: Todd Lipcon
• Created: 26/May/13 17:45
• Updated: 12/May/16 18:25
• Resolved: 18/Aug/14 21:14

Description

When we first implemented the Native CRC code, we only did so for direct byte buffers, because these correspond directly to native heap memory and thus make it easy to access via JNI. We'd generally assumed that accessing byte[] arrays from JNI was not efficient enough, but now that I know more about JNI I don't think that's true – we just need to make sure that the critical sections where we lock the buffers are short.

Issue Links

• is duplicated by: HADOOP-10838 Byte array native checksumming
• is related to: HADOOP-9934 the generic (non x86) libhadoop CRC code makes unaligned loads and stores
• is related to: HBASE-8323 Low hanging checksum improvements
• is superceded by: HADOOP-10838 Byte array native checksumming