[HBASE-8323] Low hanging checksum improvements

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: Performance
• Assignee: Unassigned
• Reporter: Enis Soztutar
• Created: 11/Apr/13 00:57
• Updated: 16/Jun/22 05:59
• Resolved: 15/May/15 19:45

Description

Over at Hadoop land, tlipcon had done some improvements for checksums, a native implementation for CRC32C (HADOOP-7445) and bulk verify of checksums (HADOOP-7444).

In HBase, we can do

• Also develop a bulk verify API. Regardless of hbase.hstore.bytes.per.checksum we always want to verify of the whole checksum for the hfile block.
• Enable NativeCrc32 to be used as a checksum algo. It is not clear how much gain we can expect over pure java CRC32.

Though, longer term we should focus on convincing hdfs guys for inline checksums (HDFS-2699)

Issue Links

• duplicates: HBASE-11927 Use Native Hadoop Library for HFile checksum (And flip default from CRC32 to CRC32C)
• is related to: HBASE-8322 Re-enable hbase checksums by default
• relates to: HADOOP-9601 Support native CRC on byte arrays