[HADOOP-7444] Add Checksum API to verify and calculate checksums "in bulk"

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.23.0
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 06/Jul/11 04:40
• Updated: 15/Nov/11 00:50
• Resolved: 13/Jul/11 16:13

Description

Currently, the various checksum types only provide the capability to calculate the checksum of a range of a byte array. For HDFS-2080, it's advantageous to provide an API that, given a buffer with some number of "checksum chunks", can either calculate or verify the checksums of all of the chunks. For example, given a 4KB buffer and a 512-byte chunk size, it would calculate or verify 8 CRC32s in one call.

This allows efficient JNI-based checksum implementations since the cost of crossing the JNI boundary is amortized across many computations.

Issue Links

• blocks: HDFS-2129 Simplify BlockReader to not inherit from FSInputChecker
• blocks: HADOOP-7443 Add CRC32C as another DataChecksum implementation
• blocks: HADOOP-7445 Implement bulk checksum verification using efficient native code
• is part of: HDFS-2080 Speed up DFS read path by lessening checksum overhead