[HDFS-2080] Speed up DFS read path by lessening checksum overhead

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.1
• Component/s: hdfs-client, performance
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 17/Jun/11 08:52
• Updated: 10/Mar/15 02:12
• Resolved: 03/Nov/11 06:59

Description

I've developed a series of patches that speeds up the HDFS read path by a factor of about 2.5x (~300M/sec to ~800M/sec for localhost reading from buffer cache) and also will make it easier to allow for advanced users (eg hbase) to skip a buffer copy.

Issue Links

• incorporates: HADOOP-7443 Add CRC32C as another DataChecksum implementation
• incorporates: HADOOP-7444 Add Checksum API to verify and calculate checksums "in bulk"
• incorporates: HADOOP-7445 Implement bulk checksum verification using efficient native code
• incorporates: HADOOP-7446 Implement CRC32C native code using SSE4.2 instructions
• relates to: HDFS-3528 Use native CRC32 in DFS write path

Sub-Tasks

• Simplify BlockReader to not inherit from FSInputChecker: Closed
• Switch default checksum to CRC32C: Closed