[HDFS-3528] Use native CRC32 in DFS write path

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha
• Fix Version/s: 2.6.0
• Component/s: datanode, hdfs-client, performance
• Assignee: James Thomas
• Reporter: Todd Lipcon
• Created: 12/Jun/12 22:22
• Updated: 01/Dec/14 03:09
• Resolved: 29/Aug/14 00:51

Description

HDFS-2080 improved the CPU efficiency of the read path by using native SSE-enabled code for CRC verification. Benchmarks of the write path show that it's often CPU bound by checksums as well, so we should make the same improvement there.

Issue Links

• breaks: HADOOP-11064 UnsatisifedLinkError with hadoop 2.4 JARs on hadoop-2.6 due to NativeCRC32 method changes
• is part of: HDFS-4225 Improve HDFS write performance
• is related to: HDFS-2080 Speed up DFS read path by lessening checksum overhead
• relates to: HDFS-5395 write(ByteBuffer) method
• relates to: MAPREDUCE-5962 Support CRC32C in IFile
• relates to: HDFS-3529 Use direct buffers for data in write path

Sub-Tasks

• Byte array native checksumming on client side (HDFS changes): Closed