[MAPREDUCE-5962] Support CRC32C in IFile

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.5.0
• Component/s: performance, task
• Assignee: Todd Lipcon Assign to me
• Reporter: Todd Lipcon
• Created: 07/Jul/14 16:56
• Updated: 24/Apr/18 01:31
• Resolved: 24/Apr/18 01:31

Description

Currently, the IFile format used by the MR shuffle checksums all data using the zlib CRC32 polynomial. If we allow use of CRC32C instead, we can get a large reduction in CPU usage by leveraging the native hardware CRC32C implementation (approx half a second of CPU time savings per GB checksummed).

Issue Links

• is related to: HDFS-3528 Use native CRC32 in DFS write path
• relates to: MAPREDUCE-2841 Task level native optimization
• relates to: HADOOP-10859 Native implementation of java Checksum interface