-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Won't Fix
-
Affects Version/s: 2.5.0
-
Fix Version/s: None
-
Component/s: performance, task
-
Labels:None
Currently, the IFile format used by the MR shuffle checksums all data using the zlib CRC32 polynomial. If we allow use of CRC32C instead, we can get a large reduction in CPU usage by leveraging the native hardware CRC32C implementation (approx half a second of CPU time savings per GB checksummed).
- is related to
-
HDFS-3528 Use native CRC32 in DFS write path
-
- Closed
-
- relates to
-
MAPREDUCE-2841 Task level native optimization
-
- Resolved
-
-
HADOOP-10859 Native implementation of java Checksum interface
-
- Resolved
-