[HADOOP-7761] Improve performance of raw comparisons

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0
• Fix Version/s: 0.23.1
• Component/s: io, performance, util
• Target Version/s: 0.23.1
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 20/Oct/11 17:32
• Updated: 20/Jan/16 10:42
• Resolved: 27/Oct/11 05:57

Description

Guava has a nice implementation of lexicographical byte-array comparison that uses sun.misc.Unsafe to compare unsigned byte arrays long-at-a-time. Their benchmarks show it as being 2x more CPU-efficient than the equivalent pure-Java implementation. We can easily integrate this into WritableComparator.compareBytes to improve CPU performance in the shuffle.

Issue Links

• breaks: HADOOP-12720 Misuse of sun.misc.Unsafe by org.apache.hadoop.io.FastByteComparisons$LexicographicalComparerHolder$UnsafeComparer.compareTo causes misaligned memory access coredumps
• is related to: MAPREDUCE-1639 Grouping using hashing instead of sorting
• is related to: AVRO-939 Java: optimize BinaryData#compareBytes() to use sun.misc.Unsafe when available
• is related to: HBASE-4012 Further optimize byte comparison methods