Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.23.0
-
None
-
Reviewed
Description
Guava has a nice implementation of lexicographical byte-array comparison that uses sun.misc.Unsafe to compare unsigned byte arrays long-at-a-time. Their benchmarks show it as being 2x more CPU-efficient than the equivalent pure-Java implementation. We can easily integrate this into WritableComparator.compareBytes to improve CPU performance in the shuffle.
Attachments
Attachments
Issue Links
- breaks
-
HADOOP-12720 Misuse of sun.misc.Unsafe by org.apache.hadoop.io.FastByteComparisons$LexicographicalComparerHolder$UnsafeComparer.compareTo causes misaligned memory access coredumps
- Open
- is related to
-
MAPREDUCE-1639 Grouping using hashing instead of sorting
- Open
-
AVRO-939 Java: optimize BinaryData#compareBytes() to use sun.misc.Unsafe when available
- Patch Available
-
HBASE-4012 Further optimize byte comparison methods
- Closed