[MAPREDUCE-1126] shuffle should use serialization to get comparator

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: task
• Hadoop Flags: Reviewed
• Assignee: Aaron Kimball
• Reporter: Doug Cutting
• Created: 20/Oct/09 17:14
• Updated: 29/Jul/14 18:56
• Resolved: 29/Jul/14 18:56

Description

Currently the key comparator is defined as a Java class. Instead we should use the Serialization API to create key comparators. This would permit, e.g., Avro-based comparators to be used, permitting efficient sorting of complex data types without having to write a RawComparator in Java.

Issue Links

• blocks: MAPREDUCE-815 Add AvroInputFormat and AvroOutputFormat so that hadoop can use Avro Serialization
• depends upon: HADOOP-6438 Add configuration getters/setters to serialization classes
• depends upon: HADOOP-6420 String-to-String Maps should be embeddable in Configuration
• depends upon: HADOOP-6443 Serialization classes accept invalid metadata
• is blocked by: HADOOP-6323 Serialization should provide comparators
• is related to: MAPREDUCE-1183 Serializable job components: Mapper, Reducer, InputFormat, OutputFormat et al
• is related to: MAPREDUCE-1360 Record skipping should work with more serializations
• relates to: MAPREDUCE-1462 Enable context-specific and stateful serializers in MapReduce
• relates to: MAPREDUCE-1452 Add a low-level MapReduce API