[HADOOP-686] job.setOutputValueComparatorClass(theClass) should be supported

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 0.13.0

Environment

all environment

• Assignee: Unassigned
• Reporter: Feng Jiang
• Created: 07/Nov/06 03:31
• Updated: 08/Jul/09 16:51
• Resolved: 06/Jun/07 06:17

Description

if the input of Reduce phase is :

K2, V3
K2, V2
K1, V5
K1, V3
K1, V4

in the current hadoop, the reduce output could be:
K1, (V5, V3, V4)
K2, (V3, V2)

But I hope hadoop supports job.setOutputValueComparatorClass(theClass), so that i can make values are in order, and the output could be:
K1, (V3, V4, V5)
K2, (V2, V3)

This feature is very important, I think. Without it, we have to take the sorting by ourselves, and have to worry about the possibility that the values are too large to fit into memory. Then the codes becomes too hard to read. That is the reason why i think this feature is so important, and should be done in the hadoop framework.

Issue Links

• duplicates: HADOOP-485 allow a different comparator for grouping keys in calls to reduce