[MAPREDUCE-4891] Pluggable merge at reduce side

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0-alpha1
• Assignee: Unassigned
• Reporter: Haifeng Chen
• Created: 19/Dec/12 03:05
• Updated: 12/May/16 18:22
• Resolved: 19/Dec/12 06:32

Description

The current implementation of sort in MapReduce is cooperated by Map side sort and Reduce side merge. MAPREDUCE-2454 provided pluggable sort at the Map side currently and pluggable shuffle at Reduce side, while no pluggable merger provided.

Considering a general need of hash grouping and join, we may need to replace both the Map Sort and Reduce merge with a more light weight hash grouping alorithm. A general pluggable merge would help support this need.

Issue Links

• is related to: MAPREDUCE-2454 Allow external sorter plugin for MR