[HADOOP-591] Reducer sort should even out the pass factors in merging different pass

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Assignee: Devaraj Das
• Reporter: Runping Qi
• Created: 10/Oct/06 17:16
• Updated: 08/Jul/09 16:51
• Resolved: 04/Dec/06 23:07

Description

When multiple pass merging is needed during sort, the current sort implementation in SequenceFile class uses a simple "greedy" way to select pass factors, resulting uneven pass factor in different passes. For example, if the factor pass is 100 (the default), and there are 101 segments to be merged. The current implementation will first merge the first 100 segments into one and then merge the big output file with the last segment with pass factor 2. It will be better off to use pass factors 11 in the first pass and pass factor 10 in the second pass.

Issue Links

• is part of: HADOOP-611 SequenceFile.Sorter should have a merge method that returns an iterator