[MAPREDUCE-5947] Map phase merge can better utilize memory

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.0
• Component/s: performance, task
• Labels: newbie performance
• Assignee: jaehoon ko
• Reporter: jaehoon ko
• Created: 27/Jun/14 01:15
• Updated: 27/Jun/14 01:16

Description

Map phase merge reads spills from disk and writes intermediate results back to disk, and so on. I think it is possible to use memory to store intermediate results, thereby reducing disk IO. Because kvbuffer is nullified right before merge, we have at least io.sort.mb amount of heap available.
MAPREDUCE-4511 can be considered as an effort to utilize memory better through read ahead, but number of disk IO is unchanged.

Please give me your thoughts. I'd like to take up this issue.