[HADOOP-3475] MapOutputBuffer allocates 4x as much space to record capacity as intended

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.17.0
• Fix Version/s: 0.17.1
• Hadoop Flags: Reviewed
• Assignee: Christopher Douglas
• Reporter: Christopher Douglas
• Created: 30/May/08 22:40
• Updated: 02/May/13 02:29
• Resolved: 17/Jun/08 22:39

Description

When setting up its accounting tables, MapTask.MapOutputBuffer calculates the size of each of its buffers as:

Where #total_bytes is $io.sort.mb * 2^20 and #accounting_bytes is $io.sort.record.percent * #total_bytes
kvbuffer: #total_bytes - #accounting_bytes
kvoffsets: 1/4 of #accounting_bytes
kvindices: 3/4 of #accounting_bytes

kvoffsets and kvindices are ints, not bytes; the size of both arrays is incorrectly calculated.

Issue Links

• is depended upon by: HADOOP-3550 Reduce tasks failing with OOM