[MAPREDUCE-5821] IFile merge allocates new byte array for every value

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.1
• Fix Version/s: 2.4.1
• Component/s: performance, task
• Target Version/s: 2.4.1
• Hadoop Flags: Reviewed
• Assignee: Todd Lipcon
• Reporter: Todd Lipcon
• Created: 04/Apr/14 17:49
• Updated: 09/Sep/14 20:13
• Resolved: 14/May/14 17:52

Description

I wrote a standalone benchmark of the MapOutputBuffer and found that it did a lot of allocations during the merge phase. After looking at an allocation profile, I found that IFile.Reader.nextRawValue() would always allocate a new byte array for every value, so the allocation rate goes way up during the merge phase of the mapper. I imagine this also affects the reducer input, though I didn't profile that.

Issue Links

• relates to: HADOOP-5494 IFile.Reader should have a nextRawKey/nextRawValue