[TEZ-3083] [Umbrella] Reduce merge memory overhead

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Jonathan Turner Eagles
• Created: 30/Jan/16 03:02
• Updated: 30/Jan/16 04:00

Description

1. Should we start accounting for these overheads in the memory calculations ?
2. Put a hard limit on how many in-mem segments will be maintained, before they're aggregated to disk.
3. Eventually, get the In-Memory merge functional so that we don't maintain 500K tiny buffers.
4. Evaluate a lower overhead collection to store mapoutputs, inmemoryreader, and segments.
5. Remove DataInputBuffer from TezMerger.Segment

Sub-Tasks

• Evaluate a lower overhead collection to store large collections: Open
• Reduce message size when empty partitions is high: Open