[TEZ-1911] MergeManager's unconditionalReserve() should check for memory limits before allocating memory to IntermediateMemoryToMemoryMerger

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.1, 0.8.3
• Target Version/s: 0.8.3
• Hadoop Flags: Reviewed
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 05/Jan/15 02:58
• Updated: 18/May/16 04:55
• Resolved: 26/Feb/16 06:27

Description

Currently, IntermediateMemoryToMemoryMerger invokes unconditionalReserve() to get the memory needed for intermediate mem-to-mem merging. It could potentially cause issue in the following scenario

1. tez.runtime.io.sort.factor set to 100 and assume shuffled data (e.g 60% memory occupied) haven't reached TEZ_RUNTIME_SHUFFLE_MERGE_PERCENT_DEFAULT
2. Assume that it reaches the sort.factor threshold before reaching merge threshold. This would kick in IntermediateMemoryToMemoryMerger.

In IntermediateMemoryToMemoryMerger, it would try to allocate additional 60% without any boundary checks. This could lead to OOM depending on the tez.runtime.io.sort.mb setting.

Issue Links

• blocks: TEZ-3116 Umbrella ticket for shuffle fixes
• duplicates: TEZ-1944 OOM when using tez.runtime.shuffle.memory-to-memory.enable=true
• relates to: TEZ-3147 Intermediate mem-to-mem: Fix early exit when only one segment can fit into memory