[TEZ-2214] FetcherOrderedGrouped can get stuck indefinitely when MergeManager misses memToDiskMerging

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.4, 0.6.1
• Target Version/s: 0.7.0, 0.5.4, 0.6.1
• Hadoop Flags: Reviewed
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 20/Mar/15 13:05
• Updated: 30/Jun/15 04:52
• Resolved: 25/Mar/15 23:00

Description

Scenario:

• commitMemory & usedMemory are beyond their allowed threshold.
• InMemoryMerge kicks off and is in the process of flushing memory contents to disk
• As it progresses, it releases memory segments as well (but not yet over).
• Fetchers who need memory < maxSingleShuffleLimit, get scheduled.
• If fetchers are fast, this quickly adds up to commitMemory & usedMemory. Since InMemoryMerge is already in progress, this wouldn't trigger another merge().
• Pretty soon all fetchers would be stalled and get into the following state.

Thread 9351: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Compiled frame; information may be imprecise)
 - java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
 - org.apache.tez.runtime.library.common.shuffle.orderedgrouped.MergeManager.waitForShuffleToMergeMemory() @bci=17, line=337 (Interpreted frame)
 - org.apache.tez.runtime.library.common.shuffle.orderedgrouped.FetcherOrderedGrouped.run() @bci=34, line=157 (Interpreted frame)

• Even if InMemoryMerger completes, "commitedMem & usedMem" are beyond their threshold and no other fetcher threads (all are in stalled state) are there to release memory. This causes fetchers to wait indefinitely.