- commitMemory & usedMemory are beyond their allowed threshold.
- InMemoryMerge kicks off and is in the process of flushing memory contents to disk
- As it progresses, it releases memory segments as well (but not yet over).
- Fetchers who need memory < maxSingleShuffleLimit, get scheduled.
- If fetchers are fast, this quickly adds up to commitMemory & usedMemory. Since InMemoryMerge is already in progress, this wouldn't trigger another merge().
- Pretty soon all fetchers would be stalled and get into the following state.
- Even if InMemoryMerger completes, "commitedMem & usedMem" are beyond their threshold and no other fetcher threads (all are in stalled state) are there to release memory. This causes fetchers to wait indefinitely.