[FLINK-20758] Use region file mechanism for shuffle data reading before we switch to managed memory

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.12.0
• Component/s: Runtime / Network
• Labels: usability
• Assignee: Yingjie Cao
• Reporter: Yingjie Cao
• Created: 24/Dec/20 03:59
• Updated: 25/Mar/21 02:42
• Resolved: 25/Mar/21 02:42

Description

FLINK-15981 implemented region file based data reader to solve the direct memory OOM issue introduced by usage of unmanaged direct memory, however only for BoundedBlockingResultPartition. We can introduce it to sort-merge based blocking shuffle to avoid the similar direct memory OOM problem which can improve the usability a lot.