[FLINK-20740] Use managed memory to avoid direct memory OOM error for sort-merge shuffle (introduce a separated buffer pool)

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.13.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Yingjie Cao
• Reporter: Yingjie Cao
• Created: 23/Dec/20 04:28
• Updated: 26/Mar/21 07:16
• Resolved: 26/Mar/21 07:16

Description

Currently, sort-merge blocking shuffle uses some unmanaged memory for data writing and reading, which means users must increase the size of direct memory, otherwise, one may encounter direct memory OOM error, which is really bad for usability.

As discussed in the mailing list, a separated buffer pool will be introduced to be used by the shuffle reading phase, this buffer pool is part of Flink framework and has different life cycle with the tasks.

Issue Links

• links to: GitHub Pull Request #15199