[FLINK-15981] Control the direct memory in FileChannelBoundedData.FileBufferReader

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.10.0, 1.10.1, 1.10.2, 1.11.0, 1.11.1
• Fix Version/s: 1.12.0
• Component/s: Runtime / Network
• Labels: pull-request-available
• Assignee: Zhijiang
• Reporter: Jingsong Lee
• Created: 11/Feb/20 03:02
• Updated: 14/Jan/21 09:26
• Resolved: 05/Nov/20 14:59

Description

Now, the default blocking BoundedData is FileChannelBoundedData. In its reader, will create new direct buffer 64KB.

When parallelism greater than 100, users need configure "taskmanager.memory.task.off-heap.size" to avoid direct memory OOM. It is hard to configure, and it cost a lot of memory. Consider 1000 parallelism, maybe we need 1GB+ for a task manager.

This is not conducive to the scenario of less slots and large parallelism. Batch jobs could run little by little, but memory shortage would consume a lot.

If we provided N-Input operators, maybe things will be worse. This means the number of subpartitions that can be requested at the same time will be more. We have no idea how much memory.

Here are my rough thoughts:

• Obtain memory from network buffers.
• provide "The maximum number of subpartitions that can be requested at the same time".

Issue Links

• is related to: FLINK-15989 Rewrap OutOfMemoryError in allocateUnpooledOffHeap with better message
• links to: GitHub Pull Request #13523
• mentioned in: Page Loading...