[FLINK-15172] Optimize the operator algorithm to lazily allocate memory

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Component/s: Table SQL / Runtime
• Labels: pull-request-available
• Assignee: Jingsong Lee
• Reporter: Jingsong Lee
• Created: 10/Dec/19 09:24
• Updated: 24/Feb/20 01:52
• Resolved: 24/Feb/20 01:52

Description

Now after FLINK-14063 , operators will get all manage memory of TaskManager,  The cost of over allocate memory is very high, lead to performance regression of small batch sql jobs:

• Allocate memory will have the cost of memory management algorithm.
• Allocate memory will have the cost of memory initialization, will set all memory to zero. And this initialization will require the operating system to actually allocate physical memory.

• Over allocate memory will squash the file cache too.

We can optimize the operator algorithm, apply lazy allocation, and avoid meaningless memory allocation.

Issue Links

• links to: GitHub Pull Request #10797