FLINK-14063 , operators will get all manage memory of TaskManager, The cost of over allocate memory is very high, lead to performance regression of small batch sql jobs:
- Allocate memory will have the cost of memory management algorithm.
- Allocate memory will have the cost of memory initialization, will set all memory to zero. And this initialization will require the operating system to actually allocate physical memory.
- Over allocate memory will squash the file cache too.
We can optimize the operator algorithm, apply lazy allocation, and avoid meaningless memory allocation.