[FLINK-3206] Heap size for non-pre-allocated off-heap memory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.0
• Component/s: Deployment / Scripts
• Assignee: Maximilian Michels
• Reporter: Greg Hogan
• Created: 06/Jan/16 16:27
• Updated: 21/Jan/16 17:44
• Resolved: 21/Jan/16 17:23

Description

In taskmanager.sh the heap size is adjusted for off-heap memory only with pre-allocation set to true. In TaskManager.scala the computation is reversed to compute the directMemorySize. The effect is the JVM heap settings are too high and the assumed size of direct memory is also too high.

taskmanager.memory.fraction: 0.9
taskmanager.memory.off-heap: true
taskmanager.heap.mb: 18000

With taskmanager.memory.preallocate: false

13:44:29,015 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Xms18000M
13:44:29,015 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Xmx18000M
13:44:30,591 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Limiting managed memory to 0.9 of the maximum memory size (161999 MB), memory will be allocated lazily.

With taskmanager.memory.preallocate: true

13:53:44,127 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Xms1800M
13:53:44,127 INFO  org.apache.flink.runtime.taskmanager.TaskManager              -     -Xmx1800M
13:53:45,743 INFO  org.apache.flink.runtime.taskmanager.TaskManager              - Using 0.9 of the maximum memory size for managed off-heap memory (15524 MB).