[FLINK-9904] Allow users to control MaxDirectMemorySize

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.4.2, 1.5.1, 1.7.2, 1.8.0, 1.9.0
• Component/s: Deployment / Scripts
• Labels: pull-request-available
• Flags: Patch
• Assignee: Ji Liu
• Reporter: Himanshu Roongta
• Created: 20/Jul/18 17:26
• Updated: 09/Jun/20 16:30
• Resolved: 09/Jun/20 15:21

Description

For people who use docker image and run flink in pods, currently, there is no way to update 

MaxDirectMemorySize

(Well one can create a custom version of taskmanager.sh)

 

As a result, it starts with a value of 8388607T . If the param taskmanager.memory.preallocate is set to false (default) the clean up will only occur when the MaxDirectMemorySize limit is hit and a gc full cycle kicks in. However with pods especially in kuberenete they will get killed because pods do not run at such a high value. (In our case we run 8GB per pod)

 

The fix would be to allow it be configurable via flink-conf. We can still have a default of 8388607T to avoid a breaking change. 

 

Issue Links

• Is contained by: FLINK-13980 FLIP-49 Unified Memory Configuration for TaskExecutors
• Is contained by: FLINK-16614 FLIP-116 Unified Memory Configuration for Job Manager
• links to: GitHub Pull Request #8266