[FLINK-9969] Unreasonable memory requirements to complete examples/batch/WordCount

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0, 1.5.1, 1.5.2, 1.6.0
• Fix Version/s: 1.5.3, 1.6.0
• Component/s: Runtime / Coordination
• Labels: pull-request-available
• Assignee: Till Rohrmann
• Reporter: Piotr Nowojski
• Created: 26/Jul/18 12:24
• Updated: 28/Feb/19 14:00
• Resolved: 03/Aug/18 14:39

Description

setup on AWS EMR:

• 5 worker nodes (m4.4xlarge nodes) 

• 1 master node (m4.large)

following command fails with out of memory errors:

export HADOOP_CLASSPATH=`hadoop classpath`
./bin/flink run -m yarn-cluster -p 20 -yn 5 -ys 4 -ytm 16000 examples/batch/WordCount.jar

Only increasing memory over 17.2GB example completes. At the same time after disabling flip6 following command succeeds:

export HADOOP_CLASSPATH=`hadoop classpath`
./bin/flink run -m yarn-cluster -p 20 -yn 5 -ys 4 -ytm 1000 examples/batch/WordCount.jar

Issue Links

• links to: GitHub Pull Request #6479