Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-9969

Unreasonable memory requirements to complete examples/batch/WordCount

    XMLWordPrintableJSON

    Details

      Description

      setup on AWS EMR:

      • 5 worker nodes (m4.4xlarge nodes) 
      • 1 master node (m4.large)

      following command fails with out of memory errors:

      export HADOOP_CLASSPATH=`hadoop classpath`
      ./bin/flink run -m yarn-cluster -p 20 -yn 5 -ys 4 -ytm 16000 examples/batch/WordCount.jar

      Only increasing memory over 17.2GB example completes. At the same time after disabling flip6 following command succeeds:

      export HADOOP_CLASSPATH=`hadoop classpath`
      ./bin/flink run -m yarn-cluster -p 20 -yn 5 -ys 4 -ytm 1000 examples/batch/WordCount.jar

        Attachments

        1. yarn_logs
          993 kB
          Piotr Nowojski

          Issue Links

            Activity

              People

              • Assignee:
                trohrmann Till Rohrmann
                Reporter:
                pnowojski Piotr Nowojski
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: