Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6217

ContaineredTaskManagerParameters sets off heap memory size incorrectly

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: YARN
    • Labels:
      None

      Description

      Thanks Bill Liu for triaging the issue.

      When taskmanager.memory.off-heap is disabled, we observed that the total memory that Flink allocates exceed the total memory of the container:

      For a 8G container the JobManager starts the container with the following parameter:

      $JAVA_HOME/bin/java -Xms6072m -Xmx6072m -XX:MaxDirectMemorySize=6072m ...
      

      The total amount of heap memory plus the off-heap memory exceeds the total amount of memory of the container. As a result YARN occasionally kills the container.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user haohui opened a pull request:

          https://github.com/apache/flink/pull/3648

          FLINK-6217 ContaineredTaskManagerParameters sets off-heap memory size incorrectly

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/haohui/flink FLINK-6217

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3648.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3648


          commit 6da863509775d6f371acdecd706db5f3a7181bf2
          Author: Haohui Mai <wheat9@apache.org>
          Date: 2017-03-29T23:57:52Z

          FLINK-6217 ContaineredTaskManagerParameters sets off-heap memory size incorrectly.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user haohui opened a pull request: https://github.com/apache/flink/pull/3648 FLINK-6217 ContaineredTaskManagerParameters sets off-heap memory size incorrectly You can merge this pull request into a Git repository by running: $ git pull https://github.com/haohui/flink FLINK-6217 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3648.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3648 commit 6da863509775d6f371acdecd706db5f3a7181bf2 Author: Haohui Mai <wheat9@apache.org> Date: 2017-03-29T23:57:52Z FLINK-6217 ContaineredTaskManagerParameters sets off-heap memory size incorrectly.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user haohui commented on the issue:

          https://github.com/apache/flink/pull/3648

          The failed tests are being addressed in FLINK-6175.

          Show
          githubbot ASF GitHub Bot added a comment - Github user haohui commented on the issue: https://github.com/apache/flink/pull/3648 The failed tests are being addressed in FLINK-6175 .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user haohui commented on the issue:

          https://github.com/apache/flink/pull/3648

          The test failure seems unrelated. Cannot reproduce locally.

          Show
          githubbot ASF GitHub Bot added a comment - Github user haohui commented on the issue: https://github.com/apache/flink/pull/3648 The test failure seems unrelated. Cannot reproduce locally.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3648

          Good fix, thanks @haohui

          I was wondering - I am a trying to advocate fewer dependencies in Flink (there is always the problem of shading and conflicts) so if there is a way to do this without Guava, that'd be great. I think `commons-lang3` has a joiner as well, or we might just add a joiner to Flink...

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3648 Good fix, thanks @haohui I was wondering - I am a trying to advocate fewer dependencies in Flink (there is always the problem of shading and conflicts) so if there is a way to do this without Guava, that'd be great. I think `commons-lang3` has a joiner as well, or we might just add a joiner to Flink...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user haohui commented on the issue:

          https://github.com/apache/flink/pull/3648

          Totally Agree @StephanEwen please take another look.

          Show
          githubbot ASF GitHub Bot added a comment - Github user haohui commented on the issue: https://github.com/apache/flink/pull/3648 Totally Agree @StephanEwen please take another look.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3648

          +1 to merge this

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3648 +1 to merge this
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user tillrohrmann commented on the issue:

          https://github.com/apache/flink/pull/3648

          Thanks for your contribution @haohui and the review @StephanEwen. Changes look good to me. Merging this PR.

          Show
          githubbot ASF GitHub Bot added a comment - Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3648 Thanks for your contribution @haohui and the review @StephanEwen. Changes look good to me. Merging this PR.
          Hide
          till.rohrmann Till Rohrmann added a comment -

          Fixed via b1f186425c3a15de9d7c0f66e3d3d4d3a7e6b259

          Show
          till.rohrmann Till Rohrmann added a comment - Fixed via b1f186425c3a15de9d7c0f66e3d3d4d3a7e6b259
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3648

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3648

            People

            • Assignee:
              wheat9 Haohui Mai
              Reporter:
              wheat9 Haohui Mai
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development