Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-3151

YARN kills Flink TM containers due to memory overuse (outside heap/offheap)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.10.1, 1.0.0
    • Fix Version/s: 0.10.2, 1.0.0
    • Component/s: Runtime / Coordination
    • Labels:
      None
    • Environment:

      48 cores

      Description

      A Flink user who's running Flink on YARN with 1 processing slot, 2 GB of TM memory on a machine with 48 reported CPU cores is running into issues with TM containers being killed due to memory overuse.

      Setting the YARN memory cutoff to 0.5 resolves the problem, but its not really a feasible approach.
      Another solution to the problem was downgrading netty again from 4.0.31.Final to 4.0.27.Final resolved the issue.
      We upgraded Netty between 0.9 and 0.10.

      Most likely netty changed its behavior between the releases.

        Attachments

          Activity

            People

            • Assignee:
              uce Ufuk Celebi
              Reporter:
              rmetzger Robert Metzger
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: