[FLINK-3151] YARN kills Flink TM containers due to memory overuse (outside heap/offheap)

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.10.1, 1.0.0
• Fix Version/s: 0.10.2, 1.0.0
• Component/s: Runtime / Coordination

Environment

48 cores

• Assignee: Ufuk Celebi
• Reporter: Robert Metzger
• Created: 09/Dec/15 09:17
• Updated: 29/Jan/17 14:42
• Resolved: 13/Jan/16 10:27

Description

A Flink user who's running Flink on YARN with 1 processing slot, 2 GB of TM memory on a machine with 48 reported CPU cores is running into issues with TM containers being killed due to memory overuse.

Setting the YARN memory cutoff to 0.5 resolves the problem, but its not really a feasible approach.
Another solution to the problem was downgrading netty again from 4.0.31.Final to 4.0.27.Final resolved the issue.
We upgraded Netty between 0.9 and 0.10.

Most likely netty changed its behavior between the releases.