[FLINK-24401] TM cannot exit after Metaspace OOM

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.0, 1.13.0
• Fix Version/s: 1.13.6, 1.14.3, 1.15.0
• Component/s: Runtime / Coordination, Runtime / Task
• Labels: pull-request-available
• Assignee: Rui Fan
• Reporter: Rui Fan
• Created: 29/Sep/21 03:54
• Updated: 15/Dec/21 01:44
• Resolved: 08/Nov/21 07:51

Description

Hi masters, from the code and log, we can see that OOM will terminateJVM directly, but Metaspace OutOfMemoryError will graceful shutdown. The code comment mentions: it does not usually require more class loading to fail again with the Metaspace OutOfMemoryError..

But we encountered: after Metaspace OutOfMemoryError, java.lang.NoClassDefFoundError: Could not initialize class org.apache.flink.runtime.taskexecutor.TaskManagerRunner$Result., makes Tm unable to exit, keeps trying again, keeps NoClassDefFoundError, keeps class loading failure, until kill tm by manually.

I want to add a catch Throwable in the onFatalError method, and directly terminateJVM() in the catch. Is there any problem with this strategy? 

 

code link

picture:

 

 

 

 

Issue Links

• links to: GitHub Pull Request #17668