[BOOKKEEPER-1030] Better management of OutOfMemory errors

• Type: Wish
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 4.4.0
• Component/s: bookkeeper-server
• Labels: ghmigrated iktriage0
• Assignee: Enrico Olivelli
• Reporter: Enrico Olivelli
• Created: 10/Apr/17 08:16
• Updated: 17/Oct/17 21:26

Description

Some bookie in production run OutOfMemory and the Bookie became really unstable.

17-01-24-13-11-42 Unexpected exception while writing 0@91465 :
17-01-24-13-11-42 java.lang.OutOfMemoryError: Java heap space

That error was inside "WriteEntryProcessorV3" class in a catch (Throwable ) clause.

I'm running the Bookie inside a Java process not started with the "standard" scripts.

My idea to handle this kind of errors is to add a global "System errors handler" to attach to any critical thread/operation and to pass any uncatched exception to it.
In case of existing "catch Throwable" the code will call the system handler as well.

Maybe we can provide a default implementation which only logs the error to the logger, a more invasive implementation which calls Runtime#halt.

I'm trying in production with ExitOnOutOfMemoryError option but it does not give any chance to report the status of the JVM