Some bookie in production run OutOfMemory and the Bookie became really unstable.
17-01-24-13-11-42 Unexpected exception while writing 0@91465 :
17-01-24-13-11-42 java.lang.OutOfMemoryError: Java heap space
That error was inside "WriteEntryProcessorV3" class in a catch (Throwable ) clause.
I'm running the Bookie inside a Java process not started with the "standard" scripts.
My idea to handle this kind of errors is to add a global "System errors handler" to attach to any critical thread/operation and to pass any uncatched exception to it.
In case of existing "catch Throwable" the code will call the system handler as well.
Maybe we can provide a default implementation which only logs the error to the logger, a more invasive implementation which calls Runtime#halt.
I'm trying in production with ExitOnOutOfMemoryError option but it does not give any chance to report the status of the JVM