Catching Throwable which includes both java Error and Exception is not a good practice. If a java error is thrown, say OOM, the application code should let the jvm handle it.
In one concrete instance, the JD was experiencing an OOM due to a faulty user CR. Instead of trying to send CASes by reference it was trying to pass them by value. The OOM was caught and immediately wrapped in RuntimeException and subsequently rethrown. The jvm did not have a chance to catch the OOM leaving JD process running and eventually it hung with multiple JPs in limbo.
Review all ducc code where we catch Throwable and determine if this can be changed to catching Exceptions, letting the jvm handle the Errors. Optionally, ducc can plug in an UncaughtErrorHandler to the java main thread where we can decide what to do with each error that is caught. I think all java Errors should cause the jvm to exit immediately via halt().
Since Ducc daemons, JPs, JDs, etc use other frameworks like Spring, ActiveMQ, Jetty we should test what happens if Throwables are not caught in ducc code. Hopefully these frameworks dont catch Throwables and allow them to float up to be caught by java or custom UncaughtErrorHandler.