[AMBARI-21248] Exception needs to be handled properly for mail alert

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.2
• Fix Version/s: 2.5.2
• Component/s: ambari-server
• Assignee: Dmytro Sen
• Reporter: Richard Zang
• Created: 14/Jun/17 17:26
• Updated: 19/Jun/17 17:06
• Resolved: 19/Jun/17 16:25

Description

In org.apache.ambari.server.state.services.AlertNoticeDispatchService, we have below code snippet:

... 
protected void runOneIteration() throws Exception { 
… 
String targetType = target.getNotificationType(); 
NotificationDispatcher dispatcher = m_dispatchFactory.getDispatcher(targetType); 
… 
if (dispatcher.isDigestSupported()) { 
...

If the database has a change to AlertTargetEntity's targetType column (unlikely but let's assume it happened), then dispatcher can be null and the subsequent if clause will throw an exception. In https://github.com/google/guava/blob/master/guava/src/com/google/common/util/concurrent/AbstractScheduledService.java, we see that the exception may happen silently since shutDown() and notifyFailed() does not log anything.
...

class Task implements Runnable {
    @Override
    public void run() {
        lock.lock();
        try {
            if (runningTask.isCancelled()) { // task may have been cancelled while blocked on the lock. return; }

                AbstractScheduledService.this.runOneIteration();
            } catch (Throwable t) {
                try {
                    shutDown();
                } catch (Exception ignored) {
                    logger.log(Level.WARNING, "Error while attempting to shut down the service after failure.", ignored);
                }

                notifyFailed(t);
                runningTask.cancel(false); // prevent future invocations. 
            } finally {
                lock.unlock();
            }

        }
    }
}

So, runOneIteration will shutdown the scheduler if it hits an uncaught exception. We should wrap a try/catch.

Issue Links

• links to: Review Board