[AURORA-1582] Task History Pruning attempts can fail silently

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.12.0
• Sprint: Twitter Aurora Q1'16 Sprint 17
• Story Points: 3
• Assignee: Zameer Manji
• Reporter: Zameer Manji
• Created: 14/Jan/16 20:29
• Updated: 28/Jan/16 04:51
• Resolved: 22/Jan/16 01:39

Description

As discovered in AURORA-1580, task history pruning attempts can fail and if they do fail, they fail silently. The root cause seems to be that AsyncModule's AsyncProcessor threads just log the unhandled exception if it exists:

  private static void evaluateResult(Runnable runnable, Throwable throwable, Logger logger) {
    // See java.util.concurrent.ThreadPoolExecutor#afterExecute(Runnable, Throwable)
    // for more details and an implementation example.
    if (throwable == null) {
      if (runnable instanceof Future) {
        try {
          Future<?> future = (Future<?>) runnable;
          if (future.isDone()) {
            future.get();
          }
        } catch (InterruptedException ie) {
          Thread.currentThread().interrupt();
        } catch (ExecutionException ee) {
          logger.error(ee.toString(), ee);
        }
      }
    } else {
      logger.error(throwable.toString(), throwable);
    }
  }

I think instead of silently failing if work on these threads fail, we should shut down the scheduler, much like how if the preemptor or other guava service fails we shut down the scheduler. This way the scheduler does not enter an undefined state and operators are informed of the abnormal behaviour.

Issue Links

• is related to: AURORA-1593 PubSubEventModule fails to dispatch events to TaskHistoryPruner on startup
• is related to: AURORA-1596 Invalid connection error from MyBatis.
• relates to: AURORA-1580 java.util.NoSuchElementException from Tasks.getLatestEvent with DbTaskStore