Uploaded image for project: 'Aurora'
  1. Aurora
  2. AURORA-1582

Task History Pruning attempts can fail silently

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None
    • Sprint:
      Twitter Aurora Q1'16 Sprint 17
    • Story Points:
      3

      Description

      As discovered in AURORA-1580, task history pruning attempts can fail and if they do fail, they fail silently. The root cause seems to be that AsyncModule's AsyncProcessor threads just log the unhandled exception if it exists:

        private static void evaluateResult(Runnable runnable, Throwable throwable, Logger logger) {
          // See java.util.concurrent.ThreadPoolExecutor#afterExecute(Runnable, Throwable)
          // for more details and an implementation example.
          if (throwable == null) {
            if (runnable instanceof Future) {
              try {
                Future<?> future = (Future<?>) runnable;
                if (future.isDone()) {
                  future.get();
                }
              } catch (InterruptedException ie) {
                Thread.currentThread().interrupt();
              } catch (ExecutionException ee) {
                logger.error(ee.toString(), ee);
              }
            }
          } else {
            logger.error(throwable.toString(), throwable);
          }
        }
      

      I think instead of silently failing if work on these threads fail, we should shut down the scheduler, much like how if the preemptor or other guava service fails we shut down the scheduler. This way the scheduler does not enter an undefined state and operators are informed of the abnormal behaviour.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                zmanji Zameer Manji
                Reporter:
                zmanji Zameer Manji
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: