[AURORA-1395] RescheduleCalculator precondition fails when using DbTaskStore

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: Scheduler
• Sprint: Twitter Aurora Q2'15 Sprint 7, Twitter Aurora Q2'15 Sprint 8, Twitter Aurora Q2'15 Sprint 9
• Story Points: 5
• Assignee: Bill Farner
• Reporter: Bill Farner
• Created: 10/Jul/15 22:34
• Updated: 24/Dec/15 20:02
• Resolved: 13/Aug/15 16:02

Description

When enabling the DB task store, i frequently encounter this exception due to a precondition check fail in RescheduleCalculator:

E0710 22:33:48.688 THREAD138 org.apache.aurora.scheduler.events.PubsubEventModule$1.handleException: Failed to dispatch event to public void org.apache.aurora.scheduler.async.TaskThrottler.taskChangedState(org.apache.aurora.scheduler.events.PubsubEvent$TaskStateChange): java.lang.IllegalStateException
java.lang.IllegalStateException
        at com.google.common.base.Preconditions.checkState(Preconditions.java:161)
        at org.apache.aurora.scheduler.async.RescheduleCalculator$RescheduleCalculatorImpl$1.apply(RescheduleCalculator.java:103)
        at org.apache.aurora.scheduler.async.RescheduleCalculator$RescheduleCalculatorImpl$1.apply(RescheduleCalculator.java:85)
        at org.apache.aurora.scheduler.async.RescheduleCalculator$RescheduleCalculatorImpl.getFlappingPenaltyMs(RescheduleCalculator.java:159)
        at org.apache.aurora.scheduler.async.TaskThrottler.taskChangedState(TaskThrottler.java:72)
        at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.google.common.eventbus.EventSubscriber.handleEvent(EventSubscriber.java:74)
        at com.google.common.eventbus.SynchronizedEventSubscriber.handleEvent(SynchronizedEventSubscriber.java:47)
        at com.google.common.eventbus.EventBus.dispatch(EventBus.java:322)
        at com.google.common.eventbus.AsyncEventBus.access$001(AsyncEventBus.java:34)
        at com.google.common.eventbus.AsyncEventBus$1.run(AsyncEventBus.java:117)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Issue Links

• duplicates: AURORA-1486 Updater hangs forever if slave removed during update
• is related to: AURORA-1386 Transaction isolation on DB stores is too strict
• relates to: AURORA-1389 Use a more strict DB transaction isolation.