[AURORA-1729] Unclean curator teardown when scheduler fails a log write.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16.0
• Component/s: Scheduler, Service Discovery
• Assignee: John Sirois
• Reporter: John Sirois
• Created: 01/Jul/16 16:42
• Updated: 19/Sep/16 18:49
• Resolved: 04/Jul/16 14:22

Description

As discovered by Stephan Erb here:

W0701 12:14:53.070 [Thread-11, GuiceUtils$4:163] Trapped uncaught exception: org.apache.aurora.scheduler.storage.Storage$StorageException: There was a problem committing the transaction to the log. org.apache.aurora.scheduler.storage.Storage$StorageException: There was a problem committing the transaction to the log.
        at org.apache.aurora.scheduler.storage.log.LogStorage.lambda$doInTransaction$85(LogStorage.java:524) ~[aurora-0.14.0.jar:na]
        at org.apache.aurora.scheduler.storage.log.LogStorage$$Lambda$184/796968060.apply(Unknown Source) ~[na:na]
        at org.apache.aurora.scheduler.storage.db.DbStorage.transactionedWrite(DbStorage.java:161) ~[aurora-0.14.0.jar:na]
        at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101) ~[mybatis-guice-3.7.jar:3.7]
...skipping...
I0701 12:15:08.627 [Thread-11, StateMachine$Builder:389] SchedulerLifecycle state machine transition LEADER_AWAITING_REGISTRATION -> DEAD
I0701 12:15:08.632922 10792 sched.cpp:1907] Asked to stop the driver
I0701 12:15:08.633 [Thread-11, StateMachine$Builder:389] storage state machine transition READY -> STOPPED
I0701 12:15:08.636 [Curator-Framework-0, CuratorFrameworkImpl:821] backgroundOperationsLoop exiting
I0701 12:15:08.657 [main-EventThread, ClientCnxn$EventThread:519] EventThread shut down for session: 0x253eae5a8df106a
I0701 12:15:08.658 [Thread-11, ZooKeeper:684] Session: 0x253eae5a8df106a closed
I0701 12:15:08.660 [main, SchedulerMain:98] Stopping scheduler services.
E0701 12:15:08.661 [Curator-PathChildrenCache-0, PathChildrenCache:571]  java.lang.IllegalStateException: instance must be started before calling this method
        at com.google.common.base.Preconditions.checkState(Preconditions.java:174) ~[guava-19.0.jar:na]
        at org.apache.curator.framework.imps.CuratorFrameworkImpl.getChildren(CuratorFrameworkImpl.java:391) ~[curator-framework-2.10.0.jar:na]
        at org.apache.curator.framework.recipes.cache.PathChildrenCache.refresh(PathChildrenCache.java:508) ~[curator-recipes-2.10.0.jar:na]
        at org.apache.curator.framework.recipes.cache.RefreshOperation.invoke(RefreshOperation.java:35) ~[curator-recipes-2.10.0.jar:na]
        at org.apache.curator.framework.recipes.cache.PathChildrenCache$9.run(PathChildrenCache.java:772) ~[curator-recipes-2.10.0.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45-internal]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45-internal]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_45-internal]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_45-internal]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_45-internal]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_45-internal]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45-internal]