[AURORA-1510] ConcurrentModificationException in PreemptorService

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Sprint: Twitter Aurora Q4'15 Sprint 15
• Story Points: 2
• Assignee: Zameer Manji
• Reporter: Zameer Manji
• Created: 01/Oct/15 19:46
• Updated: 10/Nov/15 17:59
• Resolved: 29/Oct/15 21:05

Description

On the latest master code:

E1001 03:41:58.910 THREAD595 com.google.common.util.concurrent.ServiceManager$ServiceListener.failed: Service PreemptorService [FAILE
D] has failed in the RUNNING state.
java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1429)
        at java.util.HashMap$EntryIterator.next(HashMap.java:1463)
        at java.util.HashMap$EntryIterator.next(HashMap.java:1461)
        at com.google.common.collect.AbstractMapBasedMultimap$KeySet$1.next(AbstractMapBasedMultimap.java:936)
        at java.util.Collections$UnmodifiableCollection$1.next(Collections.java:1042)
        at com.google.common.collect.Iterators$5.next(Iterators.java:558)
        at com.google.common.collect.Iterators.addAll(Iterators.java:362)
        at com.google.common.collect.Sets.newHashSet(Sets.java:238)
        at com.google.common.collect.Sets.newHashSet(Sets.java:218)
        at org.apache.aurora.scheduler.preemptor.PendingTaskProcessor$1.apply(PendingTaskProcessor.java:134)
        at org.apache.aurora.scheduler.preemptor.PendingTaskProcessor$1.apply(PendingTaskProcessor.java:119)
        at org.apache.aurora.scheduler.storage.db.DbStorage.read(DbStorage.java:138)
        at org.mybatis.guice.transactional.TransactionalMethodInterceptor.invoke(TransactionalMethodInterceptor.java:101)
        at org.apache.aurora.common.inject.TimedInterceptor.invoke(TimedInterceptor.java:84)
        at org.apache.aurora.scheduler.storage.log.LogStorage.read(LogStorage.java:659)
        at org.apache.aurora.scheduler.storage.CallOrderEnforcingStorage.read(CallOrderEnforcingStorage.java:115)
        at org.apache.aurora.scheduler.preemptor.PendingTaskProcessor.run(PendingTaskProcessor.java:119)
        at org.apache.aurora.scheduler.preemptor.PreemptorModule$PreemptorService.runOneIteration(PreemptorModule.java:148)
        at com.google.common.util.concurrent.AbstractScheduledService$1$1.run(AbstractScheduledService.java:174)
        at com.google.common.util.concurrent.Callables$3.run(Callables.java:95)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Issue Links

• is related to: AURORA-1511 PreemptorService failure does not trigger shutdown