[SOLR-13058] Probable synchronization bug in OverseerTaskProcessor

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 8.0
• Fix Version/s: 8.0
• Component/s: SolrCloud
• Labels: overseer
• Assignee: Unassigned
• Reporter: Gus Heck
• Created: 11/Dec/18 20:32
• Updated: 08/Jun/19 15:01
• Resolved: 12/Dec/18 00:15

Description

 

Hit this stack trace:

 

solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=746, name=OverseerThreadFactory-164-thread-4, state=RUNNABLE, group=Overseer collection creation process.]
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at __randomizedtesting.SeedInfo.seed([8B3B8797FB9A174E:36FB84D55667AB6]:0)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > Caused by: java.util.ConcurrentModificationException
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at __randomizedtesting.SeedInfo.seed([8B3B8797FB9A174E]:0)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at java.util.HashMap$KeyIterator.next(HashMap.java:1461)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at java.util.AbstractCollection.toString(AbstractCollection.java:461)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at org.apache.solr.cloud.OverseerTaskProcessor.printTrackingMaps(OverseerTaskProcessor.java:614)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at org.apache.solr.cloud.OverseerTaskProcessor.access$700(OverseerTaskProcessor.java:67)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:518)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
solr/beast-results/TimeRoutedAliasUpdateProcessorTest/1/683/stdout- [junit4] > at java.lang.Thread.run(Thread.java:748)

while beasting TimeRoutedAliasUpdateProcessorTest. It appears to be possibly due to this seemingly incorrect synchronization:  

synchronized (runningTasks) {
  runningZKTasks.remove(id);
}

That should probably be synchronizing on the collection it is modifying similar to these snippets from the same class 

synchronized (runningZKTasks) {
  log.info("RunningZKTasks: {}", runningZKTasks.toString());
}

synchronized (runningZKTasks) {
  log.info("RunningZKTasks: {}", runningZKTasks.toString());
}

synchronized (runningZKTasks) {
  runningZKTasks.add(head.getId());
}