[SOLR-17413] UpdateLog Replay can throw ConcurrentModificationException from sharing the request

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 9.8
• Labels: newdev pull-request-available
• Assignee: Jason Gerlowski
• Reporter: David Smiley
• Created: 21/Aug/24 02:55
• Updated: 28/Oct/24 19:11
• Resolved: 28/Oct/24 19:11

Description

I saw org.apache.solr.cloud.BasicDistributedZkTest fail with a stack trace revealing we have a real issue with UpdateLog replay. Essentially, a SolrQueryRequest is not threadsafe but we replay logs in parallel sharing the same request instance. Creating DistributedUpdateProcessor ends up adding to a shared HashMap in req.getContext() that should not be shared.

  2> WARNING: Uncaught exception in thread: Thread[replayUpdatesExecutor-590-thread-2,5,TGRP-BasicDistributedZkTest]
  2> java.util.ConcurrentModificationException
  2> 	at __randomizedtesting.SeedInfo.seed([F2227B12A8FC234]:0)
  2> 	at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1135)
  2> 	at org.apache.solr.update.processor.DistributedUpdateProcessorFactory.addParamToDistributedRequestWhitelist(DistributedUpdateProcessorFactory.java:46)
  2> 	at org.apache.solr.update.processor.DistributedUpdateProcessor.<init>(DistributedUpdateProcessor.java:190)
  2> 	at org.apache.solr.update.processor.DistributedUpdateProcessor.<init>(DistributedUpdateProcessor.java:160)
  2> 	at org.apache.solr.update.processor.DistributedZkUpdateProcessor.<init>(DistributedZkUpdateProcessor.java:114)
  2> 	at org.apache.solr.update.processor.DistributedUpdateProcessorFactory.getInstance(DistributedUpdateProcessorFactory.java:59)
  2> 	at org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:242)
  2> 	at org.apache.solr.update.processor.UpdateRequestProcessorChain.createProcessor(UpdateRequestProcessorChain.java:214)
  2> 	at org.apache.solr.update.UpdateLog$LogReplayer.lambda$doReplay$0(UpdateLog.java:2103)
  2> 	at java.base/java.lang.ThreadLocal$SuppliedThreadLocal.initialValue(ThreadLocal.java:305)
  2> 	at java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:195)
  2> 	at java.base/java.lang.ThreadLocal.get(ThreadLocal.java:172)
  2> 	at org.apache.solr.update.UpdateLog$LogReplayer.lambda$execute$2(UpdateLog.java:2342)
  2> 	at org.apache.solr.util.OrderedExecutor.lambda$execute$0(OrderedExecutor.java:68)
  2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$1(ExecutorUtil.java:449)
  2> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base/java.lang.Thread.run(Thread.java:829)

Issue Links

• is caused by: SOLR-12338 Replay buffering tlog in parallel
• relates to: SOLR-15983 ClassCastException in UpdateLog$LogReplayer.doReplay
• links to: GitHub Pull Request #2765