[SOLR-16848] Flaky DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 9.3
• Assignee: Houston Putman
• Reporter: Alex Deparvu
• Created: 21/Jun/23 15:45
• Updated: 21/Jul/23 20:42
• Resolved: 28/Jun/23 17:01

Description

Some stats first:

• Past 7 days trend:

Class: org.apache.solr.cloud.DeleteReplicaTest
Method: raceConditionOnDeleteAndRegisterReplica
Failures: 11.22% (44 / 392)

• Test failure is caused by a NullPointerException:

ERROR (coreZkRegister-772-thread-1-processing-127.0.0.1:40471_solr) [n:127.0.0.1:40471_solr c:raceDeleteReplicaCollection s:shard1 r:core_node4 x:raceDeleteReplicaCollection_shard1_replica_n2] o.a.s.c.DeleteReplicaTest Failed to delete replica
 => java.lang.NullPointerException: Cannot invoke "org.apache.solr.core.CoreContainer.getZkController()" because the return value of "org.apache.solr.embedded.JettySolrRunner.getCoreContainer()" is null

a more complete trace

o.a.s.c.DeleteReplicaTest Failed to delete replica
  2>           => java.lang.NullPointerException

  2>    at org.apache.solr.cloud.DeleteReplicaTest.lambda$raceConditionOnDeleteAndRegisterReplica$10(DeleteReplicaTest.java:350)
  2> java.lang.NullPointerException: null
  2>    at org.apache.solr.cloud.DeleteReplicaTest.lambda$raceConditionOnDeleteAndRegisterReplica$10(DeleteReplicaTest.java:350)
  2>    at org.apache.solr.core.ZkContainer.lambda$registerInZk$1(ZkContainer.java:211) [main/:?]
  2>    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  2>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  2>    at java.lang.Thread.run(Thread.java:829) [?:?]

o.a.s.c.u.ExecutorUtil Uncaught exception java.lang.AssertionError: Failed to delete replica thrown by thread: coreZkRegister-1586-thread-1-processing-127.0.0.1:34497_solr
  2>           => java.lang.Exception: Submitter stack trace
  2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:256)

java.lang.Exception: Submitter stack trace
  2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:256)
  2> 	at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:244) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.lambda$loadInternal$12(CoreContainer.java:1025)
  2> 	at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:234)
  2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
  2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
  2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
  2> 	at java.lang.Thread.run(Thread.java:829) [?:?]

Caused by: java.lang.Exception: Submitter stack trace
  2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:256)
  2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:140) ~[?:?]
  2> 	at com.codahale.metrics.InstrumentedExecutorService.submit(InstrumentedExecutorService.java:122)
  2> 	at org.apache.solr.core.CoreContainer.loadInternal(CoreContainer.java:1009) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.load(CoreContainer.java:753)
  2> 	at org.apache.solr.servlet.CoreContainerProvider.createCoreContainer(CoreContainerProvider.java:411)
  2> 	at org.apache.solr.servlet.CoreContainerProvider.init(CoreContainerProvider.java:230)
  2> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:407)

Issue Links

• links to: GitHub Pull Request #1718
• links to: GitHub Pull Request #1727