[SOLR-17435] TestStressReorder reproducible failures: RejectedExecutionException from o.a.lucene.search.TaskExecutor

• Type: Test
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Chris M. Hostetter
• Created: 04/Sep/24 15:15
• Updated: 25/Sep/24 19:27

Description

I'm seeing this on main, reproduces reliably...

ERROR: The following test(s) have failed:
  - org.apache.solr.search.TestStressReorder.testStressReorderVersions (:solr:core)
    Test history: https://ge.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.search.TestStressReorder&tests.test=testStressReorderVersions http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.search.TestStressReorder.testStressReorderVersions
    Test output: /home/hossman/lucene/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestStressReorder.txt
    Reproduce with: ./gradlew :solr:core:test --tests "org.apache.solr.search.TestStressReorder.testStressReorderVersions" -Ptests.jvms=5 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=3F1CED560F2D6629 -Ptests.file.encoding=ISO-8859-1

First exception i noticed in console...

...
  2> 5615 INFO  (READER11) [n: c: s: r: x: t:] o.a.s.c.S.Request webapp=null path=null params={qt=/get&ids=49&wt=json} status=0 QTime=0
  2> 5615 INFO  (READER8) [n: c: s: r: x: t:] o.a.s.c.S.Request webapp=null path=null params={qt=/get&ids=12&wt=json} status=0 QTime=0
  2> 5603 ERROR (READER24) [n: c: s: r: x: t:] o.a.s.h.RequestHandlerBase Server exception
  2>           => java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$385/0x0000000100596c40@4b8dfe1 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@215a8ea0[Running, pool size = 1, active threads = 1, queued tasks = 894, completed tasks = 57525]
  2>    at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055)
  2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$385/0x0000000100596c40@4b8dfe1 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@215a8ea0[Running, pool size = 1, active threads = 1, queued tasks = 894, completed tasks = 57525]
  2>    at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2055) ~[?:?]
  2>    at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) ~[?:?]
  2>    at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) ~[?:?]
  2>    at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:430) ~[solr-solrj-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT fbd96cf5d3a2187c587d48f9f8c735493a4a0899 [snapshot build, details omitted]]
  2>    at org.apache.lucene.search.TaskExecutor$TaskGroup.invokeAll(TaskExecutor.java:152) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
  2>    at org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:76) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
  2>    at org.apache.lucene.index.TermStates.build(TermStates.java:116) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
  2>    at org.apache.lucene.search.TermQuery.createWeight(TermQuery.java:275) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
  2>    at org.apache.lucene.search.IndexSearcher.createWeight(IndexSearcher.java:882) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
  2>    at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:560) ~[lucene-core-9.11.1.jar:9.11.1 0c087dfdd10e0f6f3f6faecc6af4415e671a9e69 - 2024-06-23 12:31:02]
  2>    at org.apache.solr.search.SolrIndexSearcher.buildAndRunCollectorChain(SolrIndexSearcher.java:276) ~[main/:?]
  2>    at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1920) ~[main/:?]
  2>    at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1717) ~[main/:?]
  2>    at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:726) ~[main/:?]
  2>    at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:721) ~[main/:?]
  2>    at org.apache.solr.handler.component.QueryComponent.doProcessUngroupedSearch(QueryComponent.java:1690) ~[main/:?]
  2>    at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:432) ~[main/:?]
  2>    at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:457) ~[main/:?]
  2>    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:236) [main/:?]
  2>    at org.apache.solr.core.SolrCore.execute(SolrCore.java:2881) [main/:?]
  2>    at org.apache.solr.util.TestHarness.query(TestHarness.java:353) [solr-test-framework-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT fbd96cf5d3a2187c587d48f9f8c735493a4a0899 [snapshot build, details omitted]]
  2>    at org.apache.solr.util.TestHarness.query(TestHarness.java:333) [solr-test-framework-10.0.0-SNAPSHOT.jar:10.0.0-SNAPSHOT fbd96cf5d3a2187c587d48f9f8c735493a4a0899 [snapshot build, details omitted]]
  2>    at org.apache.solr.search.TestStressReorder$2.run(TestStressReorder.java:333) [test/:?]
  2> 5615 INFO  (READER11) [n: c: s: r: x: t:] o.a.s.c.S.Request webapp=null path=null params={qt=/get&ids=83&wt=json} status=0 QTime=0
  2> 5615 INFO  (READER24) [n: c: s: r: x: t:] o.a.s.c.S.Request webapp=null path=null params={q=id:144&omitHeader=true&wt=json} status=500 QTime=15
...

...i'll attached the full output.

Issue Links

• is caused by: SOLR-13350 Explore collector managers for multi-threaded search