Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-8727

Limit Threadpools by default

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 5.2.1
    • Fix Version/s: None
    • Component/s: SolrJ
    • Labels:
      None

      Description

      Yesterday we had a problem in our prodution cluster, it was running out of native threads:

      null:java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
      	at org.apache.solr.servlet.HttpSolrCall.sendError(HttpSolrCall.java:593)
      	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:465)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
      	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
      	at org.eclipse.jetty.server.Server.handle(Server.java:497)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
      	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
      	at java.lang.Thread.run(Unknown Source)
      Caused by: java.lang.OutOfMemoryError: unable to create new native thread
      	at java.lang.Thread.start0(Native Method)
      	at java.lang.Thread.start(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.addWorker(Unknown Source)
      	at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)
      	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:135)
      	at java.util.concurrent.ExecutorCompletionService.submit(Unknown Source)
      	at org.apache.solr.handler.component.HttpShardHandler.submit(HttpShardHandler.java:250)
      	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:352)
      	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
      	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2064)
      	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)
      	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:450)
      	... 22 more
      

      After digging a little bit through the source code I found several ThreadPools which a default maxCoreSize of Integer.MAX_VALUE. I think we should figure out a better default then this.

      Going to create the corresponding pull reuquest on github for this.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                noble.paul Noble Paul
                Reporter:
                bjoernhaeuser Björn Häuser
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: