Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-15234

PRS has issues with many concurrent creations

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: SolrCloud
    • Labels:
      None

      Description

      Running in a cloud environment, I am creating lots of 2x2 collections on a single node with 10 threads using PRS. These tests work fine without PRS.

      I see the following errors in my logs:

      2021-03-09 05:57:46.311 ERROR (OverseerStateUpdate-72069990450593800-hostname:8983_solr-n_0000000002) [   ] o.a.s.c.c.PerReplicaStatesOps Multi-op exception: [core_node8:2:A:L, core_node6:4:D, core_node2:4:D, core_node4:2:A:L]
      

      It would be good to include MDC logging on this if we can, so that I can know what collection the error describes.

      A little bit later:

      2021-03-09 05:57:46.694 INFO  (qtp594858858-16) [   x:collection-78_shard2_replica_n7] o.a.s.h.a.PrepRecoveryOp Going to wait for coreNodeName: core_node6, state: recovering, checkLive: true, onlyIfLeader: true, onlyIfLeaderActive: true
      2021-03-09 05:57:46.695 ERROR (qtp594858858-16) [   x:collection-78_shard2_replica_n7] o.a.s.h.RequestHandlerBase org.apache.solr.common.SolrException: core not found:collection-78_shard2_replica_n7
      	at org.apache.solr.handler.admin.PrepRecoveryOp.execute(PrepRecoveryOp.java:71)
      	at org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:367)
      	at org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
      	at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
      	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:214)
      	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:836)
      	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:800)
      	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:545)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:518)
      	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:432)
      	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
      	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
      	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
      	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
      	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
      	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
      	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
      	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
      	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
      	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
      	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
      	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:191)
      	at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
      	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
      	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
      	at org.eclipse.jetty.server.Server.handle(Server.java:516)
      	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
      	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
      	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
      	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
      	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
      	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
      	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
      	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
      	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
      	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
      	at java.base/java.lang.Thread.run(Thread.java:834)
      2021-03-09 05:57:46.695 INFO  (qtp594858858-16) [   x:collection-78_shard2_replica_n7] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={nodeName=hostname:8983_solr&onlyIfLeaderActive=true&core=collection-78_shard2_replica_n7&coreNodeName=core_node6&action=PREPRECOVERY&checkLive=true&state=recovering&onlyIfLeader=true&wt=javabin&version=2} status=400 QTime=1
      

      This might be an unrelated error happening during delete, but I can tell because I don't know what the first error refers to.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mdrob Mike Drob
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: