Solr
  1. Solr
  2. SOLR-6941

DistributedQueue#containsTaskWithRequestId can fail with NPE.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.10.4, 5.0, 6.0
    • Component/s: None
    • Labels:
      None

      Description

      I've seen this happen some recently. Seems data can be return as null and we need to guard against it.

        Issue Links

          Activity

          Hide
          Mark Miller added a comment -

          I ran into this fail while working on SOLR-4509.

          Show
          Mark Miller added a comment - I ran into this fail while working on SOLR-4509 .
          Hide
          Mark Miller added a comment -
             [junit4] ERROR   81.9s J0 | TestRebalanceLeaders.testDistribSearch <<<
             [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:33947/_d/iz, http://127.0.0.1:41117/_d/iz, http://127.0.0.1:33021/_d/iz, http://127.0.0.1:44859/_d/iz, http://127.0.0.1:46670/_d/iz]
             [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9EE8FA44BCDCD049:1F0E745CCB83B075]:0)
             [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:332)
             [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1015)
             [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:793)
             [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:736)
             [junit4]    > 	at org.apache.solr.cloud.TestRebalanceLeaders.issueCommands(TestRebalanceLeaders.java:292)
             [junit4]    > 	at org.apache.solr.cloud.TestRebalanceLeaders.rebalanceLeaderTest(TestRebalanceLeaders.java:119)
             [junit4]    > 	at org.apache.solr.cloud.TestRebalanceLeaders.doTest(TestRebalanceLeaders.java:85)
             [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868)
             [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
             [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41117/_d/iz: java.lang.NullPointerException
             [junit4]    > 	at org.apache.solr.common.cloud.ZkStateReader.fromJSON(ZkStateReader.java:140)
             [junit4]    > 	at org.apache.solr.common.cloud.ZkNodeProps.load(ZkNodeProps.java:92)
             [junit4]    > 	at org.apache.solr.cloud.DistributedQueue.containsTaskWithRequestId(DistributedQueue.java:125)
             [junit4]    > 	at org.apache.solr.handler.admin.CollectionsHandler.overseerCollectionQueueContains(CollectionsHandler.java:688)
             [junit4]    > 	at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:713)
             [junit4]    > 	at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:693)
             [junit4]    > 	at org.apache.solr.handler.admin.CollectionsHandler.rejoinElection(CollectionsHandler.java:488)
             [junit4]    > 	at org.apache.solr.handler.admin.CollectionsHandler.insurePreferredIsLeader(CollectionsHandler.java:403)
             [junit4]    > 	at org.apache.solr.handler.admin.CollectionsHandler.handleBalanceLeaders(CollectionsHandler.java:310)
             [junit4]    > 	at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:276)
             [junit4]    > 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
             [junit4]    > 	at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:740)
             [junit4]    > 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:266)
             [junit4]    > 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:204)
             [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
             [junit4]    > 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:138)
             [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
             [junit4]    > 	at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)
             [junit4]    > 	at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)
             [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
             [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
             [junit4]    > 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
             [junit4]    > 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125)
             [junit4]    > 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
             [junit4]    > 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
             [junit4]    > 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059)
             [junit4]    > 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
             [junit4]    > 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
             [junit4]    > 	at org.eclipse.jetty.server.Server.handle(Server.java:497)
             [junit4]    > 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313)
             [junit4]    > 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248)
             [junit4]    > 	at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
             [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626)
             [junit4]    > 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546)
             [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
             [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558)
             [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:214)
             [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:210)
             [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:353)
             [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:321)
             [junit4]    > 	... 47 more
          
          Show
          Mark Miller added a comment - [junit4] ERROR 81.9s J0 | TestRebalanceLeaders.testDistribSearch <<< [junit4] > Throwable #1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:33947/_d/iz, http://127.0.0.1:41117/_d/iz, http://127.0.0.1:33021/_d/iz, http://127.0.0.1:44859/_d/iz, http://127.0.0.1:46670/_d/iz] [junit4] > at __randomizedtesting.SeedInfo.seed([9EE8FA44BCDCD049:1F0E745CCB83B075]:0) [junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:332) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1015) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:793) [junit4] > at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:736) [junit4] > at org.apache.solr.cloud.TestRebalanceLeaders.issueCommands(TestRebalanceLeaders.java:292) [junit4] > at org.apache.solr.cloud.TestRebalanceLeaders.rebalanceLeaderTest(TestRebalanceLeaders.java:119) [junit4] > at org.apache.solr.cloud.TestRebalanceLeaders.doTest(TestRebalanceLeaders.java:85) [junit4] > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:868) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:41117/_d/iz: java.lang.NullPointerException [junit4] > at org.apache.solr.common.cloud.ZkStateReader.fromJSON(ZkStateReader.java:140) [junit4] > at org.apache.solr.common.cloud.ZkNodeProps.load(ZkNodeProps.java:92) [junit4] > at org.apache.solr.cloud.DistributedQueue.containsTaskWithRequestId(DistributedQueue.java:125) [junit4] > at org.apache.solr.handler.admin.CollectionsHandler.overseerCollectionQueueContains(CollectionsHandler.java:688) [junit4] > at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:713) [junit4] > at org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:693) [junit4] > at org.apache.solr.handler.admin.CollectionsHandler.rejoinElection(CollectionsHandler.java:488) [junit4] > at org.apache.solr.handler.admin.CollectionsHandler.insurePreferredIsLeader(CollectionsHandler.java:403) [junit4] > at org.apache.solr.handler.admin.CollectionsHandler.handleBalanceLeaders(CollectionsHandler.java:310) [junit4] > at org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:276) [junit4] > at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144) [junit4] > at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:740) [junit4] > at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:266) [junit4] > at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:204) [junit4] > at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [junit4] > at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:138) [junit4] > at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [junit4] > at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [junit4] > at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) [junit4] > at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [junit4] > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [junit4] > at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [junit4] > at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [junit4] > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [junit4] > at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [junit4] > at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [junit4] > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [junit4] > at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [junit4] > at org.eclipse.jetty.server.Server.handle(Server.java:497) [junit4] > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:313) [junit4] > at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [junit4] > at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:626) [junit4] > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:546) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:558) [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:214) [junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:210) [junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:353) [junit4] > at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:321) [junit4] > ... 47 more
          Hide
          Mark Miller added a comment -

          Odd - I also see failures with this leader rebalance test where the data is there but throwing an assert when our UTF-8 code tries to parse it.

          Show
          Mark Miller added a comment - Odd - I also see failures with this leader rebalance test where the data is there but throwing an assert when our UTF-8 code tries to parse it.
          Hide
          Mark Miller added a comment -

          And now I see a different fail on this test. I'm going to spin out another issue for the flaky test.

          Show
          Mark Miller added a comment - And now I see a different fail on this test. I'm going to spin out another issue for the flaky test.
          Hide
          Mark Miller added a comment -

          Seems TestReplicaProperties can fail with this as well.

          Show
          Mark Miller added a comment - Seems TestReplicaProperties can fail with this as well.
          Hide
          ASF subversion and git services added a comment -

          Commit 1651380 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1651380 ]

          SOLR-6941: DistributedQueue#containsTaskWithRequestId can fail with NPE.

          Show
          ASF subversion and git services added a comment - Commit 1651380 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1651380 ] SOLR-6941 : DistributedQueue#containsTaskWithRequestId can fail with NPE.
          Hide
          ASF subversion and git services added a comment -

          Commit 1651381 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1651381 ]

          SOLR-6941: DistributedQueue#containsTaskWithRequestId can fail with NPE.

          Show
          ASF subversion and git services added a comment - Commit 1651381 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1651381 ] SOLR-6941 : DistributedQueue#containsTaskWithRequestId can fail with NPE.
          Hide
          Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          Show
          Anshum Gupta added a comment - Bulk close after 5.0 release.
          Hide
          Shalin Shekhar Mangar added a comment -

          Reopening to backport to 4.10.4

          Show
          Shalin Shekhar Mangar added a comment - Reopening to backport to 4.10.4
          Hide
          ASF subversion and git services added a comment -

          Commit 1662478 from shalin@apache.org in branch 'dev/branches/lucene_solr_4_10'
          [ https://svn.apache.org/r1662478 ]

          SOLR-6941: DistributedQueue#containsTaskWithRequestId can fail with NPE.

          Show
          ASF subversion and git services added a comment - Commit 1662478 from shalin@apache.org in branch 'dev/branches/lucene_solr_4_10' [ https://svn.apache.org/r1662478 ] SOLR-6941 : DistributedQueue#containsTaskWithRequestId can fail with NPE.
          Hide
          Michael McCandless added a comment -

          Bulk close for 4.10.4 release

          Show
          Michael McCandless added a comment - Bulk close for 4.10.4 release

            People

            • Assignee:
              Mark Miller
              Reporter:
              Mark Miller
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development