Lucene - Core
  1. Lucene - Core
  2. LUCENE-4401

ArrayIndexOutOfBoundsException for surround parser

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0-BETA
    • Fix Version/s: 4.0, 3.6.2, 5.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I got the following exception when I query solr with "pcnt(kk w hit) OR (ipad) OR (iphoine))" and the defType is 'surround'.

      18:16:45
      SEVERE
      SolrCore
      java.lang.ArrayIndexOutOfBoundsException: 2147483647 at org.apache.lucene.search.similarities.TFIDFSimilarity$SloppyTFIDFDocScorer.score(TFIDFSimilarity.java:793) at org.apache.lucene.search.spans.SpanScorer.score(SpanScorer.java:93) at org.apache.lucene.search.DisjunctionSumScorer.afterNext(DisjunctionSumScorer.java:94) at org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:82) at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:284) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:573) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:275) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1390) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1265) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:390) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:411) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:206) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:351) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:636)
      18:16:45
      SEVERE
      SolrDispatchFilter
      null:java.lang.ArrayIndexOutOfBoundsException: 2147483647 at org.apache.lucene.search.similarities.TFIDFSimilarity$SloppyTFIDFDocScorer.score(TFIDFSimilarity.java:793) at org.apache.lucene.search.spans.SpanScorer.score(SpanScorer.java:93) at org.apache.lucene.search.DisjunctionSumScorer.afterNext(DisjunctionSumScorer.java:94) at org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:82) at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:284) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:573) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:275) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:1390) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:1265) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:390) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:411) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:206) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:351) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47) at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:636)
      18:16:45
      SEVERE
      SolrCore
      org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:http://192.168.50.78:8985/solr/ac201209w3_s1,​ http://192.168.50.76:8985/solr/ac201209w3_s1 at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:300) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:351) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:636) Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:http://192.168.50.78:8985/solr/ac201209w3_s1,​ http://192.168.50.76:8985/solr/ac201209w3_s1 at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:324) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:167) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) ... 1 more Caused by: org.apache.solr.common.SolrException: Server at http://192.168.50.78:8985/solr/ac201209w3_s1 returned non ok status:500,​ message:Server Error at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:182) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:289) ... 10 more
      18:16:45
      SEVERE
      SolrDispatchFilter
      null:org.apache.solr.common.SolrException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:http://192.168.50.78:8985/solr/ac201209w3_s1,​ http://192.168.50.76:8985/solr/ac201209w3_s1 at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:300) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1656) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:454) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:484) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111) at org.eclipse.jetty.server.Server.handle(Server.java:351) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454) at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230) at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534) at java.lang.Thread.run(Thread.java:636) Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:http://192.168.50.78:8985/solr/ac201209w3_s1,​ http://192.168.50.76:8985/solr/ac201209w3_s1 at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:324) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:167) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:1) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) ... 1 more Caused by: org.apache.solr.common.SolrException: Server at http://192.168.50.78:8985/solr/ac201209w3_s1 returned non ok status:500,​ message:Server Error at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:182) at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:289) ... 10 more

        Activity

        Hide
        Robert Muir added a comment -

        Its bogus to call score() on a DISI set at NO_MORE_DOCS, i figured I would add some asserts to see if we can catch this:

        Index: lucene/core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java
        ===================================================================
        --- lucene/core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java	(revision 1386719)
        +++ lucene/core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java	(working copy)
        @@ -91,6 +91,7 @@
           private void afterNext() throws IOException {
             final Scorer sub = subScorers[0];
             doc = sub.docID();
        +    assert doc != NO_MORE_DOCS;
             score = sub.score();
             nrMatchers = 1;
             countMatches(1);
        

        This tripped an existing test (I don't know why we arent ever tripping the assert in TermScorer itself):

        [junit4:junit4] Suite: org.apache.lucene.search.TestBoolean2
        [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestBoolean2 -Dtests.method=testRandomQueries -Dtests.seed=A626109456992152 -Dtests.slow=true -Dtests.locale=fi -Dtests.timezone=Europe/Bratislava -Dtests.file.encoding=US-ASCII
        [junit4:junit4] FAILURE 0.58s J3 | TestBoolean2.testRandomQueries <<<
        [junit4:junit4]    > Throwable #1: java.lang.AssertionError
        [junit4:junit4]    > 	at __randomizedtesting.SeedInfo.seed([A626109456992152:F80DA0786E430CCC]:0)
        [junit4:junit4]    > 	at org.apache.lucene.search.DisjunctionSumScorer.afterNext(DisjunctionSumScorer.java:94)
        [junit4:junit4]    > 	at org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:82)
        [junit4:junit4]    > 	at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:282)
        [junit4:junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:574)
        [junit4:junit4]    > 	at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:287)
        [junit4:junit4]    > 	at org.apache.lucene.search.QueryUtils.checkFirstSkipTo(QueryUtils.java:373)
        [junit4:junit4]    > 	at org.apache.lucene.search.QueryUtils.check(QueryUtils.java:115)
        [junit4:junit4]    > 	at org.apache.lucene.search.QueryUtils.check(QueryUtils.java:118)
        [junit4:junit4]    > 	at org.apache.lucene.search.QueryUtils.check(QueryUtils.java:109)
        [junit4:junit4]    > 	at org.apache.lucene.search.TestBoolean2.testRandomQueries(TestBoolean2.java:275)
        
        Show
        Robert Muir added a comment - Its bogus to call score() on a DISI set at NO_MORE_DOCS, i figured I would add some asserts to see if we can catch this: Index: lucene/core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java =================================================================== --- lucene/core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java (revision 1386719) +++ lucene/core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java (working copy) @@ -91,6 +91,7 @@ private void afterNext() throws IOException { final Scorer sub = subScorers[0]; doc = sub.docID(); + assert doc != NO_MORE_DOCS; score = sub.score(); nrMatchers = 1; countMatches(1); This tripped an existing test (I don't know why we arent ever tripping the assert in TermScorer itself): [junit4:junit4] Suite: org.apache.lucene.search.TestBoolean2 [junit4:junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestBoolean2 -Dtests.method=testRandomQueries -Dtests.seed=A626109456992152 -Dtests.slow=true -Dtests.locale=fi -Dtests.timezone=Europe/Bratislava -Dtests.file.encoding=US-ASCII [junit4:junit4] FAILURE 0.58s J3 | TestBoolean2.testRandomQueries <<< [junit4:junit4] > Throwable #1: java.lang.AssertionError [junit4:junit4] > at __randomizedtesting.SeedInfo.seed([A626109456992152:F80DA0786E430CCC]:0) [junit4:junit4] > at org.apache.lucene.search.DisjunctionSumScorer.afterNext(DisjunctionSumScorer.java:94) [junit4:junit4] > at org.apache.lucene.search.DisjunctionSumScorer.nextDoc(DisjunctionSumScorer.java:82) [junit4:junit4] > at org.apache.lucene.search.BooleanScorer2.score(BooleanScorer2.java:282) [junit4:junit4] > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:574) [junit4:junit4] > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:287) [junit4:junit4] > at org.apache.lucene.search.QueryUtils.checkFirstSkipTo(QueryUtils.java:373) [junit4:junit4] > at org.apache.lucene.search.QueryUtils.check(QueryUtils.java:115) [junit4:junit4] > at org.apache.lucene.search.QueryUtils.check(QueryUtils.java:118) [junit4:junit4] > at org.apache.lucene.search.QueryUtils.check(QueryUtils.java:109) [junit4:junit4] > at org.apache.lucene.search.TestBoolean2.testRandomQueries(TestBoolean2.java:275)
        Hide
        Robert Muir added a comment -
        Index: core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java
        ===================================================================
        --- core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java	(revision 1386719)
        +++ core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java	(working copy)
        @@ -90,7 +90,9 @@
           
           private void afterNext() throws IOException {
             final Scorer sub = subScorers[0];
        +    int docBefore = doc;
             doc = sub.docID();
        +    assert doc != NO_MORE_DOCS : sub + ",numScorers=" + numScorers + ",docBefore=" + doc;
             score = sub.score();
             nrMatchers = 1;
             countMatches(1);
        

        Boiled it down a little bit more: the problem is not disjunctionsumscorer it seems.

        The problem is it already returned NO_MORE_DOCS and somebody called nextDoc() again on it after that (which is illegal)

        Show
        Robert Muir added a comment - Index: core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java =================================================================== --- core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java (revision 1386719) +++ core/src/java/org/apache/lucene/search/DisjunctionSumScorer.java (working copy) @@ -90,7 +90,9 @@ private void afterNext() throws IOException { final Scorer sub = subScorers[0]; + int docBefore = doc; doc = sub.docID(); + assert doc != NO_MORE_DOCS : sub + ",numScorers=" + numScorers + ",docBefore=" + doc; score = sub.score(); nrMatchers = 1; countMatches(1); Boiled it down a little bit more: the problem is not disjunctionsumscorer it seems. The problem is it already returned NO_MORE_DOCS and somebody called nextDoc() again on it after that (which is illegal)
        Hide
        Robert Muir added a comment -

        Here's a patch: I had confused myself before.

        I added an assert to BS2's nextDoc here, so we know if some consumer of it is calling nextDoc() after it already returned NO_MORE_DOCS.

        Thats fine: the bug was actually in BS2's nextDoc itself, it was missing an if. But from what I can tell this doesnt need to be per-scorer-per-doc, we can just do it per-doc in afterNext.

        I also added the assert to ConstantScorer so we feel ok with BS1.

        Separately we cant really test this situation very well as long as the javadocs for nextDoc say, Returns the following:

        • -1 or NO_MORE_DOCS if nextDoc() or
          advance(int) were not called yet.
        • NO_MORE_DOCS if the iterator has exhausted.
        • Otherwise it should return the doc ID it is currently on.

        This prevents us from being able to easily assert that nobody is calling nextDoc()/advance() after the enum is exhausted, since we cannot differentiate 'exhausted' from 'uninitialized'.

        I think we should clarify the javadocs, such that if nextDoc()/advance() are not called yet, you can still return NO_MORE_DOCS, but only if you somehow know you are exhausted-before-you-start. NO_MORE_DOCS should mean NO_MORE_DOCS.

        It could also be everyone reads it this way already, and I'm just being super-anal.

        Show
        Robert Muir added a comment - Here's a patch: I had confused myself before. I added an assert to BS2's nextDoc here, so we know if some consumer of it is calling nextDoc() after it already returned NO_MORE_DOCS. Thats fine: the bug was actually in BS2's nextDoc itself, it was missing an if. But from what I can tell this doesnt need to be per-scorer-per-doc, we can just do it per-doc in afterNext. I also added the assert to ConstantScorer so we feel ok with BS1. Separately we cant really test this situation very well as long as the javadocs for nextDoc say, Returns the following: -1 or NO_MORE_DOCS if nextDoc() or advance(int) were not called yet. NO_MORE_DOCS if the iterator has exhausted. Otherwise it should return the doc ID it is currently on. This prevents us from being able to easily assert that nobody is calling nextDoc()/advance() after the enum is exhausted, since we cannot differentiate 'exhausted' from 'uninitialized'. I think we should clarify the javadocs, such that if nextDoc()/advance() are not called yet, you can still return NO_MORE_DOCS, but only if you somehow know you are exhausted-before-you-start. NO_MORE_DOCS should mean NO_MORE_DOCS. It could also be everyone reads it this way already, and I'm just being super-anal.
        Hide
        Robert Muir added a comment -

        And for the record: DisjunctionMaxScorer doesnt have the bug: because it doesnt need to do additional things in its next() like compute the number of matching scorers, it just returns NO_MORE_DOCS here so everything is ok with that.

        Show
        Robert Muir added a comment - And for the record: DisjunctionMaxScorer doesnt have the bug: because it doesnt need to do additional things in its next() like compute the number of matching scorers, it just returns NO_MORE_DOCS here so everything is ok with that.
        Hide
        Michael McCandless added a comment -

        +1 for the patch: tricky!

        +1 to sharpen when a DocsEnum can return NO_MORE_DOCS before nextDoc: it should only be if the enum knows it has zero docs. But I'm not even sure we should allow that ... why not always make it -1 ...? We can do that separately...

        Show
        Michael McCandless added a comment - +1 for the patch: tricky! +1 to sharpen when a DocsEnum can return NO_MORE_DOCS before nextDoc: it should only be if the enum knows it has zero docs. But I'm not even sure we should allow that ... why not always make it -1 ...? We can do that separately...
        Hide
        Robert Muir added a comment -

        OK I will fix the bug (which i caused), and also test 3.x (fortunately this bug is not yet released there).

        But separately lets sharpen the javadocs, AssertingCodec and all that and really know things are right here.

        Show
        Robert Muir added a comment - OK I will fix the bug (which i caused), and also test 3.x (fortunately this bug is not yet released there). But separately lets sharpen the javadocs, AssertingCodec and all that and really know things are right here.
        Hide
        Robert Muir added a comment -

        I committed (also backported to 3.x branch, the bug does not affect any releases but would have affected unreleased 3.6.2 code, as it was caused by my previous bugfix: LUCENE-3505).

        I will tackle the NO_MORE_DOCS on a separate issue.

        Thank you, Liu Chao for reporting this!

        Show
        Robert Muir added a comment - I committed (also backported to 3.x branch, the bug does not affect any releases but would have affected unreleased 3.6.2 code, as it was caused by my previous bugfix: LUCENE-3505 ). I will tackle the NO_MORE_DOCS on a separate issue. Thank you, Liu Chao for reporting this!
        Hide
        Liu Chao added a comment -

        You guys are awesome.

        Show
        Liu Chao added a comment - You guys are awesome.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Michael McCandless
        http://svn.apache.org/viewvc?view=revision&revision=1397323

        LUCENE-4477/LUCENE-4401: remove dup test

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Michael McCandless http://svn.apache.org/viewvc?view=revision&revision=1397323 LUCENE-4477 / LUCENE-4401 : remove dup test
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Michael McCandless
        http://svn.apache.org/viewvc?view=revision&revision=1397284

        LUCENE-4477/LUCENE-4401: add test case

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Michael McCandless http://svn.apache.org/viewvc?view=revision&revision=1397284 LUCENE-4477 / LUCENE-4401 : add test case
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Robert Muir
        http://svn.apache.org/viewvc?view=revision&revision=1386770

        LUCENE-4401: don't call score() on NO_MORE_DOCS in DisjunctionSumScorer

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Robert Muir http://svn.apache.org/viewvc?view=revision&revision=1386770 LUCENE-4401 : don't call score() on NO_MORE_DOCS in DisjunctionSumScorer
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Robert Muir
            Reporter:
            Liu Chao
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development