Solr
  1. Solr
  2. SOLR-3790

NullPointerException and ConcurrentModificationException are thrown when searching with highlighter

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.5
    • Fix Version/s: 3.6.2, 4.0, 6.0
    • Component/s: highlighter
    • Labels:
      None
    • Environment:

      Ubuntu 12.04, Solr 3.5

      Description

      Hi, Im using Solr 3.5.

      I found strange phenomenon in solr log.
      This phenomenon has two exceptions that NullPointerException and ConcurrentModificationException.
      These exceptions are thrown by discrete threads.

      I think, The same two queries thrown at same time because error log recorded at same time. But I can't reproduce it.
      Does anyone have the same phenomenon?

      2012-08-21 00:07:49,868 [qtp979682126-953] INFO  SolrCore - [c3] webapp=/solr path=/select params={hl.tag.post=[%02]&sort=score+desc&hl.tag.pre=[%01]&hl.fl=*&wt=javabin&hl=true&defType=normalizeQParser&rows=40&version=2&fl=score,*&hl.useFastVectorHighlighter=true&start=0&q=hoge&fq=((ap.l:"771")+AND+(*:*))} hits=4 status=500 QTime=164 
      2012-08-21 00:07:49,868 [qtp979682126-938] INFO  SolrCore - [c3] webapp=/solr path=/select params={hl.tag.post=[%02]&sort=score+desc&hl.tag.pre=[%01]&hl.fl=*&wt=javabin&hl=true&defType=normalizeQParser&rows=12&version=2&fl=score,*&hl.useFastVectorHighlighter=true&start=0&q=hoge&fq=((ap.l:"771")+AND+(*:*))} hits=4 status=500 QTime=1094 
      2012-08-21 00:07:49,873 [qtp979682126-953] ERROR SolrDispatchFilter - java.util.ConcurrentModificationException
              at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:778)
              at java.util.LinkedList$ListItr.next(LinkedList.java:713)
              at org.apache.solr.highlight.SolrHighlighter.getHighlightFields(SolrHighlighter.java:106)
              at org.apache.solr.util.SolrPluginUtils.optimizePreFetchDocs(SolrPluginUtils.java:259)
              at org.apache.solr.handler.component.QueryComponent.doPrefetch(QueryComponent.java:478)
              at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:385)
              at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:194)
              at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
              at org.apache.solr.core.SolrCore.execute(SolrCore.java:1372)
              at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:486)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:520)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:973)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:417)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:907)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
              at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
              at org.eclipse.jetty.server.Server.handle(Server.java:350)
              at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:442)
              at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:941)
              at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:801)
              at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:224)
              at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:51)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:586)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:44)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)
              at java.lang.Thread.run(Thread.java:679)
      
      2012-08-21 00:07:49,873 [qtp979682126-938] ERROR SolrDispatchFilter - java.lang.NullPointerException
              at org.apache.solr.highlight.SolrHighlighter.getHighlightFields(SolrHighlighter.java:107)
              at org.apache.solr.util.SolrPluginUtils.optimizePreFetchDocs(SolrPluginUtils.java:259)
              at org.apache.solr.handler.component.QueryComponent.doPrefetch(QueryComponent.java:478)
              at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:385)
              at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:194)
              at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
              at org.apache.solr.core.SolrCore.execute(SolrCore.java:1372)
              at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
              at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1337)
              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:486)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:520)
              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:973)
              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:417)
              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:907)
              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
              at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110)
              at org.eclipse.jetty.server.Server.handle(Server.java:350)
              at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:442)
              at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:941)
              at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:801)
              at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:224)
              at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:51)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:586)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:44)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:598)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:533)
              at java.lang.Thread.run(Thread.java:679)
      

      This is rarely accident? These are discrete bug?

      See: Similar phenomenon
      http://lucene.472066.n3.nabble.com/Null-Pointer-Exception-in-SOLR-td3954952.html

        Activity

        Hide
        Koji Sekiguchi added a comment -

        Hmm, this is not good. I think storedHighlightFieldNames in SolrIndexSearcher could be initialized twice in getStoredHighlightFieldNames() method:

          public Collection<String> getStoredHighlightFieldNames() {
            if (storedHighlightFieldNames == null) {
              storedHighlightFieldNames = new LinkedList<String>();
              for (String fieldName : fieldNames) {
                try {
                  SchemaField field = schema.getField(fieldName);
                  if (field.stored() &&
                          ((field.getType() instanceof org.apache.solr.schema.TextField) ||
                          (field.getType() instanceof org.apache.solr.schema.StrField))) {
                    storedHighlightFieldNames.add(fieldName);
                  }
                } catch (RuntimeException e) { // getField() throws a SolrException, but it arrives as a RuntimeException
                    log.warn("Field \"" + fieldName + "\" found in index, but not defined in schema.");
                }
              }
            }
            return storedHighlightFieldNames;
          }
        

        I think the initialization block should go to constructor. Meanwhile, Aoki-san, you can avoid the bug by specifying the explicit field names for hl.fl parameter instead of using "*".

        Show
        Koji Sekiguchi added a comment - Hmm, this is not good. I think storedHighlightFieldNames in SolrIndexSearcher could be initialized twice in getStoredHighlightFieldNames() method: public Collection< String > getStoredHighlightFieldNames() { if (storedHighlightFieldNames == null ) { storedHighlightFieldNames = new LinkedList< String >(); for ( String fieldName : fieldNames) { try { SchemaField field = schema.getField(fieldName); if (field.stored() && ((field.getType() instanceof org.apache.solr.schema.TextField) || (field.getType() instanceof org.apache.solr.schema.StrField))) { storedHighlightFieldNames.add(fieldName); } } catch (RuntimeException e) { // getField() throws a SolrException, but it arrives as a RuntimeException log.warn( "Field \" " + fieldName + " \ " found in index, but not defined in schema." ); } } } return storedHighlightFieldNames; } I think the initialization block should go to constructor. Meanwhile, Aoki-san, you can avoid the bug by specifying the explicit field names for hl.fl parameter instead of using "*".
        Hide
        shou aoki added a comment -

        Thank you Mr.Sekiguchi.
        I read your blog often, I respect for you.

        OK, I'll investigate that hl.fl parameter assigned expect "*".
        Thank you.

        Show
        shou aoki added a comment - Thank you Mr.Sekiguchi. I read your blog often, I respect for you. OK, I'll investigate that hl.fl parameter assigned expect "*". Thank you.
        Hide
        Koji Sekiguchi added a comment -

        This bug has not been filed but fixed in trunk and branch_4x at r1231606:

        http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java?r1=1229401&r2=1231606&diff_format=h

        I'll backport it to 3.6 branch and close the ticket shortly.

        Show
        Koji Sekiguchi added a comment - This bug has not been filed but fixed in trunk and branch_4x at r1231606: http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java?r1=1229401&r2=1231606&diff_format=h I'll backport it to 3.6 branch and close the ticket shortly.
        Hide
        Koji Sekiguchi added a comment -

        Done.

        Show
        Koji Sekiguchi added a comment - Done.
        Hide
        Commit Tag Bot added a comment -

        [branch_4x commit] Koji Sekiguchi
        http://svn.apache.org/viewvc?view=revision&revision=1381495

        SOLR-3790: ConcurrentModificationException could be thrown when using hl.fl=*

        Show
        Commit Tag Bot added a comment - [branch_4x commit] Koji Sekiguchi http://svn.apache.org/viewvc?view=revision&revision=1381495 SOLR-3790 : ConcurrentModificationException could be thrown when using hl.fl=*
        Hide
        Uwe Schindler added a comment -

        Closed after release.

        Show
        Uwe Schindler added a comment - Closed after release.

          People

          • Assignee:
            Koji Sekiguchi
            Reporter:
            shou aoki
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development