Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9956

Solr java.lang.ArrayIndexOutOfBoundsException when indexed large amount of documents

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsAdd voteVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 6.2.1, 6.3
    • None
    • SolrCloud
    • Ubuntu 14.04.4 LTS

    Description

      I'm using solr 6.3.0. I indexed a big amount of docuements into one solr collection with one shard, it's 60G in the disk, which has around 2506889 documents.

      I frequently get the ArrayIndexOutOfBoundsException when I send a simple stats request, for example:

      http://localhost:8983/solr/staging-update/select?start=0&rows=0&version=2.2&q=*:*&stats=true&timeAllowed=60000&wt=json&stats.field=asp_community_facet&stats.field=asp_group_facet

      The solr log capture following exception as well as in the response like below:

      {
      
          "responseHeader": {
              "zkConnected": true,
              "status": 500,
              "QTime": 11,
              "params": {
                  "q": "*:*",
                  "stats": "true",
                  "start": "0",
                  "timeAllowed": "60000",
                  "rows": "0",
                  "version": "2.2",
                  "wt": "json",
                  "stats.field": [
                      "asp_community_facet",
                      "asp_group_facet"
                  ]
              }
          },
          "response": {
              "numFound": 2506211,
              "start": 0,
              "docs": [ ]
          },
          "error": {
              "msg": "28",
              "trace": "java.lang.ArrayIndexOutOfBoundsException: 28\n\tat org.apache.solr.request.DocValuesStats.accumMulti(DocValuesStats.java:213)\n\tat org.apache.solr.request.DocValuesStats.getCounts(DocValuesStats.java:135)\n\tat org.apache.solr.handler.component.StatsField.computeLocalStatsValues(StatsField.java:424)\n\tat org.apache.solr.handler.component.StatsComponent.process(StatsComponent.java:58)\n\tat org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2213)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:518)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)\n\tat java.lang.Thread.run(Thread.java:745)\n",
              "code": 500
          }
      
      }
      

      I tried to remove some documents by reduce the document amount to 2334089, then the query get correct response, like below:

      {
      
          "responseHeader": {
              "zkConnected": true,
              "status": 0,
              "QTime": 154,
              "params": {
                  "q": "*:*",
                  "stats": "true",
                  "start": "0",
                  "timeAllowed": "60000",
                  "rows": "0",
                  "version": "2.2",
                  "wt": "json",
                  "stats.field": [
                      "asp_community_facet",
                      "asp_group_facet"
                  ]
              }
          },
          "response": {
              "numFound": 2334089,
              "start": 0,
              "docs": [ ]
          },
          "stats": {
              "stats_fields": {
                  "asp_community_facet": {
                      "min": "American Film",
                      "max": "World Music",
                      "count": 1954337,
                      "missing": 407503
                  },
                  "asp_group_facet": {
                      "min": "All Disciplines",
                      "max": "Social Sciences",
                      "count": 2407087,
                      "missing": 354869
                  }
              }
          }
      }
      

      My solr memory setting in /etc/default/solr.in.sh is like below:

      SOLR_JAVA_MEM="-Xms30G -Xmx60G"
      

      So, I thought it's probably not a memory issue. By the way, I have a solr 5.3 server with the same amount of documents (2506889) in a one shard core which does not have this exception.

      JiaJun

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned Assign to me
            jiajun Zhu JiaJun

            Dates

              Created:
              Updated:

              Slack

                Issue deployment