Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9956

Solr java.lang.ArrayIndexOutOfBoundsException when indexed large amount of documents

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 6.2.1, 6.3
    • Fix Version/s: None
    • Component/s: SolrCloud
    • Labels:
    • Environment:

      Ubuntu 14.04.4 LTS

      Description

      I'm using solr 6.3.0. I indexed a big amount of docuements into one solr collection with one shard, it's 60G in the disk, which has around 2506889 documents.

      I frequently get the ArrayIndexOutOfBoundsException when I send a simple stats request, for example:

      http://localhost:8983/solr/staging-update/select?start=0&rows=0&version=2.2&q=*:*&stats=true&timeAllowed=60000&wt=json&stats.field=asp_community_facet&stats.field=asp_group_facet

      The solr log capture following exception as well as in the response like below:

      {
      
          "responseHeader": {
              "zkConnected": true,
              "status": 500,
              "QTime": 11,
              "params": {
                  "q": "*:*",
                  "stats": "true",
                  "start": "0",
                  "timeAllowed": "60000",
                  "rows": "0",
                  "version": "2.2",
                  "wt": "json",
                  "stats.field": [
                      "asp_community_facet",
                      "asp_group_facet"
                  ]
              }
          },
          "response": {
              "numFound": 2506211,
              "start": 0,
              "docs": [ ]
          },
          "error": {
              "msg": "28",
              "trace": "java.lang.ArrayIndexOutOfBoundsException: 28\n\tat org.apache.solr.request.DocValuesStats.accumMulti(DocValuesStats.java:213)\n\tat org.apache.solr.request.DocValuesStats.getCounts(DocValuesStats.java:135)\n\tat org.apache.solr.handler.component.StatsField.computeLocalStatsValues(StatsField.java:424)\n\tat org.apache.solr.handler.component.StatsComponent.process(StatsComponent.java:58)\n\tat org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:295)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:153)\n\tat org.apache.solr.core.SolrCore.execute(SolrCore.java:2213)\n\tat org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:654)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)\n\tat org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:518)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)\n\tat org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)\n\tat org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)\n\tat java.lang.Thread.run(Thread.java:745)\n",
              "code": 500
          }
      
      }
      

      I tried to remove some documents by reduce the document amount to 2334089, then the query get correct response, like below:

      {
      
          "responseHeader": {
              "zkConnected": true,
              "status": 0,
              "QTime": 154,
              "params": {
                  "q": "*:*",
                  "stats": "true",
                  "start": "0",
                  "timeAllowed": "60000",
                  "rows": "0",
                  "version": "2.2",
                  "wt": "json",
                  "stats.field": [
                      "asp_community_facet",
                      "asp_group_facet"
                  ]
              }
          },
          "response": {
              "numFound": 2334089,
              "start": 0,
              "docs": [ ]
          },
          "stats": {
              "stats_fields": {
                  "asp_community_facet": {
                      "min": "American Film",
                      "max": "World Music",
                      "count": 1954337,
                      "missing": 407503
                  },
                  "asp_group_facet": {
                      "min": "All Disciplines",
                      "max": "Social Sciences",
                      "count": 2407087,
                      "missing": 354869
                  }
              }
          }
      }
      

      My solr memory setting in /etc/default/solr.in.sh is like below:

      SOLR_JAVA_MEM="-Xms30G -Xmx60G"
      

      So, I thought it's probably not a memory issue. By the way, I have a solr 5.3 server with the same amount of documents (2506889) in a one shard core which does not have this exception.

      JiaJun

        Attachments

        1. solr.log
          3.30 MB
          Zhu JiaJun

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              jiajun Zhu JiaJun
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: