Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-6065

Solr should give you clear error if you try to add too many docs

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: update
    • Labels:
      None

      Description

      LUCENE-5843 recently fixed IndexWriter to throw a specific exception in the event that an index grows too big, preventing people from creating indexes they can't later search. That issue also added a new test-only setter that can be used to lower the limit for hte purposes of testing.

      For the Solr side of things, we should primarily focus on making sure we have some cloud based tests that set the lower limit and then verify that users get clean error messages when they exceed it for a single shard.

      original bug report

      yamazaki reported an error on solr-user where, on opening a new searcher, he got an IAE from BaseCompositeReader because the numDocs was greater then Integer.MAX_VALUE.

      I'm surprised that in a straight forward setup (ie: no "AddIndex" merging) IndexWriter will even let you add more docs then max int. We should investigate if this makes sense and either add logic in IndexWriter to prevent this from happening, or add logic to Solr's UpdateHandler to prevent things from getting that far.

      ie: we should be failing to "add" too many documents, and leaving the index usable – not accepting the add and leaving hte index in an unusable state.

      stack trace reported by user...

      ERROR org.apache.solr.core.CoreContainer  – Unable to create core: collection1
      org.apache.solr.common.SolrException: Error opening new searcher
          at org.apache.solr.core.SolrCore.<init>(SolrCore.java:821)
          at org.apache.solr.core.SolrCore.<init>(SolrCore.java:618)
          at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949)
          at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984)
          at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597)
          at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
          at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
          at java.util.concurrent.FutureTask.run(FutureTask.java:138)
          at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
          at java.lang.Thread.run(Thread.java:662)
      Caused by: org.apache.solr.common.SolrException: Error opening new searcher
          at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1438)
          at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1550)
          at org.apache.solr.core.SolrCore.<init>(SolrCore.java:796)
          ... 13 more
      Caused by: org.apache.solr.common.SolrException: Error opening Reader
          at org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:172)
          at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:183)
          at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:179)
          at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1414)
          ... 15 more
      Caused by: java.lang.IllegalArgumentException: Too many documents,
      composite IndexReaders cannot exceed 2147483647
          at org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:77)
          at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:368)
          at org.apache.lucene.index.StandardDirectoryReader.<init>(StandardDirectoryReader.java:42)
          at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:71)
          at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783)
          at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
          at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:88)
          at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:34)
          at org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:169)
          ... 18 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                hossman Chris M. Hostetter
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: