[SOLR-6065] Solr should give you clear error if you try to add too many docs

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: update
• Assignee: Unassigned
• Reporter: Chris M. Hostetter
• Created: 13/May/14 15:50
• Updated: 11/Jan/18 20:18

Description

LUCENE-5843 recently fixed IndexWriter to throw a specific exception in the event that an index grows too big, preventing people from creating indexes they can't later search. That issue also added a new test-only setter that can be used to lower the limit for hte purposes of testing.

For the Solr side of things, we should primarily focus on making sure we have some cloud based tests that set the lower limit and then verify that users get clean error messages when they exceed it for a single shard.

original bug report

yamazaki reported an error on solr-user where, on opening a new searcher, he got an IAE from BaseCompositeReader because the numDocs was greater then Integer.MAX_VALUE.

I'm surprised that in a straight forward setup (ie: no "AddIndex" merging) IndexWriter will even let you add more docs then max int. We should investigate if this makes sense and either add logic in IndexWriter to prevent this from happening, or add logic to Solr's UpdateHandler to prevent things from getting that far.

ie: we should be failing to "add" too many documents, and leaving the index usable – not accepting the add and leaving hte index in an unusable state.

stack trace reported by user...

ERROR org.apache.solr.core.CoreContainer  – Unable to create core: collection1
org.apache.solr.common.SolrException: Error opening new searcher
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:821)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:618)
    at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:949)
    at org.apache.solr.core.CoreContainer.create(CoreContainer.java:984)
    at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:597)
    at org.apache.solr.core.CoreContainer$2.call(CoreContainer.java:592)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
    at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1438)
    at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1550)
    at org.apache.solr.core.SolrCore.<init>(SolrCore.java:796)
    ... 13 more
Caused by: org.apache.solr.common.SolrException: Error opening Reader
    at org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:172)
    at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:183)
    at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:179)
    at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1414)
    ... 15 more
Caused by: java.lang.IllegalArgumentException: Too many documents,
composite IndexReaders cannot exceed 2147483647
    at org.apache.lucene.index.BaseCompositeReader.<init>(BaseCompositeReader.java:77)
    at org.apache.lucene.index.DirectoryReader.<init>(DirectoryReader.java:368)
    at org.apache.lucene.index.StandardDirectoryReader.<init>(StandardDirectoryReader.java:42)
    at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:71)
    at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783)
    at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52)
    at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:88)
    at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:34)
    at org.apache.solr.search.SolrIndexSearcher.getReader(SolrIndexSearcher.java:169)
    ... 18 more

Issue Links

• depends upon: LUCENE-5843 IndexWriter should refuse to create an index with more than INT_MAX docs