Lucene - Core
  1. Lucene - Core
  2. LUCENE-2420

"fdx size mismatch" overflow causes RuntimeException

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.1
    • Fix Version/s: None
    • Component/s: core/index
    • Labels:
      None
    • Environment:

      CentOS 5.4

    • Lucene Fields:
      New

      Description

      I just saw the following error:

      java.lang.RuntimeException: after flush: fdx size mismatch: -512764976 docs vs 30257618564 length in bytes of _0.fdx file exists?=true
      at org.apache.lucene.index.StoredFieldsWriter.closeDocStore(StoredFieldsWriter.java:97)
      at org.apache.lucene.index.DocFieldProcessor.closeDocStore(DocFieldProcessor.java:51)
      at org.apache.lucene.index.DocumentsWriter.closeDocStore(DocumentsWriter.java:371)
      at org.apache.lucene.index.IndexWriter.flushDocStores(IndexWriter.java:1724)
      at org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3565)
      at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3491)
      at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3482)
      at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1658)
      at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1621)
      at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1585)

      Note the negative SegmentWriteState.numDocsInStore. I assume this is because Lucene has a limit of 2 ^ 31 - 1 = 2147483647 (sizeof(int)) documents per index, though I couldn't find this documented clearly anywhere. It would have been nice to get this error earlier, back when I exceeded the limit, rather than now, after a bunch of indexing that was apparently doomed to fail.

      Hence, two suggestions:

      • State clearly somewhere that the maximum number of documents in a Lucene index is sizeof(int).
      • Throw an exception when an IndexWriter first exceeds this number rather than only on close.

        Activity

        Steven Bethard created issue -
        Hide
        Shai Erera added a comment -

        I remember that the Integer.MAX_VAL is documented somewhere. I can try to look it up later. But a lot of places in the API use int as the doc Id (IndexReader, ScoreDoc, even IndexWriter.max/numDocs()), and so I think there's a strong hint about that limitation.

        As for throwing the exception sooner, I don't think it will be correct. IndexWriter implements transaction semantics. Until you call commit() or close(), whatever operations you've made are not officially in the index yet. If your JVM dies before that, they will get lost. Therefore throwing the exception earlier would be wrong. Also think that you intend to index 1000 docs and delete 100,000. Would you want to get the exception while adding the docs, knowing that you are about to delete much more soon?

        Show
        Shai Erera added a comment - I remember that the Integer.MAX_VAL is documented somewhere. I can try to look it up later. But a lot of places in the API use int as the doc Id (IndexReader, ScoreDoc, even IndexWriter.max/numDocs()), and so I think there's a strong hint about that limitation. As for throwing the exception sooner, I don't think it will be correct. IndexWriter implements transaction semantics. Until you call commit() or close(), whatever operations you've made are not officially in the index yet. If your JVM dies before that, they will get lost. Therefore throwing the exception earlier would be wrong. Also think that you intend to index 1000 docs and delete 100,000. Would you want to get the exception while adding the docs, knowing that you are about to delete much more soon?
        Hide
        Steven Bethard added a comment -

        I finally found the documentation saying that the maximum number of documents is ~274 billion:
        http://lucene.apache.org/java/3_0_1/fileformats.html

        Google queries that failed to find this:
        lucene index maximum documents
        lucene document limit
        lucene max docs

        Maybe a bullet could be added to the FAQ (which does turn up for most of these queries)?
        http://wiki.apache.org/lucene-java/LuceneFAQ

        As far as the exception goes, regardless of the transaction semantics, I really don't think the code works correctly after numeric overflow. Once SegmentWriteState.numDocsInStore is negative, I would expect code like StoredFieldsWriter.flush to fail:

        synchronized public void flush(SegmentWriteState state) throws IOException {
        if (state.numDocsInStore > 0) {
        ...

        Perhaps I'm wrong, but it seems like this is going to do the wrong thing when SegmentWriteState.numDocsInStore is negative. If I'm not wrong, then it seems sensible to me to raise an exception on numeric overflow.

        Show
        Steven Bethard added a comment - I finally found the documentation saying that the maximum number of documents is ~274 billion: http://lucene.apache.org/java/3_0_1/fileformats.html Google queries that failed to find this: lucene index maximum documents lucene document limit lucene max docs Maybe a bullet could be added to the FAQ (which does turn up for most of these queries)? http://wiki.apache.org/lucene-java/LuceneFAQ As far as the exception goes, regardless of the transaction semantics, I really don't think the code works correctly after numeric overflow. Once SegmentWriteState.numDocsInStore is negative, I would expect code like StoredFieldsWriter.flush to fail: synchronized public void flush(SegmentWriteState state) throws IOException { if (state.numDocsInStore > 0) { ... Perhaps I'm wrong, but it seems like this is going to do the wrong thing when SegmentWriteState.numDocsInStore is negative. If I'm not wrong, then it seems sensible to me to raise an exception on numeric overflow.
        Hide
        Shai Erera added a comment -

        That documentation discusses the limitation around the number of unique terms Lucene can handle, which sums up to ~274 billion: "which means the maximum number of unique terms in any single index segment is ~2.1 billion times the term index interval (default 128) = ~274 billion".

        One line below this you can find this documentation: "Similarly, Lucene uses a Java int to refer to document numbers, and the index file format uses an Int32 on-disk to store document numbers" which suggests the Integer.MAX_VAL limitation ... but I agree it could have been spelled out more clearly.

        If numDocsInStore is negative, SegmentFieldsWriter.flush() doesn't do anything ... you mean that that code should throw the exception?

        Show
        Shai Erera added a comment - That documentation discusses the limitation around the number of unique terms Lucene can handle, which sums up to ~274 billion: " which means the maximum number of unique terms in any single index segment is ~2.1 billion times the term index interval (default 128) = ~274 billion ". One line below this you can find this documentation: " Similarly, Lucene uses a Java int to refer to document numbers, and the index file format uses an Int32 on-disk to store document numbers " which suggests the Integer.MAX_VAL limitation ... but I agree it could have been spelled out more clearly. If numDocsInStore is negative, SegmentFieldsWriter.flush() doesn't do anything ... you mean that that code should throw the exception?
        Hide
        Steven Bethard added a comment -

        Thanks, yeah, something more explicit and easier to find with Google would be good.

        Yeah, I mean that code should throw the exception. Actually, anywhere that checks the value of numDocsInStore could/should throw the exception, e.g.

        synchronized public void flush(SegmentWriteState state) throws IOException {
        if (state.numDocsInStore > 0) {

        synchronized public void closeDocStore(SegmentWriteState state) throws IOException {
        final int inc = state.numDocsInStore - lastDocID;
        if (inc > 0) {

        Don't know if SegmentWriteState.numDocsInStore is used anywhere else (I haven't loaded it into an IDE to look at files other than StoredFieldsWriter), but in at least these two cases it would be easy to give an exception explaining that the maximum number of documents had been exceeded.

        Alternatively, you could try to fix the code to work correctly after integer overflow (to support the transaction use case you described above) though it's less obvious to me how to do that correctly everywhere. Probably involves changing some "> 0"s into "!= 0"s and being careful in a few other ways.

        Show
        Steven Bethard added a comment - Thanks, yeah, something more explicit and easier to find with Google would be good. Yeah, I mean that code should throw the exception. Actually, anywhere that checks the value of numDocsInStore could/should throw the exception, e.g. synchronized public void flush(SegmentWriteState state) throws IOException { if (state.numDocsInStore > 0) { synchronized public void closeDocStore(SegmentWriteState state) throws IOException { final int inc = state.numDocsInStore - lastDocID; if (inc > 0) { Don't know if SegmentWriteState.numDocsInStore is used anywhere else (I haven't loaded it into an IDE to look at files other than StoredFieldsWriter), but in at least these two cases it would be easy to give an exception explaining that the maximum number of documents had been exceeded. Alternatively, you could try to fix the code to work correctly after integer overflow (to support the transaction use case you described above) though it's less obvious to me how to do that correctly everywhere. Probably involves changing some "> 0"s into "!= 0"s and being careful in a few other ways.
        Mark Thomas made changes -
        Field Original Value New Value
        Workflow jira [ 12509617 ] Default workflow, editable Closed status [ 12563580 ]
        Mark Thomas made changes -
        Workflow Default workflow, editable Closed status [ 12563580 ] jira [ 12584315 ]

          People

          • Assignee:
            Unassigned
            Reporter:
            Steven Bethard
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development