Lucene - Core
  1. Lucene - Core
  2. LUCENE-2658

TestIndexWriterExceptions random failure: AIOOBE in ByteBlockPool.allocSlice

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.1, 4.0-ALPHA
    • Fix Version/s: 2.9.4, 3.0.3, 3.1, 4.0-ALPHA
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      TestIndexWriterExceptions threw this today, and its reproducable

      1. LUCENE-2658_30_test.patch
        4 kB
        Robert Muir
      2. LUCENE-2658_environment.patch
        15 kB
        Robert Muir
      3. LUCENE-2658.patch
        3 kB
        Michael McCandless

        Activity

        Hide
        Robert Muir added a comment -

        attached is my current modifications to trunk (unrelated to this failure completely)

        this is because, i have a "single test seed" that controls all behavior, so i want to make sure the random seed i give you will actually work.

        if you apply the patch, just run

        ant test-core -Dtestcase=TestIndexWriterExceptions -Dtests.seed=1285011726042

        junit-sequential:
            [junit] Testsuite: org.apache.lucene.index.TestIndexWriterExceptions
            [junit] Testcase: testRandomExceptionsThreads(org.apache.lucene.index.TestIndexWriterExceptions):   FAILED
            [junit] thread Indexer 0: hit unexpected failure
            [junit] junit.framework.AssertionFailedError: thread Indexer 0: hit unexpected failure
            [junit]     at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:773)
            [junit]     at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:746)
            [junit]     at org.apache.lucene.index.TestIndexWriterExceptions.testRandomExceptionsThreads(TestIndexWriterExceptio
        ns.java:195)
            [junit]
            [junit]
            [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 1.257 sec
            [junit]
            [junit] ------------- Standard Output ---------------
            [junit] Indexer 2: unexpected exception3
            [junit] java.lang.ArrayIndexOutOfBoundsException: 483
            [junit]     at org.apache.lucene.index.ByteSliceReader.nextSlice(ByteSliceReader.java:108)
            [junit]     at org.apache.lucene.index.ByteSliceReader.writeTo(ByteSliceReader.java:90)
            [junit]     at org.apache.lucene.index.TermVectorsTermsWriterPerField.finish(TermVectorsTermsWriterPerField.java:186
        )
            [junit]     at org.apache.lucene.index.TermsHashPerField.finish(TermsHashPerField.java:552)
            [junit]     at org.apache.lucene.index.TermsHashPerField.finish(TermsHashPerField.java:554)
            [junit]     at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:208)
            [junit]     at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:24
        8)
            [junit]     at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:839)
            [junit]     at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:820)
            [junit]     at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2162)
            [junit]     at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2134)
            [junit]     at org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:98
        )
            [junit] Indexer 0: unexpected exception3
            [junit] java.lang.ArrayIndexOutOfBoundsException: 507
            [junit]     at org.apache.lucene.index.ByteSliceReader.nextSlice(ByteSliceReader.java:108)
            [junit]     at org.apache.lucene.index.ByteSliceReader.writeTo(ByteSliceReader.java:90)
            [junit]     at org.apache.lucene.index.TermVectorsTermsWriterPerField.finish(TermVectorsTermsWriterPerField.java:186
        )
            [junit]     at org.apache.lucene.index.TermsHashPerField.finish(TermsHashPerField.java:552)
            [junit]     at org.apache.lucene.index.TermsHashPerField.finish(TermsHashPerField.java:554)
            [junit]     at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:208)
            [junit]     at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:24
        8)
            [junit]     at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:839)
            [junit]     at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:820)
            [junit]     at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2162)
            [junit]     at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2134)
            [junit]     at org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:98
        )
            [junit] Indexer 1: unexpected exception3
            [junit] java.lang.ArrayIndexOutOfBoundsException: 15
            [junit]     at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:122)
            [junit]     at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:526)
            [junit]     at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:547)
            [junit]     at org.apache.lucene.index.TermVectorsTermsWriterPerField.newTerm(TermVectorsTermsWriterPerField.java:22
        5)
            [junit]     at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:375)
            [junit]     at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:513)
            [junit]     at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:173)
            [junit]     at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:24
        8)
            [junit]     at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:839)
            [junit]     at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:820)
            [junit]     at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2162)
            [junit]     at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2134)
            [junit]     at org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:98
        )
            [junit] Indexer 3: unexpected exception3
            [junit] java.lang.ArrayIndexOutOfBoundsException: 15
            [junit]     at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:122)
            [junit]     at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:526)
            [junit]     at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:547)
            [junit]     at org.apache.lucene.index.TermVectorsTermsWriterPerField.newTerm(TermVectorsTermsWriterPerField.java:22
        5)
            [junit]     at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:375)
            [junit]     at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:513)
            [junit]     at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:173)
            [junit]     at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:24
        8)
            [junit]     at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:839)
            [junit]     at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:820)
            [junit]     at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2162)
            [junit]     at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2134)
            [junit]     at org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:98
        )
            [junit] NOTE: random codec of testcase 'testRandomExceptionsThreads' was: MockVariableIntBlock(baseBlockSize=67)
            [junit] NOTE: random locale of testcase 'testRandomExceptionsThreads' was: ar_AE
            [junit] NOTE: random timezone of testcase 'testRandomExceptionsThreads' was: America/Winnipeg
            [junit] ------------- ---------------- ---------------
            [junit] Test org.apache.lucene.index.TestIndexWriterExceptions FAILED
        
        Show
        Robert Muir added a comment - attached is my current modifications to trunk (unrelated to this failure completely) this is because, i have a "single test seed" that controls all behavior, so i want to make sure the random seed i give you will actually work. if you apply the patch, just run ant test-core -Dtestcase=TestIndexWriterExceptions -Dtests.seed=1285011726042 junit-sequential: [junit] Testsuite: org.apache.lucene.index.TestIndexWriterExceptions [junit] Testcase: testRandomExceptionsThreads(org.apache.lucene.index.TestIndexWriterExceptions): FAILED [junit] thread Indexer 0: hit unexpected failure [junit] junit.framework.AssertionFailedError: thread Indexer 0: hit unexpected failure [junit] at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:773) [junit] at org.apache.lucene.util.LuceneTestCase$LuceneTestCaseRunner.runChild(LuceneTestCase.java:746) [junit] at org.apache.lucene.index.TestIndexWriterExceptions.testRandomExceptionsThreads(TestIndexWriterExceptio ns.java:195) [junit] [junit] [junit] Tests run: 2, Failures: 1, Errors: 0, Time elapsed: 1.257 sec [junit] [junit] ------------- Standard Output --------------- [junit] Indexer 2: unexpected exception3 [junit] java.lang.ArrayIndexOutOfBoundsException: 483 [junit] at org.apache.lucene.index.ByteSliceReader.nextSlice(ByteSliceReader.java:108) [junit] at org.apache.lucene.index.ByteSliceReader.writeTo(ByteSliceReader.java:90) [junit] at org.apache.lucene.index.TermVectorsTermsWriterPerField.finish(TermVectorsTermsWriterPerField.java:186 ) [junit] at org.apache.lucene.index.TermsHashPerField.finish(TermsHashPerField.java:552) [junit] at org.apache.lucene.index.TermsHashPerField.finish(TermsHashPerField.java:554) [junit] at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:208) [junit] at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:24 8) [junit] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:839) [junit] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:820) [junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2162) [junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2134) [junit] at org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:98 ) [junit] Indexer 0: unexpected exception3 [junit] java.lang.ArrayIndexOutOfBoundsException: 507 [junit] at org.apache.lucene.index.ByteSliceReader.nextSlice(ByteSliceReader.java:108) [junit] at org.apache.lucene.index.ByteSliceReader.writeTo(ByteSliceReader.java:90) [junit] at org.apache.lucene.index.TermVectorsTermsWriterPerField.finish(TermVectorsTermsWriterPerField.java:186 ) [junit] at org.apache.lucene.index.TermsHashPerField.finish(TermsHashPerField.java:552) [junit] at org.apache.lucene.index.TermsHashPerField.finish(TermsHashPerField.java:554) [junit] at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:208) [junit] at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:24 8) [junit] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:839) [junit] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:820) [junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2162) [junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2134) [junit] at org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:98 ) [junit] Indexer 1: unexpected exception3 [junit] java.lang.ArrayIndexOutOfBoundsException: 15 [junit] at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:122) [junit] at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:526) [junit] at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:547) [junit] at org.apache.lucene.index.TermVectorsTermsWriterPerField.newTerm(TermVectorsTermsWriterPerField.java:22 5) [junit] at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:375) [junit] at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:513) [junit] at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:173) [junit] at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:24 8) [junit] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:839) [junit] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:820) [junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2162) [junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2134) [junit] at org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:98 ) [junit] Indexer 3: unexpected exception3 [junit] java.lang.ArrayIndexOutOfBoundsException: 15 [junit] at org.apache.lucene.index.ByteBlockPool.allocSlice(ByteBlockPool.java:122) [junit] at org.apache.lucene.index.TermsHashPerField.writeByte(TermsHashPerField.java:526) [junit] at org.apache.lucene.index.TermsHashPerField.writeVInt(TermsHashPerField.java:547) [junit] at org.apache.lucene.index.TermVectorsTermsWriterPerField.newTerm(TermVectorsTermsWriterPerField.java:22 5) [junit] at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:375) [junit] at org.apache.lucene.index.TermsHashPerField.add(TermsHashPerField.java:513) [junit] at org.apache.lucene.index.DocInverterPerField.processFields(DocInverterPerField.java:173) [junit] at org.apache.lucene.index.DocFieldProcessorPerThread.processDocument(DocFieldProcessorPerThread.java:24 8) [junit] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:839) [junit] at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:820) [junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2162) [junit] at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:2134) [junit] at org.apache.lucene.index.TestIndexWriterExceptions$IndexerThread.run(TestIndexWriterExceptions.java:98 ) [junit] NOTE: random codec of testcase 'testRandomExceptionsThreads' was: MockVariableIntBlock(baseBlockSize=67) [junit] NOTE: random locale of testcase 'testRandomExceptionsThreads' was: ar_AE [junit] NOTE: random timezone of testcase 'testRandomExceptionsThreads' was: America/Winnipeg [junit] ------------- ---------------- --------------- [junit] Test org.apache.lucene.index.TestIndexWriterExceptions FAILED
        Hide
        Robert Muir added a comment -

        i tested this on 3.x too (actually on both 3.x and trunk you do not need my patch, just use the seed)

        Show
        Robert Muir added a comment - i tested this on 3.x too (actually on both 3.x and trunk you do not need my patch, just use the seed)
        Hide
        Michael McCandless added a comment -

        This was a real issue!

        It happens if you hit an exception while processing term vectors, and, your docs have multiple fields with term vectors enabled.

        Show
        Michael McCandless added a comment - This was a real issue! It happens if you hit an exception while processing term vectors, and, your docs have multiple fields with term vectors enabled.
        Hide
        Tim Smith added a comment -

        Is this related to/same as LUCENE-2501?

        Show
        Tim Smith added a comment - Is this related to/same as LUCENE-2501 ?
        Hide
        Robert Muir added a comment -

        Tim, i didnt see your problem coming from termvectorswriter... i think it might be a different bug.

        Show
        Robert Muir added a comment - Tim, i didnt see your problem coming from termvectorswriter... i think it might be a different bug.
        Hide
        Michael McCandless added a comment -

        Unfortunately, no, not directly anyway... – this bug is specific to term vectors.

        Under the hood, term vectors and postings (doc/frq/pos) use the same class (ByteBlockPool), and in this case, after an exception, the term vectors code was failing to reset the ByteBlockPool.

        Though it is possible the same idea is happening on LUCENE-2501, ie, an exception at a bad time leaving the ByteBlockPool in the same state... can you post the infoStream output on LUCENE-2501?

        Show
        Michael McCandless added a comment - Unfortunately, no, not directly anyway... – this bug is specific to term vectors. Under the hood, term vectors and postings (doc/frq/pos) use the same class (ByteBlockPool), and in this case, after an exception, the term vectors code was failing to reset the ByteBlockPool. Though it is possible the same idea is happening on LUCENE-2501 , ie, an exception at a bad time leaving the ByteBlockPool in the same state... can you post the infoStream output on LUCENE-2501 ?
        Hide
        Tim Smith added a comment -

        sadly i haven't been able to gather the infostream for LUCENE-2501
        there's a comment on LUCENE-2501 that seems to indicate the exception that started it all though (CorruptIndexException: docs out of order (607 <= 607 ))

        Show
        Tim Smith added a comment - sadly i haven't been able to gather the infostream for LUCENE-2501 there's a comment on LUCENE-2501 that seems to indicate the exception that started it all though (CorruptIndexException: docs out of order (607 <= 607 ))
        Hide
        Robert Muir added a comment -

        This test fails about 1% of the time currently... i applied the patch and ran it many times:

        junit-sequential:
        [junit] Testsuite: org.apache.lucene.index.TestIndexWriterExceptions
        [junit] Tests run: 1800, Failures: 0, Errors: 0, Time elapsed: 1,291.131 sec

        +1 to commit

        Show
        Robert Muir added a comment - This test fails about 1% of the time currently... i applied the patch and ran it many times: junit-sequential: [junit] Testsuite: org.apache.lucene.index.TestIndexWriterExceptions [junit] Tests run: 1800, Failures: 0, Errors: 0, Time elapsed: 1,291.131 sec +1 to commit
        Hide
        Robert Muir added a comment -

        reopening for possible 2.9.4/3.0.3 backport.

        Show
        Robert Muir added a comment - reopening for possible 2.9.4/3.0.3 backport.
        Hide
        Robert Muir added a comment -

        I'll hack on the 3.0.x/2.9.x tests and see if i can make them fail (I think i can just turn on term vecs on more fields... but will see)

        Show
        Robert Muir added a comment - I'll hack on the 3.0.x/2.9.x tests and see if i can make them fail (I think i can just turn on term vecs on more fields... but will see)
        Hide
        Robert Muir added a comment -

        ok, here is a test that reproduces the failure on the 3.0 branch.

        I'll see about merging mike's fix now

        Show
        Robert Muir added a comment - ok, here is a test that reproduces the failure on the 3.0 branch. I'll see about merging mike's fix now
        Hide
        Robert Muir added a comment -

        Committed revision 1028827 (3.0.x)
        Committed revision 1028849 (2.9.x)

        Show
        Robert Muir added a comment - Committed revision 1028827 (3.0.x) Committed revision 1028849 (2.9.x)

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development