Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-7887

TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreadsOnlyOnce() failure: numTerms2=0 vs -8

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 7.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Non-reproducing failure from https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1338/:

      Checking out Revision 9f56698d33d1db9fab6a0d6f63b360b334f71583 (refs/remotes/origin/master)
      [...]
         [junit4] Suite: org.apache.lucene.index.TestIndexWriterWithThreads
         [junit4]   1> Thread-12393: ERROR: unexpected Throwable:
         [junit4]   1> java.lang.AssertionError: numTerms2=0 vs -8
         [junit4]   1> 	at org.apache.lucene.index.BufferedUpdatesStream.checkDeleteStats(BufferedUpdatesStream.java:376)
         [junit4]   1> 	at org.apache.lucene.index.BufferedUpdatesStream.push(BufferedUpdatesStream.java:85)
         [junit4]   1> 	at org.apache.lucene.index.IndexWriter.publishFrozenUpdates(IndexWriter.java:2655)
         [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterFlushQueue$FlushTicket.finishFlush(DocumentsWriterFlushQueue.java:205)
         [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterFlushQueue$SegmentFlushTicket.publish(DocumentsWriterFlushQueue.java:248)
         [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterFlushQueue.innerPurge(DocumentsWriterFlushQueue.java:116)
         [junit4]   1> 	at org.apache.lucene.index.DocumentsWriterFlushQueue.forcePurge(DocumentsWriterFlushQueue.java:138)
         [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter.purgeBuffer(DocumentsWriter.java:200)
         [junit4]   1> 	at org.apache.lucene.index.IndexWriter.purge(IndexWriter.java:5004)
         [junit4]   1> 	at org.apache.lucene.index.DocumentsWriter$ForcedPurgeEvent.process(DocumentsWriter.java:751)
         [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5061)
         [junit4]   1> 	at org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5049)
         [junit4]   1> 	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1719)
         [junit4]   1> 	at org.apache.lucene.index.TestIndexWriterWithThreads$IndexerThread.run(TestIndexWriterWithThreads.java:86)
         [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
         [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterWithThreads -Dtests.method=testIOExceptionDuringWriteSegmentWithThreadsOnlyOnce -Dtests.seed=66484822FD0371B5 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt -Dtests.locale=es-CR -Dtests.timezone=America/Manaus -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
         [junit4] FAILURE 0.06s J2 | TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreadsOnlyOnce <<<
         [junit4]    > Throwable #1: java.lang.AssertionError: hit unexpected Throwable
         [junit4]    > 	at __randomizedtesting.SeedInfo.seed([66484822FD0371B5:2A23C9E43BA35250]:0)
         [junit4]    > 	at org.apache.lucene.index.TestIndexWriterWithThreads._testMultipleThreadsFailure(TestIndexWriterWithThreads.java:300)
         [junit4]    > 	at org.apache.lucene.index.TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreadsOnlyOnce(TestIndexWriterWithThreads.java:497)
         [junit4]    > 	at java.lang.Thread.run(Thread.java:748)
         [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/checkout/lucene/build/core/test/J2/temp/lucene.index.TestIndexWriterWithThreads_66484822FD0371B5-001
         [junit4]   2> NOTE: test params are: codec=Asserting(Lucene70): {date=PostingsFormat(name=LuceneVarGapFixedInterval), field=PostingsFormat(name=LuceneVarGapFixedInterval), docid=PostingsFormat(name=Asserting), titleTokenized=PostingsFormat(name=LuceneFixedGap), body=PostingsFormat(name=LuceneVarGapFixedInterval), title=Lucene50(blocksize=128)}, docValues:{dv=DocValuesFormat(name=Direct), docid_intDV=DocValuesFormat(name=Memory), field=DocValuesFormat(name=Direct), titleDV=DocValuesFormat(name=Lucene70)}, maxPointsInLeafNode=1926, maxMBSortInHeap=6.378832546318826, sim=RandomSimilarity(queryNorm=false): {field=DFR GB2, titleTokenized=DFR I(n)3(800.0), body=IB LL-D1}, locale=es-CR, timezone=America/Manaus
         [junit4]   2> NOTE: Linux 3.13.0-88-generic amd64/Oracle Corporation 1.8.0_131 (64-bit)/cpus=4,threads=1,free=120430360,total=441974784
         [junit4]   2> NOTE: All tests run in this JVM: [TestLucene70NormsFormat, TestOfflineSorter, TestPrefixQuery, TestStringMSBRadixSorter, TestFieldsReader, TestSynonymQuery, TestSpanCollection, TestPackedInts, TestLongPostings, TestIndexWriterMerging, TestDocIdSetBuilder, TestAxiomaticSimilarity, TestFloatRangeFieldQueries, TestIndexWriterCommit, TestRoaringDocIdSet, TestStressAdvance, TestFieldCacheRewriteMethod, TestRamUsageEstimator, TestForUtil, TestNativeFSLockFactory, TestCompiledAutomaton, TestSegmentTermDocs, TestIOUtils, TestStressDeletes, TestHighCompressionMode, TestSearchWithThreads, TestByteBlockPool, TestIndexWriterLockRelease, TestSubScorerFreqs, TestBKD, TestSimpleExplanationsOfNonMatches, TestPackedTokenAttributeImpl, TestStopFilter, TestIndexWriterExceptions2, TestFastCompressionMode, TestManyFields, TestExceedMaxTermLength, TestTrackingDirectoryWrapper, TestAttributeSource, TestMixedCodecs, TestMergedIterator, TestAllFilesHaveCodecHeader, TestNeedsScores, TestEarlyTermination, TestNoMergePolicy, TestSetOnce, TestCachingTokenFilter, TestTwoPhaseCommitTool, TestIndexReaderClose, TestDemoParallelLeafReader, TestHugeRamFile, TestControlledRealTimeReopenThread, TestNRTReaderWithThreads, TestPayloads, TestSearchForDuplicates, TestIntsRef, TestShardSearching, TestNRTReaderCleanup, TestSegmentReader, TestSpanNotQuery, TestPerFieldPostingsFormat2, TestSpans, TestMultiFields, Test2BPositions, TestTerms, TestReaderWrapperDVTypeCheck, TestScorerPerf, TestBlockPostingsFormat, TestField, TestIndexWriterMergePolicy, TestIndexWriterReader, FiniteStringsIteratorTest, TestOperations, TestCharsRef, TestStressIndexing2, TestBagOfPostings, TestIndexWriterWithThreads]
         [junit4] Completed [189/453 (1!)] on J2 in 5.60s, 12 tests, 1 failure <<< FAILURES!
      

        Activity

        Hide
        mikemccand Michael McCandless added a comment -

        I found the cause: IW was in the process of aborting, while threads were concurrently still trying to apply deleted packets. I simplified the code here to not bother clearing the delete counters from abort.

        Show
        mikemccand Michael McCandless added a comment - I found the cause: IW was in the process of aborting, while threads were concurrently still trying to apply deleted packets. I simplified the code here to not bother clearing the delete counters from abort.
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit c9c0121d9399ff0009c51d6a32632dd0962e8c8f in lucene-solr's branch refs/heads/master from Mike McCandless
        [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c9c0121 ]

        LUCENE-7887: don't clear BufferedUpdatesStream on abort

        Show
        jira-bot ASF subversion and git services added a comment - Commit c9c0121d9399ff0009c51d6a32632dd0962e8c8f in lucene-solr's branch refs/heads/master from Mike McCandless [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c9c0121 ] LUCENE-7887 : don't clear BufferedUpdatesStream on abort
        Hide
        mikemccand Michael McCandless added a comment -

        I'll dig; looks fun

        Show
        mikemccand Michael McCandless added a comment - I'll dig; looks fun

          People

          • Assignee:
            Unassigned
            Reporter:
            steve_rowe Steve Rowe
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development