Major Description
My Jenkins found failures in Test2BBKDPoints and Test2BPoints in the same run on branch_6x; I've reported here before seeing them reproduce because the tests took 12 and 20 hours to run, respectively.
Checking out Revision 39aaa108ac8a85809080e4f7cf2b5ac0cc0d0fe9 (refs/remotes/origin/branch_6x) [...] [junit4] HEARTBEAT J0 PID(8991@localhost): 2016-03-26T01:02:55, stalled for 45731s at: Test2BBKDPoints.test2D [...] [junit4] Suite: org.apache.lucene.util.bkd.Test2BBKDPoints [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=Test2BBKDPoints -Dtests.method=test2D -Dtests.seed=E58B6D2DB5BEBF58 -Dtests.nightly=true -Dtests.slow=true -Dtests.locale=el-GR -Dtests.timezone=Europe/Warsaw -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] ERROR 45777s J0 | Test2BBKDPoints.test2D <<< [junit4] > Throwable #1: java.lang.IllegalStateException: wrong number of points in split: expected=1073743112 but actual=1 [junit4] > at __randomizedtesting.SeedInfo.seed([E58B6D2DB5BEBF58:C82EDF7D5339FEBB]:0) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1224) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:862) [junit4] > at org.apache.lucene.util.bkd.Test2BBKDPoints.test2D(Test2BBKDPoints.java:96) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > Suppressed: org.apache.lucene.index.CorruptIndexException: checksum passed (62a1e57e). possibly transient resource issue, or a Lucene or JVM bug (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/slow/jenkins/HDD-workspaces/Lucene-core-nightly-monster-6.x/lucene/build/core/test/J0/temp/lucene.util.bkd.Test2BBKDPoints_E58B6D2DB5BEBF58-001/2BBKDPoints2D-001/_0_bkd1_sort_71i.tmp"))) [junit4] > at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:409) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.verifyChecksum(BKDWriter.java:988) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1230) [junit4] > ... 38 more [junit4] 2> NOTE: leaving temporary files on disk at: /slow/jenkins/HDD-workspaces/Lucene-core-nightly-monster-6.x/lucene/build/core/test/J0/temp/lucene.util.bkd.Test2BBKDPoints_E58B6D2DB5BEBF58-001 [junit4] 2> NOTE: test params are: codec=CheapBastard, sim=ClassicSimilarity, locale=el-GR, timezone=Europe/Warsaw [junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 1.8.0_45 (64-bit)/cpus=16,threads=1,free=1512802208,total=4148166656 [junit4] 2> NOTE: All tests run in this JVM: [TestCustomSearcherSort, TestTimSorterWorstCase, TestPayloadsOnVectors, TestShardSearching, TestNumericDocValuesUpdates, TestCharsRef, TestCompiledAutomaton, TestFieldValueQuery, TestMutableValues, TestPerFieldPostingsFormat2, TestAtomicUpdate, TestDocumentsWriterDeleteQueue, TestParallelReaderEmptyIndex, Test2BNumericDocValues, TestQueryBuilder, TestNumericTokenStream, TestAllFilesHaveChecksumFooter, TestExitableDirectoryReader, TestDeterminism, TestTrackingDirectoryWrapper, TestConjunctionDISI, TestFilterLeafReader, TestWeakIdentityMap, TestBlendedTermQuery, TestIndexWriterOnVMError, TestSimpleFSDirectory, TestFixedBitDocIdSet, TestNewestSegment, TestIndexWriterForceMerge, TestTotalHitCountCollector, Test4GBStoredFields, TestBytesRef, TestSortedSetSortField, TestSentinelIntSet, TestToken, TestLogMergePolicy, TestIntsRef, TestIndexWriterDelete, TestOperations, TestNeedsScores, TestSpanFirstQuery, TestPackedInts, TestMultiTermConstantScore, TestReqExclBulkScorer, TestBinaryTerms, TestFastCompressionMode, TestDirectoryReaderReopen, TestRollingUpdates, TestPagedBytes, TestTryDelete, TestSegmentMerger, TestMinShouldMatch2, TestPositiveScoresOnlyCollector, TestByteArrayDataInput, Test2BPagedBytes, TestReusableStringReader, TestBytesRefAttImpl, TestPackedTokenAttributeImpl, TestGrowableByteArrayDataOutput, TestBlockPostingsFormat, TestBlockPostingsFormat2, TestForUtil, TestLucene50SegmentInfoFormat, TestLucene50StoredFieldsFormat, TestLucene50StoredFieldsFormatHighCompression, TestLucene50TermVectorsFormat, TestLucene53NormsFormat, TestLucene54DocValuesFormat, Test2BSortedDocValuesFixedSorted, TestAllFilesCheckIndexHeader, TestAllFilesDetectTruncation, TestDocInverterPerFieldErrorInfo, TestDocValues, TestDuelingCodecsAtNight, TestFilterDirectoryReader, TestIndexWriterDeleteByQuery, TestIndexWriterExceptions2, TestMergePolicyWrapper, TestMergeRateLimiter, TestMixedDocValuesUpdates, TestMultiTermsEnum, TestNRTReaderCleanup, TestPointValues, TestReadOnlyIndex, TestReaderWrapperDVTypeCheck, TestSegmentInfos, TestSwappedIndexFiles, TestTermVectors, TestApproximationSearchEquivalence, TestBooleanCoord, TestBooleanRewrites, TestEarlyTermination, TestIndexSearcher, TestMultiPhraseEnum, TestQueryRescorer, TestSortedSetSelector, TestClassicSimilarity, TestFilterSpans, TestSpanCollection, TestSpanContainQuery, TestSpanOrQuery, TestSpanTermQuery, TestSpansEnum, TestBufferedChecksum, TestFilterDirectory, TestMmapDirectory, TestNIOFSDirectory, TestNativeFSLockFactory, TestRateLimiter, TestSleepingLockWrapper, TestCharsRefBuilder, TestFrequencyTrackingRingBuffer, TestInPlaceMergeSorter, TestIntArrayDocIdSet, TestIntroSorter, TestLongBitSet, TestMergedIterator, TestNotDocIdSet, TestSparseFixedBitDocIdSet, TestTimSorter, TestRegExp, Test2BBKDPoints] [...] [junit4] HEARTBEAT J2 PID(8984@localhost): 2016-03-26T06:12:22, stalled for 72912s at: Test2BPoints.test2D [junit4] Suite: org.apache.lucene.index.Test2BPoints [junit4] 1> TEST: now CheckIndex [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=Test2BPoints -Dtests.method=test2D -Dtests.seed=E58B6D2DB5BEBF58 -Dtests.nightly=true -Dtests.slow=true -Dtests.locale=ms-MY -Dtests.timezone=Europe/Rome -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] ERROR 72944s J2 | Test2BPoints.test2D <<< [junit4] > Throwable #1: java.lang.IllegalStateException: this writer hit an unrecoverable error; cannot complete forceMerge [junit4] > at __randomizedtesting.SeedInfo.seed([E58B6D2DB5BEBF58:C82EDF7D5339FEBB]:0) [junit4] > at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1786) [junit4] > at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1736) [junit4] > at org.apache.lucene.index.Test2BPoints.test2D(Test2BPoints.java:133) [junit4] > at java.lang.Thread.run(Thread.java:745) [junit4] > Caused by: java.lang.IllegalStateException: wrong number of points in split: expected=1073741825 but actual=0 [junit4] > at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1224) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:862) [junit4] > at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.writeField(Lucene60PointsWriter.java:115) [junit4] > at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:58) [junit4] > at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:204) [junit4] > at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:168) [junit4] > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:117) [junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099) [junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626) [junit4] > Suppressed: org.apache.lucene.index.CorruptIndexException: checksum passed (90c634e2). possibly transient resource issue, or a Lucene or JVM bug (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/slow/jenkins/HDD-workspaces/Lucene-core-nightly-monster-6.x/lucene/build/core/test/J2/temp/lucene.index.Test2BPoints_E58B6D2DB5BEBF58-001/2BPoints2D-001/_50_bkd0_sort_1iy7.tmp"))) [junit4] > at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:409) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.verifyChecksum(BKDWriter.java:988) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1230) [junit4] > ... 10 more [junit4] 2> NOTE: leaving temporary files on disk at: /slow/jenkins/HDD-workspaces/Lucene-core-nightly-monster-6.x/lucene/build/core/test/J2/temp/lucene.index.Test2BPoints_E58B6D2DB5BEBF58-001 [junit4] 2> Mac 26, 2016 6:12:53 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException [junit4] 2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #17,5,TGRP-Test2BPoints] [junit4] 2> org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: wrong number of points in split: expected=1073741825 but actual=0 [junit4] 2> at __randomizedtesting.SeedInfo.seed([E58B6D2DB5BEBF58]:0) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648) [junit4] 2> Caused by: java.lang.IllegalStateException: wrong number of points in split: expected=1073741825 but actual=0 [junit4] 2> at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1224) [junit4] 2> at org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:862) [junit4] 2> at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.writeField(Lucene60PointsWriter.java:115) [junit4] 2> at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:58) [junit4] 2> at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:204) [junit4] 2> at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:168) [junit4] 2> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:117) [junit4] 2> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099) [junit4] 2> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588) [junit4] 2> at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626) [junit4] 2> Suppressed: org.apache.lucene.index.CorruptIndexException: checksum passed (90c634e2). possibly transient resource issue, or a Lucene or JVM bug (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/slow/jenkins/HDD-workspaces/Lucene-core-nightly-monster-6.x/lucene/build/core/test/J2/temp/lucene.index.Test2BPoints_E58B6D2DB5BEBF58-001/2BPoints2D-001/_50_bkd0_sort_1iy7.tmp"))) [junit4] 2> at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:409) [junit4] 2> at org.apache.lucene.util.bkd.BKDWriter.verifyChecksum(BKDWriter.java:988) [junit4] 2> at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1230) [junit4] 2> ... 10 more [junit4] 2> [junit4] 2> Mar 26, 2016 6:12:53 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks [junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s). [junit4] 2> NOTE: test params are: codec=CheapBastard, sim=ClassicSimilarity, locale=ms-MY, timezone=Europe/Rome [junit4] 2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 1.8.0_45 (64-bit)/cpus=16,threads=1,free=110605288,total=675807232 [junit4] 2> NOTE: All tests run in this JVM: [TestSortedSetDocValues, TestDocValuesScoring, TestHugeRamFile, TestStringHelper, TestCodecHoldsOpenFiles, TestNGramPhraseQuery, TestUsageTrackingFilterCachingPolicy, TestSpanNotQuery, TestFieldType, TestLRUQueryCache, TestFixedBitSet, TestNumericUtils, TestMultiPhraseQuery, TestFuzzyQuery, TestCachingTokenFilter, TestMultiLevelSkipList, TestNumericRangeQuery32, TestIndexWriterNRTIsCurrent, TestForceMergeForever, TestCachingCollector, TestNumericRangeQuery64, TestDocumentWriter, TestSegmentTermEnum, TestPositionIncrement, LimitedFiniteStringsIteratorTest, TestSpanBoostQuery, TestIndexWriterThreadsToSegments, TestIndexWriterFromReader, TestRecyclingIntBlockAllocator, TestSingleInstanceLockFactory, TestPrefixCodedTerms, TestMaxPosition, TestDocValuesRewriteMethod, TestRollingBuffer, TestAssertions, FuzzyTermOnShortTermsTest, TestTerms, TestMergeSchedulerExternal, TestSPIClassIterator, TestIOUtils, TestQueryCachingPolicy, TestCrash, TestIndexWriterReader, TestBooleanMinShouldMatch, TestDuelingCodecs, TestSortedNumericSortField, TestCodecUtil, TestLucene50FieldInfoFormat, TestUTF32ToUTF8, TestTermdocPerf, TestIndexWriterOnDiskFull, TestDirectPacked, TestIndexWriter, TestBytesStore, TestIndexWriterWithThreads, TestIndexWriterExceptions, TestSearcherManager, TestDeletionPolicy, TestBasics, TestFlushByRamOrCountsPolicy, TestSloppyPhraseQuery, TestNRTThreads, TestBoolean2, TestStressAdvance, TestNorms, TestSearchWithThreads, TestConcurrentMergeScheduler, TestDocumentsWriterStallControl, TestNeverDelete, TestTopDocsMerge, TestLongPostings, TestMinimize, TestNRTReaderWithThreads, TestTieredMergePolicy, TestPerFieldPostingsFormat, TestLevenshteinAutomata, TestBufferedIndexInput, TestTermsEnum, TestSimpleExplanations, TestLockFactory, TestPhraseQuery, TestCollectionUtil, TestBagOfPositions, TestOmitNorms, TestComplexExplanations, TestRegexpRandom2, TestNRTCachingDirectory, TestSegmentReader, TestFastDecompressionMode, TestParallelCompositeReader, TestTransactions, TestMultiThreadTermVectors, TestSegmentTermDocs, TestSort, TestMultiDocValues, TestFieldsReader, TestDocValuesIndexing, TestTermsEnum2, TestLegacyNumericUtils, TestPostingsOffsets, TestMixedCodecs, TestFlex, TestStressIndexing2, TestRAMDirectory, TestTermVectorsWriter, TestCodecs, TestCharTermAttributeImpl, TestParallelLeafReader, TestBytesRefHash, TestThreadedForceMerge, TestStressNRT, TestSloppyPhraseQuery2, TestDisjunctionMaxQuery, TestOmitTf, TestSameScoresWithThreads, TestTermRangeQuery, TestCrashCausesCorruptIndex, TestLiveFieldValues, TestForTooMuchCloning, TestPayloads, TestDoc, TestWildcard, TestSimilarity2, TestFieldMaskingSpanQuery, TestExternalCodecs, TestNoDeletionPolicy, TestSimpleExplanationsOfNonMatches, TestTermVectorsReader, TestPerSegmentDeletes, TestMultiValuedNumericRangeQuery, TestIntBlockPool, TestTopDocsCollector, TestUniqueTermCount, TestBooleanQuery, TestUnicodeUtil, TestAllFilesHaveCodecHeader, TestAutomatonQuery, TestDocument, TestTopFieldCollector, TestFileSwitchDirectory, TestSumDocFreq, TestIndexableField, TestWildcardRandom, TestPrefixRandom, TestDirectory, TestRegexpRandom, TestCheckIndex, TestSpanMultiTermQueryWrapper, TestBooleanQueryVisitSubscorers, TestSizeBoundedForceMerge, TestIndexFileDeleter, TestConstantScoreQuery, TestMatchAllDocsQuery, TestParallelTermEnum, TestElevationComparator, TestSubScorerFreqs, TestConjunctions, TestSimilarity, TestVirtualMethod, TestSimilarityProvider, TestSetOnce, TestSearch, TestFilterIterator, Test2BPostings, TestSameTokenSamePosition, TestDocCount, TestBinaryDocument, TestAutomatonQueryUnicode, TestAttributeSource, TestBooleanScorer, TestRamUsageEstimator, TestNoMergePolicy, TestNoMergeScheduler, TestNamedSPILoader, TestIndexWriterLockRelease, TestPhrasePrefixQuery, TestRollback, TestTwoPhaseCommitTool, TestCloseableThreadLocal, MultiCollectorTest, TestDocBoost, Test2BPositions, TestIndexCommit, TestTerm, Test2BPostingsBytes, Test2BPoints] [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=Test2BPoints -Dtests.seed=E58B6D2DB5BEBF58 -Dtests.nightly=true -Dtests.slow=true -Dtests.locale=ms-MY -Dtests.timezone=Europe/Rome -Dtests.asserts=true -Dtests.file.encoding=US-ASCII [junit4] ERROR 0.00s J2 | Test2BPoints (suite) <<< [junit4] > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=4892, name=Lucene Merge Thread #17, state=RUNNABLE, group=TGRP-Test2BPoints] [junit4] > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.IllegalStateException: wrong number of points in split: expected=1073741825 but actual=0 [junit4] > at __randomizedtesting.SeedInfo.seed([E58B6D2DB5BEBF58]:0) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648) [junit4] > Caused by: java.lang.IllegalStateException: wrong number of points in split: expected=1073741825 but actual=0 [junit4] > at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1224) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.finish(BKDWriter.java:862) [junit4] > at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.writeField(Lucene60PointsWriter.java:115) [junit4] > at org.apache.lucene.codecs.PointsWriter.mergeOneField(PointsWriter.java:58) [junit4] > at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:204) [junit4] > at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:168) [junit4] > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:117) [junit4] > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4099) [junit4] > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588) [junit4] > at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626) [junit4] > Suppressed: org.apache.lucene.index.CorruptIndexException: checksum passed (90c634e2). possibly transient resource issue, or a Lucene or JVM bug (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/slow/jenkins/HDD-workspaces/Lucene-core-nightly-monster-6.x/lucene/build/core/test/J2/temp/lucene.index.Test2BPoints_E58B6D2DB5BEBF58-001/2BPoints2D-001/_50_bkd0_sort_1iy7.tmp"))) [junit4] > at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:409) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.verifyChecksum(BKDWriter.java:988) [junit4] > at org.apache.lucene.util.bkd.BKDWriter.build(BKDWriter.java:1230) [junit4] > ... 10 more [junit4] Completed [415/415 (3!)] on J2 in 82560.59s, 2 tests, 2 errors <<< FAILURES!
FWIW, here's how the suite was run (via Jenkins):
ant -f lucene/core/build.xml clean test -Dtests.jvms=4 -Dtests.nightly=true -Dtests.monster=true -Dtests.heapsize=30g