Lucene - Core
  1. Lucene - Core
  2. LUCENE-4277

TestIndexWriterWithThreads hung in DocumentsWriterFlushControl.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.0-BETA, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      I've hit this when trying out the new runner with suite timeouts. And voila, it works Output below.

      Suite: org.apache.lucene.index.TestIndexWriterWithThreads
         > (@AfterClass output)
        2> 31.7.2012 15:28:45 com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
        2> WARNING: Suite execution timed out: org.apache.lucene.index.TestIndexWriterWithThreads
        2> ==== jstack at approximately timeout time ====
        2> "Thread-54" ID=98 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2103217a owned by "Thread-53" ID=97
        2> 	at sun.misc.Unsafe.park(Native Method)
        2> 	- waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@2103217a
        2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        2> 	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        2> 	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        2> 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:532)
        2> 
        2> "Thread-53" ID=97 WAITING on org.apache.lucene.index.DocumentsWriterFlushControl@5d4d1093
        2> 	at java.lang.Object.wait(Native Method)
        2> 	- waiting on org.apache.lucene.index.DocumentsWriterFlushControl@5d4d1093
        2> 	at java.lang.Object.wait(Object.java:485)
        2> 	at org.apache.lucene.index.DocumentsWriterFlushControl.waitForFlush(DocumentsWriterFlushControl.java:221)
        2> 	at org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:233)
        2> 	- locked org.apache.lucene.index.DocumentsWriter@548ae184
        2> 	at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:1845)
        2> 	at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:1813)
        2> 	- locked java.lang.Object@69dbcc78
        2> 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:537)
        2> 	Locked synchronizers:
        2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@2103217a
        2> 
        2> "Thread-52" ID=96 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2103217a owned by "Thread-53" ID=97
        2> 	at sun.misc.Unsafe.park(Native Method)
        2> 	- waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@2103217a
        2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        2> 	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        2> 	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        2> 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:532)
        2> 
        2> "Thread-51" ID=95 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2103217a owned by "Thread-53" ID=97
        2> 	at sun.misc.Unsafe.park(Native Method)
        2> 	- waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@2103217a
        2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        2> 	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        2> 	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        2> 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:532)
        2> 
        2> "Thread-50" ID=94 WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@2103217a owned by "Thread-53" ID=97
        2> 	at sun.misc.Unsafe.park(Native Method)
        2> 	- waiting on java.util.concurrent.locks.ReentrantLock$NonfairSync@2103217a
        2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        2> 	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        2> 	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        2> 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:532)
        2> 
        2> "TEST-TestIndexWriterWithThreads.testRollbackAndCommitWithThreads-seed#[844703FAAAFA8855]" ID=87 WAITING on org.apache.lucene.index.TestIndexWriterWithThreads$1@358ce15a
        2> 	at java.lang.Object.wait(Native Method)
        2> 	- waiting on org.apache.lucene.index.TestIndexWriterWithThreads$1@358ce15a
        2> 	at java.lang.Thread.join(Thread.java:1143)
        2> 	at java.lang.Thread.join(Thread.java:1196)
        2> 	at org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndCommitWithThreads(TestIndexWriterWithThreads.java:590)
        2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        2> 	at java.lang.reflect.Method.invoke(Method.java:597)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1569)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:81)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:739)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:775)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:789)
        2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        2> 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32)
        2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:345)
        2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:763)
        2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:424)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:748)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:650)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:684)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:695)
        2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        2> 	at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
        2> 	at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
        2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
        2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:345)
        2> 	at java.lang.Thread.run(Thread.java:619)
        2> 
        2> "SUITE-TestIndexWriterWithThreads-seed#[844703FAAAFA8855]" ID=86 RUNNABLE
        2> 	at sun.management.ThreadImpl.dumpThreads0(Native Method)
        2> 	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:374)
        2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:636)
        2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$800(ThreadLeakControl.java:60)
        2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:379)
        2> 	- locked java.lang.Object@6e7b52bf
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:557)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:81)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:491)
        2> 
        2> "Signal Dispatcher" ID=4 RUNNABLE
        2> 
        2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@6275b243
        2> 	at java.lang.Object.wait(Native Method)
        2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@6275b243
        2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        2> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
        2> 
        2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@1d0a7f35
        2> 	at java.lang.Object.wait(Native Method)
        2> 	- waiting on java.lang.ref.Reference$Lock@1d0a7f35
        2> 	at java.lang.Object.wait(Object.java:485)
        2> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        2> 
        2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$1@3f1074b4
        2> 	at java.lang.Object.wait(Native Method)
        2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$1@3f1074b4
        2> 	at java.lang.Thread.join(Thread.java:1143)
        2> 	at java.lang.Thread.join(Thread.java:1196)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:501)
        2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:398)
        2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:153)
        2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:246)
        2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
        2> 
        2> ^^==============================================
        2> 
        2> 31.7.2012 15:29:05 com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
        2> SEVERE: 6 threads leaked from SUITE scope at org.apache.lucene.index.TestIndexWriterWithThreads: 
        2>    1) Thread[id=87, name=TEST-TestIndexWriterWithThreads.testRollbackAndCommitWithThreads-seed#[844703FAAAFA8855], state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>         at java.lang.Object.wait(Native Method)
        2>         at java.lang.Thread.join(Thread.java:1143)
        2>         at java.lang.Thread.join(Thread.java:1196)
        2>         at org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndCommitWithThreads(TestIndexWriterWithThreads.java:590)
        2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        2>         at java.lang.reflect.Method.invoke(Method.java:597)
        2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1569)
        2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:81)
        2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:739)
        2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:775)
        2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:789)
        2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        2>         at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32)
        2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:345)
        2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:763)
        2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:424)
        2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:748)
        2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:650)
        2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:684)
        2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:695)
        2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        2>         at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
        2>         at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
        2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
        2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:345)
        2>         at java.lang.Thread.run(Thread.java:619)
        2>    2) Thread[id=96, name=Thread-52, state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>         at sun.misc.Unsafe.park(Native Method)
        2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        2>         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        2>         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        2>         at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:532)
        2>    3) Thread[id=94, name=Thread-50, state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>         at sun.misc.Unsafe.park(Native Method)
        2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        2>         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        2>         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        2>         at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:532)
        2>    4) Thread[id=98, name=Thread-54, state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>         at sun.misc.Unsafe.park(Native Method)
        2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        2>         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        2>         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        2>         at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:532)
        2>    5) Thread[id=95, name=Thread-51, state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>         at sun.misc.Unsafe.park(Native Method)
        2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
        2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
        2>         at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        2>         at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        2>         at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:532)
        2>    6) Thread[id=97, name=Thread-53, state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>         at java.lang.Object.wait(Native Method)
        2>         at java.lang.Object.wait(Object.java:485)
        2>         at org.apache.lucene.index.DocumentsWriterFlushControl.waitForFlush(DocumentsWriterFlushControl.java:221)
        2>         at org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:233)
        2>         at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:1845)
        2>         at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:1813)
        2>         at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:537)
        2> 31.7.2012 15:29:05 com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
        2> INFO: Starting to interrupt leaked threads:
        2>    1) Thread[id=87, name=TEST-TestIndexWriterWithThreads.testRollbackAndCommitWithThreads-seed#[844703FAAAFA8855], state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>    2) Thread[id=96, name=Thread-52, state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>    3) Thread[id=94, name=Thread-50, state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>    4) Thread[id=98, name=Thread-54, state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>    5) Thread[id=95, name=Thread-51, state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2>    6) Thread[id=97, name=Thread-53, state=WAITING, group=TGRP-TestIndexWriterWithThreads]
        2> 31.7.2012 15:29:07 com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
        2> INFO: All leaked threads terminated.
        1> org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: org.apache.lucene.store.SingleInstanceLock@7685706f: write.lock
        1> 	at org.apache.lucene.store.Lock.obtain(Lock.java:84)
        1> 	at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:595)
        1> 	at org.apache.lucene.index.TestIndexWriterWithThreads$DelayedIndexAndCloseRunnable.run(TestIndexWriterWithThreads.java:490)
        2> NOTE: test params are: codec=Lucene40: {field=PostingsFormat(name=Memory doPackFST= true), docid=PostingsFormat(name=MockSep), body=PostingsFormat(name=Memory doPackFST= true), title=PostingsFormat(name=Lucene40WithOrds), titleTokenized=PostingsFormat(name=NestedPulsing), date=PostingsFormat(name=Memory doPackFST= true)}, sim=RandomSimilarityProvider(queryNorm=false,coord=false): {field=IB SPL-D3(800.0), body=DFR I(ne)2, titleTokenized=IB LL-L3(800.0)}, locale=cs_CZ, timezone=America/Anguilla
        2> NOTE: Linux 2.6.32-38-server amd64/Sun Microsystems Inc. 1.6.0_20 (64-bit)/cpus=4,threads=2,free=122913408,total=155123712
        2> NOTE: All tests run in this JVM: [TestFieldMaskingSpanQuery, TestDocValuesTypeCompatibility, TestTermScorer, TestSnapshotDeletionPolicy, TestLongPostings, TestSort, TestIndexWriter, TestCopyBytes, TestDocumentWriter, TestDateFilter, TestParallelReaderEmptyIndex, TestDateTools, TestMergeSchedulerExternal, TestPrefixInBooleanQuery, TestIndexWriterWithThreads]
        1> 
        2> 
      ERROR   0.00s J0 | TestIndexWriterWithThreads (suite)
         > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 600000 msec).
         > 	at __randomizedtesting.SeedInfo.seed([844703FAAAFA8855]:0)
         >
      Completed on J0 in 622.52s, 3 tests, 1 error <<< FAILURES!
      
      1. LUCENE-4277.patch
        3 kB
        Simon Willnauer
      2. LUCENE-4277.patch
        2 kB
        Simon Willnauer
      3. LUCENE-4277.patch
        2 kB
        Simon Willnauer

        Activity

        Hide
        Uwe Schindler added a comment -

        Another one, killed yesterday by me:

        Hi,
        This one was hanging in TestIndexWriterWithThreads#testRollbackAndCommitWithThreads for 40 minutes, doing nothing. Seems to be a locking problem.

        Uwe


        Uwe Schindler
        H.-H.-Meier-Allee 63, D-28213 Bremen
        http://www.thetaphi.de
        eMail: uwe@thetaphi.de

        > ----Original Message----
        > From: Policeman Jenkins Server jenkins@sd-datasolutions.de
        > Sent: Monday, July 30, 2012 11:04 PM
        > To: dev@lucene.apache.org
        > Subject: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b49) -
        > Build # 30 - Failure!
        >
        > Build:
        > http://jenkins.sd-datasolutions.de/job/Lucene-Solr-4.x-Linux/30/
        > Java: 32bit/jdk1.8.0-ea-b49 -server -XX:+UseParallelGC
        >
        > All tests passed
        >
        > Build Log:
        > [...truncated 1090 lines...]
        > [junit4:junit4] 2012-07-30 21:03:27
        > [junit4:junit4] Full thread dump Java HotSpot(TM) Server VM (24.0-b16
        > mixed
        > mode):
        > [junit4:junit4]
        > [junit4:junit4] "Thread-190" prio=10 tid=0xc95a6000 nid=0xcd7 waiting
        > on condition [0xc81fe000]
        > [junit4:junit4] java.lang.Thread.State: WAITING (parking)
        > [junit4:junit4] at sun.misc.Unsafe.park(Native Method)
        > [junit4:junit4] - parking to wait for <0xe6fde900> (a
        > java.util.concurrent.locks.ReentrantLock$NonfairSync)
        > [junit4:junit4] at
        > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte
        > rrupt(
        > AbstractQueuedSynchronizer.java:834)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ab
        > stract
        > QueuedSynchronizer.java:868)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract
        > Queued
        > Synchronizer.java:1198)
        > [junit4:junit4] at
        > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLoc
        > k.java:2
        > 14)
        > [junit4:junit4] at
        > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        > [junit4:junit4] at
        > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit
        > erW
        > ithThreads.java:534)
        > [junit4:junit4]
        > [junit4:junit4] "Thread-189" prio=10 tid=0xc95a5c00 nid=0xcd6 waiting
        > on condition [0xc8e7d000]
        > [junit4:junit4] java.lang.Thread.State: WAITING (parking)
        > [junit4:junit4] at sun.misc.Unsafe.park(Native Method)
        > [junit4:junit4] - parking to wait for <0xe6fde900> (a
        > java.util.concurrent.locks.ReentrantLock$NonfairSync)
        > [junit4:junit4] at
        > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte
        > rrupt(
        > AbstractQueuedSynchronizer.java:834)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ab
        > stract
        > QueuedSynchronizer.java:868)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract
        > Queued
        > Synchronizer.java:1198)
        > [junit4:junit4] at
        > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLoc
        > k.java:2
        > 14)
        > [junit4:junit4] at
        > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        > [junit4:junit4] at
        > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit
        > erW
        > ithThreads.java:534)
        > [junit4:junit4]
        > [junit4:junit4] "Thread-188" prio=10 tid=0xc95a5400 nid=0xcd5 waiting
        > on condition [0xc8159000]
        > [junit4:junit4] java.lang.Thread.State: WAITING (parking)
        > [junit4:junit4] at sun.misc.Unsafe.park(Native Method)
        > [junit4:junit4] - parking to wait for <0xe6fde900> (a
        > java.util.concurrent.locks.ReentrantLock$NonfairSync)
        > [junit4:junit4] at
        > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte
        > rrupt(
        > AbstractQueuedSynchronizer.java:834)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ab
        > stract
        > QueuedSynchronizer.java:868)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract
        > Queued
        > Synchronizer.java:1198)
        > [junit4:junit4] at
        > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLoc
        > k.java:2
        > 14)
        > [junit4:junit4] at
        > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        > [junit4:junit4] at
        > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit
        > erW
        > ithThreads.java:534)
        > [junit4:junit4]
        > [junit4:junit4] "Thread-187" prio=10 tid=0xc95a5000 nid=0xcd4 waiting
        > on condition [0xc907d000]
        > [junit4:junit4] java.lang.Thread.State: WAITING (parking)
        > [junit4:junit4] at sun.misc.Unsafe.park(Native Method)
        > [junit4:junit4] - parking to wait for <0xe6fde900> (a
        > java.util.concurrent.locks.ReentrantLock$NonfairSync)
        > [junit4:junit4] at
        > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte
        > rrupt(
        > AbstractQueuedSynchronizer.java:834)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ab
        > stract
        > QueuedSynchronizer.java:868)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract
        > Queued
        > Synchronizer.java:1198)
        > [junit4:junit4] at
        > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLoc
        > k.java:2
        > 14)
        > [junit4:junit4] at
        > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        > [junit4:junit4] at
        > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit
        > erW
        > ithThreads.java:534)
        > [junit4:junit4]
        > [junit4:junit4] "Thread-186" prio=10 tid=0xc95a4800 nid=0xcd3 in
        > Object.wait() [0xc797d000]
        > [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor)
        > [junit4:junit4] at java.lang.Object.wait(Native Method)
        > [junit4:junit4] - waiting on <0xe6fd2160> (a
        > org.apache.lucene.index.DocumentsWriterFlushControl)
        > [junit4:junit4] at java.lang.Object.wait(Object.java:502)
        > [junit4:junit4] at
        > org.apache.lucene.index.DocumentsWriterFlushControl.waitForFlush(Docum
        > ent
        > sWriterFlushControl.java:221)
        > [junit4:junit4] - locked <0xe6fd2160> (a
        > org.apache.lucene.index.DocumentsWriterFlushControl)
        > [junit4:junit4] at
        > org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:233)
        > [junit4:junit4] - locked <0xe6fcbda0> (a
        > org.apache.lucene.index.DocumentsWriter)
        > [junit4:junit4] at
        > org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:1847)
        > [junit4:junit4] at
        > org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:1815)
        > [junit4:junit4] - locked <0xe6fc1388> (a java.lang.Object)
        > [junit4:junit4] at
        > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit
        > erW
        > ithThreads.java:539)
        > [junit4:junit4]
        > [junit4:junit4] "Thread-185" prio=10 tid=0xc95a4400 nid=0xcd2 waiting
        > on condition [0xc9275000]
        > [junit4:junit4] java.lang.Thread.State: WAITING (parking)
        > [junit4:junit4] at sun.misc.Unsafe.park(Native Method)
        > [junit4:junit4] - parking to wait for <0xe6fde900> (a
        > java.util.concurrent.locks.ReentrantLock$NonfairSync)
        > [junit4:junit4] at
        > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte
        > rrupt(
        > AbstractQueuedSynchronizer.java:834)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ab
        > stract
        > QueuedSynchronizer.java:868)
        > [junit4:junit4] at
        > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract
        > Queued
        > Synchronizer.java:1198)
        > [junit4:junit4] at
        > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLoc
        > k.java:2
        > 14)
        > [junit4:junit4] at
        > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
        > [junit4:junit4] at
        > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit
        > erW
        > ithThreads.java:534)
        > [junit4:junit4]
        > [junit4:junit4] "TEST-TestScope-
        > org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndComm
        > it WithThreads-seed#[CC3428D220522DF8]" prio=10 tid=0xc97a1c00
        > nid=0xcd1 in Object.wait() [0xc987d000]
        > [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor)
        > [junit4:junit4] at java.lang.Object.wait(Native Method)
        > [junit4:junit4] - waiting on <0xe6fe4ba8> (a
        > org.apache.lucene.index.TestIndexWriterWithThreads$1)
        > [junit4:junit4] at java.lang.Thread.join(Thread.java:1261)
        > [junit4:junit4] - locked <0xe6fe4ba8> (a
        > org.apache.lucene.index.TestIndexWriterWithThreads$1)
        > [junit4:junit4] at java.lang.Thread.join(Thread.java:1335)
        > [junit4:junit4] at
        > org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndComm
        > it
        > WithThreads(TestIndexWriterWithThreads.java:592)
        > [junit4:junit4] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
        > Method)
        > [junit4:junit4] at
        > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
        > 57)
        > [junit4:junit4] at
        > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess
        > orI
        > mpl.java:43)
        > [junit4:junit4] at java.lang.reflect.Method.invoke(Method.java:474)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedR
        > u
        > nner.java:1995)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(Random
        > izedRunner.java:132)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Randomi
        > z
        > edRunner.java:818)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Randomi
        > z
        > edRunner.java:877)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Randomi
        > z
        > edRunner.java:891)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRul
        > eSet
        > upTeardownChained.java:50)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFie
        > ldCach
        > eSanity.java:32)
        > [junit4:junit4] at
        > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBefo
        > reAfte
        > rRule.java:45)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule
        > $1.ev
        > aluate(SystemPropertiesInvariantRule.java:55)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(Tes
        > tRule
        > ReportUncaughtExceptions.java:68)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleTh
        > rea
        > dAndTestName.java:48)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestR
        > uleIgn
        > oreAfterMaxFailures.java:70)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFail
        > ure.ja
        > va:48)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Rand
        > o
        > mizedRunner.java:825)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(Randomi
        > z
        > edRunner.java:132)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(Randomized
        > Runner.java:671)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Randomi
        > z
        > edRunner.java:697)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Randomi
        > z
        > edRunner.java:736)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Randomi
        > z
        > edRunner.java:747)
        > [junit4:junit4] at
        > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBefo
        > reAfte
        > rRule.java:45)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(Tes
        > tRule
        > ReportUncaughtExceptions.java:68)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStore
        > Class
        > Name.java:38)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule
        > $1.ev
        > aluate(SystemPropertiesInvariantRule.java:55)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(Tes
        > tRule
        > NoInstanceHooksOverrides.java:53)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestR
        > uleN
        > oStaticHooksShadowing.java:52)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleA
        > ssertio
        > nsRequired.java:36)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFail
        > ure.ja
        > va:48)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestR
        > uleIgn
        > oreAfterMaxFailures.java:70)
        > [junit4:junit4] at
        > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgn
        > oreTest
        > Suites.java:55)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(Randomize
        > d
        > Runner.java:605)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(Randomi
        > z
        > edRunner.java:132)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRu
        > nner.java:551)
        > [junit4:junit4]
        > [junit4:junit4] "Service Thread" daemon prio=10 tid=0xc978c800
        > nid=0xb87 runnable [0x00000000]
        > [junit4:junit4] java.lang.Thread.State: RUNNABLE
        > [junit4:junit4]
        > [junit4:junit4] "C1 CompilerThread3" daemon prio=10 tid=0xc978ac00
        > nid=0xb86 waiting on condition [0x00000000]
        > [junit4:junit4] java.lang.Thread.State: RUNNABLE
        > [junit4:junit4]
        > [junit4:junit4] "C2 CompilerThread2" daemon prio=10 tid=0xc9789000
        > nid=0xb85 waiting on condition [0x00000000]
        > [junit4:junit4] java.lang.Thread.State: RUNNABLE
        > [junit4:junit4]
        > [junit4:junit4] "C2 CompilerThread1" daemon prio=10 tid=0xc9787800
        > nid=0xb84 waiting on condition [0x00000000]
        > [junit4:junit4] java.lang.Thread.State: RUNNABLE
        > [junit4:junit4]
        > [junit4:junit4] "C2 CompilerThread0" daemon prio=10 tid=0xc9785800
        > nid=0xb83 waiting on condition [0x00000000]
        > [junit4:junit4] java.lang.Thread.State: RUNNABLE
        > [junit4:junit4]
        > [junit4:junit4] "Signal Dispatcher" daemon prio=10 tid=0xc9784000
        > nid=0xb82 waiting on condition [0x00000000]
        > [junit4:junit4] java.lang.Thread.State: RUNNABLE
        > [junit4:junit4]
        > [junit4:junit4] "Finalizer" daemon prio=10 tid=0xc9774400 nid=0xb7f in
        > Object.wait() [0xcbb7d000]
        > [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor)
        > [junit4:junit4] at java.lang.Object.wait(Native Method)
        > [junit4:junit4] - waiting on <0xd051a9d8> (a
        > java.lang.ref.ReferenceQueue$Lock)
        > [junit4:junit4] at
        > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        > [junit4:junit4] - locked <0xd051a9d8> (a java.lang.ref.ReferenceQueue$Lock)
        > [junit4:junit4] at
        > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        > [junit4:junit4] at
        > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)
        > [junit4:junit4]
        > [junit4:junit4] "Reference Handler" daemon prio=10 tid=0xc9772c00
        > nid=0xb7e in Object.wait() [0xcbd7d000]
        > [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor)
        > [junit4:junit4] at java.lang.Object.wait(Native Method)
        > [junit4:junit4] - waiting on <0xd051c0d0> (a java.lang.ref.Reference$Lock)
        > [junit4:junit4] at java.lang.Object.wait(Object.java:502)
        > [junit4:junit4] at
        > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:142)
        > [junit4:junit4] - locked <0xd051c0d0> (a java.lang.ref.Reference$Lock)
        > [junit4:junit4]
        > [junit4:junit4] "main" prio=10 tid=0xf6708800 nid=0xb61 in
        > Object.wait() [0xf6887000]
        > [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor)
        > [junit4:junit4] at java.lang.Object.wait(Native Method)
        > [junit4:junit4] - waiting on <0xe6f7f3e8> (a
        > com.carrotsearch.randomizedtesting.RandomizedRunner$2)
        > [junit4:junit4] at java.lang.Thread.join(Thread.java:1261)
        > [junit4:junit4] - locked <0xe6f7f3e8> (a
        > com.carrotsearch.randomizedtesting.RandomizedRunner$2)
        > [junit4:junit4] at java.lang.Thread.join(Thread.java:1335)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(Randomize
        > d
        > Runner.java:561)
        > [junit4:junit4] at
        > com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunn
        > er.java:521)
        > [junit4:junit4] at
        > com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:153)
        > [junit4:junit4] at
        > com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:246)
        > [junit4:junit4] at
        > com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSa
        > fe.jav
        > a:12)
        > [junit4:junit4]
        > [junit4:junit4] "VM Thread" prio=10 tid=0xc9770400 nid=0xb7c runnable
        > [junit4:junit4] [junit4:junit4] "GC task thread#0 (ParallelGC)"
        > prio=10
        > tid=0xf6712c00 nid=0xb67 runnable [junit4:junit4] [junit4:junit4] "GC
        > task
        > thread#1 (ParallelGC)" prio=10 tid=0xf6714000 nid=0xb68 runnable
        > [junit4:junit4] [junit4:junit4] "GC task thread#2 (ParallelGC)"
        > prio=10
        > tid=0xf6715800 nid=0xb69 runnable [junit4:junit4] [junit4:junit4] "GC
        > task
        > thread#3 (ParallelGC)" prio=10 tid=0xf6717000 nid=0xb6a runnable
        > [junit4:junit4] [junit4:junit4] "GC task thread#4 (ParallelGC)"
        > prio=10
        > tid=0xf6718400 nid=0xb6b runnable [junit4:junit4] [junit4:junit4] "GC
        > task
        > thread#5 (ParallelGC)" prio=10 tid=0xf6719c00 nid=0xb6c runnable
        > [junit4:junit4] [junit4:junit4] "GC task thread#6 (ParallelGC)"
        > prio=10
        > tid=0xf671b400 nid=0xb6d runnable [junit4:junit4] [junit4:junit4] "GC
        > task
        > thread#7 (ParallelGC)" prio=10 tid=0xf671c800 nid=0xb6f runnable
        > [junit4:junit4] [junit4:junit4] "VM Periodic Task Thread" prio=10
        > tid=0xc9796800 nid=0xb88 waiting on condition [junit4:junit4]
        > [junit4:junit4] JNI global references: 178 [junit4:junit4] [junit4:junit4] Heap
        > [junit4:junit4] PSYoungGen total 43008K, used 31193K [0xe5a70000,
        > 0xecf30000, 0xf0510000)
        > [junit4:junit4] eden space 41664K, 71% used
        > [0xe5a70000,0xe779eeb0,0xe8320000)
        > [junit4:junit4] from space 1344K, 97% used
        > [0xecde0000,0xecf27550,0xecf30000)
        > [junit4:junit4] to space 1728K, 0% used
        > [0xecbd0000,0xecbd0000,0xecd80000)
        > [junit4:junit4] ParOldGen total 43712K, used 14083K [0xd0510000,
        > 0xd2fc0000, 0xe5a70000)
        > [junit4:junit4] object space 43712K, 32% used
        > [0xd0510000,0xd12d0da0,0xd2fc0000)
        > [junit4:junit4] PSPermGen total 16384K, used 11741K [0xcc510000,
        > 0xcd510000, 0xd0510000)
        > [junit4:junit4] object space 16384K, 71% used
        > [0xcc510000,0xcd087558,0xcd510000)
        > [junit4:junit4] <<< JVM J0: EOF ----
        > [junit4:junit4] JVM J1: 0.67 .. 481.32 = 480.65s
        > [junit4:junit4] Execution time total: 40 minutes 20 seconds
        > [junit4:junit4]
        > ERROR: JVM J0 threw an exception, cmd line:
        > /mnt/ssd/jenkins/tools/java/32bit/jdk1.8.0-ea-b49/jre/bin/java -server
        > - XX:+UseParallelGC -Dtests.prefix=tests -Dtests.seed=CC3428D220522DF8
        > - Xmx512M -Dtests.iters= -Dtests.verbose=false
        > -Dtests.infostream=false -
        > Dtests.lockdir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-
        > Linux/checkout/lucene/build -Dtests.codec=random -
        > Dtests.postingsformat=random -Dtests.locale=random -
        > Dtests.timezone=random -Dtests.directory=random -
        > Dtests.linedocsfile=europarl.lines.txt.gz
        > -Dtests.luceneMatchVersion=4.0 - Dtests.cleanthreads=perMethod
        > -Djava.util.logging.config.file=/dev/null - Dtests.nightly=false
        > -Dtests.weekly=false -Dtests.slow=true - Dtests.asserts.gracious=false
        > -Dtests.multiplier=3 -DtempDir=. - Dlucene.version=4.0-SNAPSHOT
        > -Djetty.testMode=1 -Djetty.insecurerandom=1
        > -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -
        > Dfile.encoding=ISO-8859-1 -classpath
        > /mnt/ssd/jenkins/workspace/Lucene-Solr-
        > 4.x-Linux/checkout/lucene/build/test-
        > framework/classes/java:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-
        > Linux/checkout/lucene/test-framework/lib/junit-
        > 4.10.jar:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-
        > Linux/checkout/lucene/test-framework/lib/randomizedtesting-runner-
        > 1.6.0.jar:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-
        > Linux/checkout/lucene/build/core/classes/java:/mnt/ssd/jenkins/workspa
        > ce/Lu
        > cene-Solr-4.x-
        > Linux/checkout/lucene/build/core/classes/test:/var/lib/jenkins/tools/a
        > nt/suppor
        > ted18/lib/ant-launcher.jar:/var/lib/jenkins/.ant/lib/ivy-
        > 2.2.0.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-
        > jdepend.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-
        > netrexx.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-
        > antlr.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-commons-
        > net.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-
        > javamail.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache-
        > regexp.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-
        > jsch.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache-
        > xalan2.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-
        > junit4.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-
        > jmf.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant.jar:/var/lib/je
        > nkins/tools/
        > ant/supported18/lib/ant-
        > junit.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache-
        > bcel.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-
        > jai.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-commons-
        > logging.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache-
        > resolver.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache-
        > oro.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-
        > swing.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache-
        > bsf.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache-
        > log4j.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-
        > testutil.jar:/mnt/ssd/jenkins/tools/java/32bit/jdk1.8.0-ea-
        > b49/lib/tools.jar:/var/lib/jenkins/.ivy2/cache/com.carrotsearch.random
        > izedtesti ng/junit4-ant/jars/junit4-ant-1.6.0.jar
        > -ea:org.apache.lucene... - ea:org.apache.solr...
        > com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe - eventsfile
        > /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-
        > Linux/checkout/lucene/build/core/test/junit4-J0-2023111371df5f3c7.even
        > ts
        > @/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-
        > Linux/checkout/lucene/build/core/test/junit4-J0-2023111371df5f3c7.suit
        > es [junit4:junit4] ERROR: JVM J0 exception: Forked process exited with
        > an error
        > code: 137
        >
        > BUILD FAILED
        > /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/checkout/build.xml:29
        > : The following error occurred while executing this line:
        > /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-
        > Linux/checkout/lucene/build.xml:50: The following error occurred while
        > executing this line:
        > /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/checkout/lucene/commo
        > n-
        > build.xml:1095: The following error occurred while executing this line:
        > /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/checkout/lucene/commo
        > n-
        > build.xml:790: At least one slave process threw an unexpected exception, first:
        > Forked process exited with an error code: 137
        >
        > Total time: 40 minutes 30 seconds
        > Build step 'Execute shell' marked build as failure Archiving artifacts
        > Recording test results Email was triggered for: Failure Sending email
        > for trigger: Failure
        >

        Show
        Uwe Schindler added a comment - Another one, killed yesterday by me: Hi, This one was hanging in TestIndexWriterWithThreads#testRollbackAndCommitWithThreads for 40 minutes, doing nothing. Seems to be a locking problem. Uwe Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: uwe@thetaphi.de > ---- Original Message ---- > From: Policeman Jenkins Server jenkins@sd-datasolutions.de > Sent: Monday, July 30, 2012 11:04 PM > To: dev@lucene.apache.org > Subject: [JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.8.0-ea-b49) - > Build # 30 - Failure! > > Build: > http://jenkins.sd-datasolutions.de/job/Lucene-Solr-4.x-Linux/30/ > Java: 32bit/jdk1.8.0-ea-b49 -server -XX:+UseParallelGC > > All tests passed > > Build Log: > [...truncated 1090 lines...] > [junit4:junit4] 2012-07-30 21:03:27 > [junit4:junit4] Full thread dump Java HotSpot(TM) Server VM (24.0-b16 > mixed > mode): > [junit4:junit4] > [junit4:junit4] "Thread-190" prio=10 tid=0xc95a6000 nid=0xcd7 waiting > on condition [0xc81fe000] > [junit4:junit4] java.lang.Thread.State: WAITING (parking) > [junit4:junit4] at sun.misc.Unsafe.park(Native Method) > [junit4:junit4] - parking to wait for <0xe6fde900> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > [junit4:junit4] at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte > rrupt( > AbstractQueuedSynchronizer.java:834) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ab > stract > QueuedSynchronizer.java:868) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract > Queued > Synchronizer.java:1198) > [junit4:junit4] at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLoc > k.java:2 > 14) > [junit4:junit4] at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) > [junit4:junit4] at > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit > erW > ithThreads.java:534) > [junit4:junit4] > [junit4:junit4] "Thread-189" prio=10 tid=0xc95a5c00 nid=0xcd6 waiting > on condition [0xc8e7d000] > [junit4:junit4] java.lang.Thread.State: WAITING (parking) > [junit4:junit4] at sun.misc.Unsafe.park(Native Method) > [junit4:junit4] - parking to wait for <0xe6fde900> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > [junit4:junit4] at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte > rrupt( > AbstractQueuedSynchronizer.java:834) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ab > stract > QueuedSynchronizer.java:868) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract > Queued > Synchronizer.java:1198) > [junit4:junit4] at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLoc > k.java:2 > 14) > [junit4:junit4] at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) > [junit4:junit4] at > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit > erW > ithThreads.java:534) > [junit4:junit4] > [junit4:junit4] "Thread-188" prio=10 tid=0xc95a5400 nid=0xcd5 waiting > on condition [0xc8159000] > [junit4:junit4] java.lang.Thread.State: WAITING (parking) > [junit4:junit4] at sun.misc.Unsafe.park(Native Method) > [junit4:junit4] - parking to wait for <0xe6fde900> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > [junit4:junit4] at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte > rrupt( > AbstractQueuedSynchronizer.java:834) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ab > stract > QueuedSynchronizer.java:868) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract > Queued > Synchronizer.java:1198) > [junit4:junit4] at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLoc > k.java:2 > 14) > [junit4:junit4] at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) > [junit4:junit4] at > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit > erW > ithThreads.java:534) > [junit4:junit4] > [junit4:junit4] "Thread-187" prio=10 tid=0xc95a5000 nid=0xcd4 waiting > on condition [0xc907d000] > [junit4:junit4] java.lang.Thread.State: WAITING (parking) > [junit4:junit4] at sun.misc.Unsafe.park(Native Method) > [junit4:junit4] - parking to wait for <0xe6fde900> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > [junit4:junit4] at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte > rrupt( > AbstractQueuedSynchronizer.java:834) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ab > stract > QueuedSynchronizer.java:868) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract > Queued > Synchronizer.java:1198) > [junit4:junit4] at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLoc > k.java:2 > 14) > [junit4:junit4] at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) > [junit4:junit4] at > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit > erW > ithThreads.java:534) > [junit4:junit4] > [junit4:junit4] "Thread-186" prio=10 tid=0xc95a4800 nid=0xcd3 in > Object.wait() [0xc797d000] > [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor) > [junit4:junit4] at java.lang.Object.wait(Native Method) > [junit4:junit4] - waiting on <0xe6fd2160> (a > org.apache.lucene.index.DocumentsWriterFlushControl) > [junit4:junit4] at java.lang.Object.wait(Object.java:502) > [junit4:junit4] at > org.apache.lucene.index.DocumentsWriterFlushControl.waitForFlush(Docum > ent > sWriterFlushControl.java:221) > [junit4:junit4] - locked <0xe6fd2160> (a > org.apache.lucene.index.DocumentsWriterFlushControl) > [junit4:junit4] at > org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:233) > [junit4:junit4] - locked <0xe6fcbda0> (a > org.apache.lucene.index.DocumentsWriter) > [junit4:junit4] at > org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:1847) > [junit4:junit4] at > org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:1815) > [junit4:junit4] - locked <0xe6fc1388> (a java.lang.Object) > [junit4:junit4] at > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit > erW > ithThreads.java:539) > [junit4:junit4] > [junit4:junit4] "Thread-185" prio=10 tid=0xc95a4400 nid=0xcd2 waiting > on condition [0xc9275000] > [junit4:junit4] java.lang.Thread.State: WAITING (parking) > [junit4:junit4] at sun.misc.Unsafe.park(Native Method) > [junit4:junit4] - parking to wait for <0xe6fde900> (a > java.util.concurrent.locks.ReentrantLock$NonfairSync) > [junit4:junit4] at > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInte > rrupt( > AbstractQueuedSynchronizer.java:834) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(Ab > stract > QueuedSynchronizer.java:868) > [junit4:junit4] at > java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Abstract > Queued > Synchronizer.java:1198) > [junit4:junit4] at > java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLoc > k.java:2 > 14) > [junit4:junit4] at > java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290) > [junit4:junit4] at > org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWrit > erW > ithThreads.java:534) > [junit4:junit4] > [junit4:junit4] "TEST-TestScope- > org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndComm > it WithThreads-seed# [CC3428D220522DF8] " prio=10 tid=0xc97a1c00 > nid=0xcd1 in Object.wait() [0xc987d000] > [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor) > [junit4:junit4] at java.lang.Object.wait(Native Method) > [junit4:junit4] - waiting on <0xe6fe4ba8> (a > org.apache.lucene.index.TestIndexWriterWithThreads$1) > [junit4:junit4] at java.lang.Thread.join(Thread.java:1261) > [junit4:junit4] - locked <0xe6fe4ba8> (a > org.apache.lucene.index.TestIndexWriterWithThreads$1) > [junit4:junit4] at java.lang.Thread.join(Thread.java:1335) > [junit4:junit4] at > org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndComm > it > WithThreads(TestIndexWriterWithThreads.java:592) > [junit4:junit4] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > [junit4:junit4] at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java: > 57) > [junit4:junit4] at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccess > orI > mpl.java:43) > [junit4:junit4] at java.lang.reflect.Method.invoke(Method.java:474) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedR > u > nner.java:1995) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(Random > izedRunner.java:132) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(Randomi > z > edRunner.java:818) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(Randomi > z > edRunner.java:877) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(Randomi > z > edRunner.java:891) > [junit4:junit4] at > org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRul > eSet > upTeardownChained.java:50) > [junit4:junit4] at > org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFie > ldCach > eSanity.java:32) > [junit4:junit4] at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBefo > reAfte > rRule.java:45) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule > $1.ev > aluate(SystemPropertiesInvariantRule.java:55) > [junit4:junit4] at > org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(Tes > tRule > ReportUncaughtExceptions.java:68) > [junit4:junit4] at > org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleTh > rea > dAndTestName.java:48) > [junit4:junit4] at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestR > uleIgn > oreAfterMaxFailures.java:70) > [junit4:junit4] at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFail > ure.ja > va:48) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(Rand > o > mizedRunner.java:825) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(Randomi > z > edRunner.java:132) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(Randomized > Runner.java:671) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(Randomi > z > edRunner.java:697) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(Randomi > z > edRunner.java:736) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(Randomi > z > edRunner.java:747) > [junit4:junit4] at > org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBefo > reAfte > rRule.java:45) > [junit4:junit4] at > org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(Tes > tRule > ReportUncaughtExceptions.java:68) > [junit4:junit4] at > org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStore > Class > Name.java:38) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule > $1.ev > aluate(SystemPropertiesInvariantRule.java:55) > [junit4:junit4] at > org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(Tes > tRule > NoInstanceHooksOverrides.java:53) > [junit4:junit4] at > org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestR > uleN > oStaticHooksShadowing.java:52) > [junit4:junit4] at > org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleA > ssertio > nsRequired.java:36) > [junit4:junit4] at > org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFail > ure.ja > va:48) > [junit4:junit4] at > org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestR > uleIgn > oreAfterMaxFailures.java:70) > [junit4:junit4] at > org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgn > oreTest > Suites.java:55) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(Randomize > d > Runner.java:605) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(Randomi > z > edRunner.java:132) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRu > nner.java:551) > [junit4:junit4] > [junit4:junit4] "Service Thread" daemon prio=10 tid=0xc978c800 > nid=0xb87 runnable [0x00000000] > [junit4:junit4] java.lang.Thread.State: RUNNABLE > [junit4:junit4] > [junit4:junit4] "C1 CompilerThread3" daemon prio=10 tid=0xc978ac00 > nid=0xb86 waiting on condition [0x00000000] > [junit4:junit4] java.lang.Thread.State: RUNNABLE > [junit4:junit4] > [junit4:junit4] "C2 CompilerThread2" daemon prio=10 tid=0xc9789000 > nid=0xb85 waiting on condition [0x00000000] > [junit4:junit4] java.lang.Thread.State: RUNNABLE > [junit4:junit4] > [junit4:junit4] "C2 CompilerThread1" daemon prio=10 tid=0xc9787800 > nid=0xb84 waiting on condition [0x00000000] > [junit4:junit4] java.lang.Thread.State: RUNNABLE > [junit4:junit4] > [junit4:junit4] "C2 CompilerThread0" daemon prio=10 tid=0xc9785800 > nid=0xb83 waiting on condition [0x00000000] > [junit4:junit4] java.lang.Thread.State: RUNNABLE > [junit4:junit4] > [junit4:junit4] "Signal Dispatcher" daemon prio=10 tid=0xc9784000 > nid=0xb82 waiting on condition [0x00000000] > [junit4:junit4] java.lang.Thread.State: RUNNABLE > [junit4:junit4] > [junit4:junit4] "Finalizer" daemon prio=10 tid=0xc9774400 nid=0xb7f in > Object.wait() [0xcbb7d000] > [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor) > [junit4:junit4] at java.lang.Object.wait(Native Method) > [junit4:junit4] - waiting on <0xd051a9d8> (a > java.lang.ref.ReferenceQueue$Lock) > [junit4:junit4] at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) > [junit4:junit4] - locked <0xd051a9d8> (a java.lang.ref.ReferenceQueue$Lock) > [junit4:junit4] at > java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) > [junit4:junit4] at > java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177) > [junit4:junit4] > [junit4:junit4] "Reference Handler" daemon prio=10 tid=0xc9772c00 > nid=0xb7e in Object.wait() [0xcbd7d000] > [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor) > [junit4:junit4] at java.lang.Object.wait(Native Method) > [junit4:junit4] - waiting on <0xd051c0d0> (a java.lang.ref.Reference$Lock) > [junit4:junit4] at java.lang.Object.wait(Object.java:502) > [junit4:junit4] at > java.lang.ref.Reference$ReferenceHandler.run(Reference.java:142) > [junit4:junit4] - locked <0xd051c0d0> (a java.lang.ref.Reference$Lock) > [junit4:junit4] > [junit4:junit4] "main" prio=10 tid=0xf6708800 nid=0xb61 in > Object.wait() [0xf6887000] > [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor) > [junit4:junit4] at java.lang.Object.wait(Native Method) > [junit4:junit4] - waiting on <0xe6f7f3e8> (a > com.carrotsearch.randomizedtesting.RandomizedRunner$2) > [junit4:junit4] at java.lang.Thread.join(Thread.java:1261) > [junit4:junit4] - locked <0xe6f7f3e8> (a > com.carrotsearch.randomizedtesting.RandomizedRunner$2) > [junit4:junit4] at java.lang.Thread.join(Thread.java:1335) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(Randomize > d > Runner.java:561) > [junit4:junit4] at > com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunn > er.java:521) > [junit4:junit4] at > com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:153) > [junit4:junit4] at > com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:246) > [junit4:junit4] at > com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSa > fe.jav > a:12) > [junit4:junit4] > [junit4:junit4] "VM Thread" prio=10 tid=0xc9770400 nid=0xb7c runnable > [junit4:junit4] [junit4:junit4] "GC task thread#0 (ParallelGC)" > prio=10 > tid=0xf6712c00 nid=0xb67 runnable [junit4:junit4] [junit4:junit4] "GC > task > thread#1 (ParallelGC)" prio=10 tid=0xf6714000 nid=0xb68 runnable > [junit4:junit4] [junit4:junit4] "GC task thread#2 (ParallelGC)" > prio=10 > tid=0xf6715800 nid=0xb69 runnable [junit4:junit4] [junit4:junit4] "GC > task > thread#3 (ParallelGC)" prio=10 tid=0xf6717000 nid=0xb6a runnable > [junit4:junit4] [junit4:junit4] "GC task thread#4 (ParallelGC)" > prio=10 > tid=0xf6718400 nid=0xb6b runnable [junit4:junit4] [junit4:junit4] "GC > task > thread#5 (ParallelGC)" prio=10 tid=0xf6719c00 nid=0xb6c runnable > [junit4:junit4] [junit4:junit4] "GC task thread#6 (ParallelGC)" > prio=10 > tid=0xf671b400 nid=0xb6d runnable [junit4:junit4] [junit4:junit4] "GC > task > thread#7 (ParallelGC)" prio=10 tid=0xf671c800 nid=0xb6f runnable > [junit4:junit4] [junit4:junit4] "VM Periodic Task Thread" prio=10 > tid=0xc9796800 nid=0xb88 waiting on condition [junit4:junit4] > [junit4:junit4] JNI global references: 178 [junit4:junit4] [junit4:junit4] Heap > [junit4:junit4] PSYoungGen total 43008K, used 31193K [0xe5a70000, > 0xecf30000, 0xf0510000) > [junit4:junit4] eden space 41664K, 71% used > [0xe5a70000,0xe779eeb0,0xe8320000) > [junit4:junit4] from space 1344K, 97% used > [0xecde0000,0xecf27550,0xecf30000) > [junit4:junit4] to space 1728K, 0% used > [0xecbd0000,0xecbd0000,0xecd80000) > [junit4:junit4] ParOldGen total 43712K, used 14083K [0xd0510000, > 0xd2fc0000, 0xe5a70000) > [junit4:junit4] object space 43712K, 32% used > [0xd0510000,0xd12d0da0,0xd2fc0000) > [junit4:junit4] PSPermGen total 16384K, used 11741K [0xcc510000, > 0xcd510000, 0xd0510000) > [junit4:junit4] object space 16384K, 71% used > [0xcc510000,0xcd087558,0xcd510000) > [junit4:junit4] <<< JVM J0: EOF ---- > [junit4:junit4] JVM J1: 0.67 .. 481.32 = 480.65s > [junit4:junit4] Execution time total: 40 minutes 20 seconds > [junit4:junit4] > ERROR: JVM J0 threw an exception, cmd line: > /mnt/ssd/jenkins/tools/java/32bit/jdk1.8.0-ea-b49/jre/bin/java -server > - XX:+UseParallelGC -Dtests.prefix=tests -Dtests.seed=CC3428D220522DF8 > - Xmx512M -Dtests.iters= -Dtests.verbose=false > -Dtests.infostream=false - > Dtests.lockdir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x- > Linux/checkout/lucene/build -Dtests.codec=random - > Dtests.postingsformat=random -Dtests.locale=random - > Dtests.timezone=random -Dtests.directory=random - > Dtests.linedocsfile=europarl.lines.txt.gz > -Dtests.luceneMatchVersion=4.0 - Dtests.cleanthreads=perMethod > -Djava.util.logging.config.file=/dev/null - Dtests.nightly=false > -Dtests.weekly=false -Dtests.slow=true - Dtests.asserts.gracious=false > -Dtests.multiplier=3 -DtempDir=. - Dlucene.version=4.0-SNAPSHOT > -Djetty.testMode=1 -Djetty.insecurerandom=1 > -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory - > Dfile.encoding=ISO-8859-1 -classpath > /mnt/ssd/jenkins/workspace/Lucene-Solr- > 4.x-Linux/checkout/lucene/build/test- > framework/classes/java:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x- > Linux/checkout/lucene/test-framework/lib/junit- > 4.10.jar:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x- > Linux/checkout/lucene/test-framework/lib/randomizedtesting-runner- > 1.6.0.jar:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x- > Linux/checkout/lucene/build/core/classes/java:/mnt/ssd/jenkins/workspa > ce/Lu > cene-Solr-4.x- > Linux/checkout/lucene/build/core/classes/test:/var/lib/jenkins/tools/a > nt/suppor > ted18/lib/ant-launcher.jar:/var/lib/jenkins/.ant/lib/ivy- > 2.2.0.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant- > jdepend.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant- > netrexx.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant- > antlr.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-commons- > net.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant- > javamail.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache- > regexp.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant- > jsch.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache- > xalan2.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant- > junit4.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant- > jmf.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant.jar:/var/lib/je > nkins/tools/ > ant/supported18/lib/ant- > junit.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache- > bcel.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant- > jai.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-commons- > logging.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache- > resolver.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache- > oro.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant- > swing.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache- > bsf.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant-apache- > log4j.jar:/var/lib/jenkins/tools/ant/supported18/lib/ant- > testutil.jar:/mnt/ssd/jenkins/tools/java/32bit/jdk1.8.0-ea- > b49/lib/tools.jar:/var/lib/jenkins/.ivy2/cache/com.carrotsearch.random > izedtesti ng/junit4-ant/jars/junit4-ant-1.6.0.jar > -ea:org.apache.lucene... - ea:org.apache.solr... > com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe - eventsfile > /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x- > Linux/checkout/lucene/build/core/test/junit4-J0-2023111371df5f3c7.even > ts > @/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x- > Linux/checkout/lucene/build/core/test/junit4-J0-2023111371df5f3c7.suit > es [junit4:junit4] ERROR: JVM J0 exception: Forked process exited with > an error > code: 137 > > BUILD FAILED > /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/checkout/build.xml:29 > : The following error occurred while executing this line: > /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x- > Linux/checkout/lucene/build.xml:50: The following error occurred while > executing this line: > /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/checkout/lucene/commo > n- > build.xml:1095: The following error occurred while executing this line: > /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/checkout/lucene/commo > n- > build.xml:790: At least one slave process threw an unexpected exception, first: > Forked process exited with an error code: 137 > > Total time: 40 minutes 30 seconds > Build step 'Execute shell' marked build as failure Archiving artifacts > Recording test results Email was triggered for: Failure Sending email > for trigger: Failure >
        Hide
        Robert Muir added a comment -

        I'm guessing we are waiting for flushes to complete, but one flushing thread (e.g. from the addDocument thread)
        has thrown some unrelated exception in flush and is still marked as being in queue but will never notify() ?

        Show
        Robert Muir added a comment - I'm guessing we are waiting for flushes to complete, but one flushing thread (e.g. from the addDocument thread) has thrown some unrelated exception in flush and is still marked as being in queue but will never notify() ?
        Hide
        Robert Muir added a comment -

        OK, I think Uwe's seed reproduces:

        • use branch_4x
        • svn up -r1367267
        • ant test -Dtestcase=TestIndexWriterWithThreads -Dtests.seed=CC3428D220522DF8 -Dtests.multiplier=3

        Its already hung for 5 minutes here, so I think this is working.

        http://jenkins.sd-datasolutions.de/job/Lucene-Solr-4.x-Linux/lastCompletedBuild/testReport/org.apache.lucene.index/TestIndexWriterWithThreads/testRollbackAndCommitWithThreads/history/?

        Show
        Robert Muir added a comment - OK, I think Uwe's seed reproduces: use branch_4x svn up -r1367267 ant test -Dtestcase=TestIndexWriterWithThreads -Dtests.seed=CC3428D220522DF8 -Dtests.multiplier=3 Its already hung for 5 minutes here, so I think this is working. http://jenkins.sd-datasolutions.de/job/Lucene-Solr-4.x-Linux/lastCompletedBuild/testReport/org.apache.lucene.index/TestIndexWriterWithThreads/testRollbackAndCommitWithThreads/history/?
        Hide
        Hoss Man added a comment -

        OK, I think Uwe's seed reproduces:

        FWIW: with that branch+revision+seed+multiplier I'm seeing it hang as well ... been going about 10 minutes.

        Show
        Hoss Man added a comment - OK, I think Uwe's seed reproduces: FWIW: with that branch+revision+seed+multiplier I'm seeing it hang as well ... been going about 10 minutes.
        Hide
        Robert Muir added a comment -

        yeah it doesnt always hang: maybe 20% of the time.

        when it hangs, cpu shoots drops to 0% after about a minute
        otherwise it passes after about a minute, cpu is 100% the entire time.

        might be a concurrency issue, ill keep hacking at adding simple single-threaded
        stuff to TestIWExceptions until i find something (or another bug, or give up).

        Show
        Robert Muir added a comment - yeah it doesnt always hang: maybe 20% of the time. when it hangs, cpu shoots drops to 0% after about a minute otherwise it passes after about a minute, cpu is 100% the entire time. might be a concurrency issue, ill keep hacking at adding simple single-threaded stuff to TestIWExceptions until i find something (or another bug, or give up).
        Hide
        Simon Willnauer added a comment -

        I can't reproduce this even with the seed & revision etc. But I found a place where we could miss to remove an entry from the flush queue. The patch fixes that issue I found but I struggle to explain in which scenario this could happen without an underlying bug. Robert can you try this patch and see if it still fails?

        Show
        Simon Willnauer added a comment - I can't reproduce this even with the seed & revision etc. But I found a place where we could miss to remove an entry from the flush queue. The patch fixes that issue I found but I struggle to explain in which scenario this could happen without an underlying bug. Robert can you try this patch and see if it still fails?
        Hide
        Robert Muir added a comment -

        With the patch, I ran the seed about 10 times, then i hit the hang again:

        [junit4:junit4] Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode):
        [junit4:junit4] 
        [junit4:junit4] "Thread-6" prio=10 tid=0x00007fec4c0f5000 nid=0x4ba8 waiting on condition [0x00007fec37efd000]
        [junit4:junit4]    java.lang.Thread.State: WAITING (parking)
        [junit4:junit4] 	at sun.misc.Unsafe.park(Native Method)
        [junit4:junit4] 	- parking to wait for  <0x00000000f6c77bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        [junit4:junit4] 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        [junit4:junit4] 	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        [junit4:junit4] 	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        [junit4:junit4] 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:534)
        [junit4:junit4] 
        [junit4:junit4] "Thread-5" prio=10 tid=0x00007fec4c0f3800 nid=0x4ba7 in Object.wait() [0x00007fec37ffe000]
        [junit4:junit4]    java.lang.Thread.State: WAITING (on object monitor)
        [junit4:junit4] 	at java.lang.Object.wait(Native Method)
        [junit4:junit4] 	- waiting on <0x00000000f6c2ec18> (a org.apache.lucene.index.DocumentsWriterFlushControl)
        [junit4:junit4] 	at java.lang.Object.wait(Object.java:485)
        [junit4:junit4] 	at org.apache.lucene.index.DocumentsWriterFlushControl.waitForFlush(DocumentsWriterFlushControl.java:221)
        [junit4:junit4] 	- locked <0x00000000f6c2ec18> (a org.apache.lucene.index.DocumentsWriterFlushControl)
        [junit4:junit4] 	at org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:231)
        [junit4:junit4] 	- locked <0x00000000f6b98518> (a org.apache.lucene.index.DocumentsWriter)
        [junit4:junit4] 	at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:1847)
        [junit4:junit4] 	at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:1815)
        [junit4:junit4] 	- locked <0x00000000f6b296f0> (a java.lang.Object)
        [junit4:junit4] 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:539)
        [junit4:junit4] 
        [junit4:junit4] "Thread-4" prio=10 tid=0x00007fec4c0f2000 nid=0x4ba6 waiting on condition [0x00007fec8c103000]
        [junit4:junit4]    java.lang.Thread.State: WAITING (parking)
        [junit4:junit4] 	at sun.misc.Unsafe.park(Native Method)
        [junit4:junit4] 	- parking to wait for  <0x00000000f6c77bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        [junit4:junit4] 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        [junit4:junit4] 	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        [junit4:junit4] 	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        [junit4:junit4] 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:534)
        [junit4:junit4] 
        [junit4:junit4] "Thread-3" prio=10 tid=0x00007fec4c0d5800 nid=0x4ba5 waiting on condition [0x00007fec8c204000]
        [junit4:junit4]    java.lang.Thread.State: WAITING (parking)
        [junit4:junit4] 	at sun.misc.Unsafe.park(Native Method)
        [junit4:junit4] 	- parking to wait for  <0x00000000f6c77bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        [junit4:junit4] 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        [junit4:junit4] 	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        [junit4:junit4] 	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        [junit4:junit4] 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:534)
        [junit4:junit4] 
        [junit4:junit4] "Thread-2" prio=10 tid=0x00007fec4c0d4000 nid=0x4ba4 waiting on condition [0x00007fec8c305000]
        [junit4:junit4]    java.lang.Thread.State: WAITING (parking)
        [junit4:junit4] 	at sun.misc.Unsafe.park(Native Method)
        [junit4:junit4] 	- parking to wait for  <0x00000000f6c77bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        [junit4:junit4] 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        [junit4:junit4] 	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        [junit4:junit4] 	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        [junit4:junit4] 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:534)
        [junit4:junit4] 
        [junit4:junit4] "Thread-1" prio=10 tid=0x00007fec4c0e6000 nid=0x4ba3 waiting on condition [0x00007fec8c406000]
        [junit4:junit4]    java.lang.Thread.State: WAITING (parking)
        [junit4:junit4] 	at sun.misc.Unsafe.park(Native Method)
        [junit4:junit4] 	- parking to wait for  <0x00000000f6c77bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        [junit4:junit4] 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        [junit4:junit4] 	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        [junit4:junit4] 	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        [junit4:junit4] 	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        [junit4:junit4] 	at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:534)
        [junit4:junit4] 
        [junit4:junit4] "TEST-TestScope-org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndCommitWithThreads-seed#[CC3428D220522DF8]" prio=10 tid=0x00007fec90238000 nid=0x4ba2 in Object.wait() [0x00007fec8c60d000]
        [junit4:junit4]    java.lang.Thread.State: WAITING (on object monitor)
        [junit4:junit4] 	at java.lang.Object.wait(Native Method)
        [junit4:junit4] 	- waiting on <0x00000000f6c7dbd8> (a org.apache.lucene.index.TestIndexWriterWithThreads$1)
        [junit4:junit4] 	at java.lang.Thread.join(Thread.java:1186)
        [junit4:junit4] 	- locked <0x00000000f6c7dbd8> (a org.apache.lucene.index.TestIndexWriterWithThreads$1)
        [junit4:junit4] 	at java.lang.Thread.join(Thread.java:1239)
        [junit4:junit4] 	at org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndCommitWithThreads(TestIndexWriterWithThreads.java:592)
        [junit4:junit4] 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        [junit4:junit4] 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        [junit4:junit4] 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        [junit4:junit4] 	at java.lang.reflect.Method.invoke(Method.java:597)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:818)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:877)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:891)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32)
        [junit4:junit4] 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:825)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(RandomizedRunner.java:132)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(RandomizedRunner.java:671)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:697)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:736)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:747)
        [junit4:junit4] 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        [junit4:junit4] 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551)
        [junit4:junit4] 
        [junit4:junit4] "Low Memory Detector" daemon prio=10 tid=0x00007fec9009c000 nid=0x4ba0 runnable [0x0000000000000000]
        [junit4:junit4]    java.lang.Thread.State: RUNNABLE
        [junit4:junit4] 
        [junit4:junit4] "CompilerThread1" daemon prio=10 tid=0x00007fec90099800 nid=0x4b9f waiting on condition [0x0000000000000000]
        [junit4:junit4]    java.lang.Thread.State: RUNNABLE
        [junit4:junit4] 
        [junit4:junit4] "CompilerThread0" daemon prio=10 tid=0x00007fec90096800 nid=0x4b9e waiting on condition [0x0000000000000000]
        [junit4:junit4]    java.lang.Thread.State: RUNNABLE
        [junit4:junit4] 
        [junit4:junit4] "Signal Dispatcher" daemon prio=10 tid=0x00007fec90094800 nid=0x4b9d waiting on condition [0x0000000000000000]
        [junit4:junit4]    java.lang.Thread.State: RUNNABLE
        [junit4:junit4] 
        [junit4:junit4] "Finalizer" daemon prio=10 tid=0x00007fec90078000 nid=0x4b9c in Object.wait() [0x00007fec941ec000]
        [junit4:junit4]    java.lang.Thread.State: WAITING (on object monitor)
        [junit4:junit4] 	at java.lang.Object.wait(Native Method)
        [junit4:junit4] 	- waiting on <0x00000000f5561300> (a java.lang.ref.ReferenceQueue$Lock)
        [junit4:junit4] 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
        [junit4:junit4] 	- locked <0x00000000f5561300> (a java.lang.ref.ReferenceQueue$Lock)
        [junit4:junit4] 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
        [junit4:junit4] 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
        [junit4:junit4] 
        [junit4:junit4] "Reference Handler" daemon prio=10 tid=0x00007fec90076000 nid=0x4b9b in Object.wait() [0x00007fec942ed000]
        [junit4:junit4]    java.lang.Thread.State: WAITING (on object monitor)
        [junit4:junit4] 	at java.lang.Object.wait(Native Method)
        [junit4:junit4] 	- waiting on <0x00000000f55611d8> (a java.lang.ref.Reference$Lock)
        [junit4:junit4] 	at java.lang.Object.wait(Object.java:485)
        [junit4:junit4] 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
        [junit4:junit4] 	- locked <0x00000000f55611d8> (a java.lang.ref.Reference$Lock)
        [junit4:junit4] 
        [junit4:junit4] "main" prio=10 tid=0x00007fec9000a000 nid=0x4b91 in Object.wait() [0x00007fec96893000]
        [junit4:junit4]    java.lang.Thread.State: WAITING (on object monitor)
        [junit4:junit4] 	at java.lang.Object.wait(Native Method)
        [junit4:junit4] 	- waiting on <0x00000000f61ab898> (a com.carrotsearch.randomizedtesting.RandomizedRunner$2)
        [junit4:junit4] 	at java.lang.Thread.join(Thread.java:1186)
        [junit4:junit4] 	- locked <0x00000000f61ab898> (a com.carrotsearch.randomizedtesting.RandomizedRunner$2)
        [junit4:junit4] 	at java.lang.Thread.join(Thread.java:1239)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:561)
        [junit4:junit4] 	at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:521)
        [junit4:junit4] 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:153)
        [junit4:junit4] 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:246)
        [junit4:junit4] 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
        [junit4:junit4] 
        [junit4:junit4] "VM Thread" prio=10 tid=0x00007fec9006f800 nid=0x4b9a runnable 
        [junit4:junit4] 
        [junit4:junit4] "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fec9001d800 nid=0x4b92 runnable 
        [junit4:junit4] 
        [junit4:junit4] "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fec9001f000 nid=0x4b93 runnable 
        [junit4:junit4] 
        [junit4:junit4] "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fec90021000 nid=0x4b94 runnable 
        [junit4:junit4] 
        [junit4:junit4] "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fec90022800 nid=0x4b95 runnable 
        [junit4:junit4] 
        [junit4:junit4] "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007fec90024800 nid=0x4b96 runnable 
        [junit4:junit4] 
        [junit4:junit4] "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007fec90026800 nid=0x4b97 runnable 
        [junit4:junit4] 
        [junit4:junit4] "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007fec90028000 nid=0x4b98 runnable 
        [junit4:junit4] 
        [junit4:junit4] "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007fec9002a000 nid=0x4b99 runnable 
        [junit4:junit4] 
        [junit4:junit4] "VM Periodic Task Thread" prio=10 tid=0x00007fec900a6800 nid=0x4ba1 waiting on condition 
        [junit4:junit4] 
        [junit4:junit4] JNI global references: 1751
        [junit4:junit4] 
        [junit4:junit4] Heap
        [junit4:junit4]  PSYoungGen      total 74816K, used 40079K [0x00000000f5560000, 0x00000000fa8e0000, 0x0000000100000000)
        [junit4:junit4]   eden space 64128K, 62% used [0x00000000f5560000,0x00000000f7c83f40,0x00000000f9400000)
        [junit4:junit4]   from space 10688K, 0% used [0x00000000f9e70000,0x00000000f9e70000,0x00000000fa8e0000)
        [junit4:junit4]   to   space 10688K, 0% used [0x00000000f9400000,0x00000000f9400000,0x00000000f9e70000)
        [junit4:junit4]  PSOldGen        total 171008K, used 0K [0x00000000e0000000, 0x00000000ea700000, 0x00000000f5560000)
        [junit4:junit4]   object space 171008K, 0% used [0x00000000e0000000,0x00000000e0000000,0x00000000ea700000)
        [junit4:junit4]  PSPermGen       total 21248K, used 12691K [0x00000000dae00000, 0x00000000dc2c0000, 0x00000000e0000000)
        [junit4:junit4]   object space 21248K, 59% used [0x00000000dae00000,0x00000000dba64de0,0x00000000dc2c0000)
        
        Show
        Robert Muir added a comment - With the patch, I ran the seed about 10 times, then i hit the hang again: [junit4:junit4] Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode): [junit4:junit4] [junit4:junit4] "Thread-6" prio=10 tid=0x00007fec4c0f5000 nid=0x4ba8 waiting on condition [0x00007fec37efd000] [junit4:junit4] java.lang.Thread.State: WAITING (parking) [junit4:junit4] at sun.misc.Unsafe.park(Native Method) [junit4:junit4] - parking to wait for <0x00000000f6c77bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) [junit4:junit4] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) [junit4:junit4] at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) [junit4:junit4] at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) [junit4:junit4] at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:534) [junit4:junit4] [junit4:junit4] "Thread-5" prio=10 tid=0x00007fec4c0f3800 nid=0x4ba7 in Object.wait() [0x00007fec37ffe000] [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor) [junit4:junit4] at java.lang.Object.wait(Native Method) [junit4:junit4] - waiting on <0x00000000f6c2ec18> (a org.apache.lucene.index.DocumentsWriterFlushControl) [junit4:junit4] at java.lang.Object.wait(Object.java:485) [junit4:junit4] at org.apache.lucene.index.DocumentsWriterFlushControl.waitForFlush(DocumentsWriterFlushControl.java:221) [junit4:junit4] - locked <0x00000000f6c2ec18> (a org.apache.lucene.index.DocumentsWriterFlushControl) [junit4:junit4] at org.apache.lucene.index.DocumentsWriter.abort(DocumentsWriter.java:231) [junit4:junit4] - locked <0x00000000f6b98518> (a org.apache.lucene.index.DocumentsWriter) [junit4:junit4] at org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:1847) [junit4:junit4] at org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:1815) [junit4:junit4] - locked <0x00000000f6b296f0> (a java.lang.Object) [junit4:junit4] at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:539) [junit4:junit4] [junit4:junit4] "Thread-4" prio=10 tid=0x00007fec4c0f2000 nid=0x4ba6 waiting on condition [0x00007fec8c103000] [junit4:junit4] java.lang.Thread.State: WAITING (parking) [junit4:junit4] at sun.misc.Unsafe.park(Native Method) [junit4:junit4] - parking to wait for <0x00000000f6c77bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) [junit4:junit4] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) [junit4:junit4] at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) [junit4:junit4] at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) [junit4:junit4] at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:534) [junit4:junit4] [junit4:junit4] "Thread-3" prio=10 tid=0x00007fec4c0d5800 nid=0x4ba5 waiting on condition [0x00007fec8c204000] [junit4:junit4] java.lang.Thread.State: WAITING (parking) [junit4:junit4] at sun.misc.Unsafe.park(Native Method) [junit4:junit4] - parking to wait for <0x00000000f6c77bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) [junit4:junit4] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) [junit4:junit4] at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) [junit4:junit4] at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) [junit4:junit4] at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:534) [junit4:junit4] [junit4:junit4] "Thread-2" prio=10 tid=0x00007fec4c0d4000 nid=0x4ba4 waiting on condition [0x00007fec8c305000] [junit4:junit4] java.lang.Thread.State: WAITING (parking) [junit4:junit4] at sun.misc.Unsafe.park(Native Method) [junit4:junit4] - parking to wait for <0x00000000f6c77bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) [junit4:junit4] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) [junit4:junit4] at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) [junit4:junit4] at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) [junit4:junit4] at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:534) [junit4:junit4] [junit4:junit4] "Thread-1" prio=10 tid=0x00007fec4c0e6000 nid=0x4ba3 waiting on condition [0x00007fec8c406000] [junit4:junit4] java.lang.Thread.State: WAITING (parking) [junit4:junit4] at sun.misc.Unsafe.park(Native Method) [junit4:junit4] - parking to wait for <0x00000000f6c77bb8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync) [junit4:junit4] at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) [junit4:junit4] at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) [junit4:junit4] at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) [junit4:junit4] at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) [junit4:junit4] at org.apache.lucene.index.TestIndexWriterWithThreads$1.run(TestIndexWriterWithThreads.java:534) [junit4:junit4] [junit4:junit4] "TEST-TestScope-org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndCommitWithThreads-seed#[CC3428D220522DF8]" prio=10 tid=0x00007fec90238000 nid=0x4ba2 in Object.wait() [0x00007fec8c60d000] [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor) [junit4:junit4] at java.lang.Object.wait(Native Method) [junit4:junit4] - waiting on <0x00000000f6c7dbd8> (a org.apache.lucene.index.TestIndexWriterWithThreads$1) [junit4:junit4] at java.lang.Thread.join(Thread.java:1186) [junit4:junit4] - locked <0x00000000f6c7dbd8> (a org.apache.lucene.index.TestIndexWriterWithThreads$1) [junit4:junit4] at java.lang.Thread.join(Thread.java:1239) [junit4:junit4] at org.apache.lucene.index.TestIndexWriterWithThreads.testRollbackAndCommitWithThreads(TestIndexWriterWithThreads.java:592) [junit4:junit4] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit4:junit4] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) [junit4:junit4] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) [junit4:junit4] at java.lang.reflect.Method.invoke(Method.java:597) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1995) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.access$1100(RandomizedRunner.java:132) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:818) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:877) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:891) [junit4:junit4] at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) [junit4:junit4] at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:32) [junit4:junit4] at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68) [junit4:junit4] at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) [junit4:junit4] at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:825) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.access$700(RandomizedRunner.java:132) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$3$1.run(RandomizedRunner.java:671) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:697) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:736) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:747) [junit4:junit4] at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) [junit4:junit4] at org.apache.lucene.util.TestRuleReportUncaughtExceptions$1.evaluate(TestRuleReportUncaughtExceptions.java:68) [junit4:junit4] at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [junit4:junit4] at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) [junit4:junit4] at org.apache.lucene.util.TestRuleNoInstanceHooksOverrides$1.evaluate(TestRuleNoInstanceHooksOverrides.java:53) [junit4:junit4] at org.apache.lucene.util.TestRuleNoStaticHooksShadowing$1.evaluate(TestRuleNoStaticHooksShadowing.java:52) [junit4:junit4] at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:36) [junit4:junit4] at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) [junit4:junit4] at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70) [junit4:junit4] at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:605) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.access$400(RandomizedRunner.java:132) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:551) [junit4:junit4] [junit4:junit4] "Low Memory Detector" daemon prio=10 tid=0x00007fec9009c000 nid=0x4ba0 runnable [0x0000000000000000] [junit4:junit4] java.lang.Thread.State: RUNNABLE [junit4:junit4] [junit4:junit4] "CompilerThread1" daemon prio=10 tid=0x00007fec90099800 nid=0x4b9f waiting on condition [0x0000000000000000] [junit4:junit4] java.lang.Thread.State: RUNNABLE [junit4:junit4] [junit4:junit4] "CompilerThread0" daemon prio=10 tid=0x00007fec90096800 nid=0x4b9e waiting on condition [0x0000000000000000] [junit4:junit4] java.lang.Thread.State: RUNNABLE [junit4:junit4] [junit4:junit4] "Signal Dispatcher" daemon prio=10 tid=0x00007fec90094800 nid=0x4b9d waiting on condition [0x0000000000000000] [junit4:junit4] java.lang.Thread.State: RUNNABLE [junit4:junit4] [junit4:junit4] "Finalizer" daemon prio=10 tid=0x00007fec90078000 nid=0x4b9c in Object.wait() [0x00007fec941ec000] [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor) [junit4:junit4] at java.lang.Object.wait(Native Method) [junit4:junit4] - waiting on <0x00000000f5561300> (a java.lang.ref.ReferenceQueue$Lock) [junit4:junit4] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) [junit4:junit4] - locked <0x00000000f5561300> (a java.lang.ref.ReferenceQueue$Lock) [junit4:junit4] at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) [junit4:junit4] at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) [junit4:junit4] [junit4:junit4] "Reference Handler" daemon prio=10 tid=0x00007fec90076000 nid=0x4b9b in Object.wait() [0x00007fec942ed000] [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor) [junit4:junit4] at java.lang.Object.wait(Native Method) [junit4:junit4] - waiting on <0x00000000f55611d8> (a java.lang.ref.Reference$Lock) [junit4:junit4] at java.lang.Object.wait(Object.java:485) [junit4:junit4] at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) [junit4:junit4] - locked <0x00000000f55611d8> (a java.lang.ref.Reference$Lock) [junit4:junit4] [junit4:junit4] "main" prio=10 tid=0x00007fec9000a000 nid=0x4b91 in Object.wait() [0x00007fec96893000] [junit4:junit4] java.lang.Thread.State: WAITING (on object monitor) [junit4:junit4] at java.lang.Object.wait(Native Method) [junit4:junit4] - waiting on <0x00000000f61ab898> (a com.carrotsearch.randomizedtesting.RandomizedRunner$2) [junit4:junit4] at java.lang.Thread.join(Thread.java:1186) [junit4:junit4] - locked <0x00000000f61ab898> (a com.carrotsearch.randomizedtesting.RandomizedRunner$2) [junit4:junit4] at java.lang.Thread.join(Thread.java:1239) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:561) [junit4:junit4] at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:521) [junit4:junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:153) [junit4:junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:246) [junit4:junit4] at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12) [junit4:junit4] [junit4:junit4] "VM Thread" prio=10 tid=0x00007fec9006f800 nid=0x4b9a runnable [junit4:junit4] [junit4:junit4] "GC task thread#0 (ParallelGC)" prio=10 tid=0x00007fec9001d800 nid=0x4b92 runnable [junit4:junit4] [junit4:junit4] "GC task thread#1 (ParallelGC)" prio=10 tid=0x00007fec9001f000 nid=0x4b93 runnable [junit4:junit4] [junit4:junit4] "GC task thread#2 (ParallelGC)" prio=10 tid=0x00007fec90021000 nid=0x4b94 runnable [junit4:junit4] [junit4:junit4] "GC task thread#3 (ParallelGC)" prio=10 tid=0x00007fec90022800 nid=0x4b95 runnable [junit4:junit4] [junit4:junit4] "GC task thread#4 (ParallelGC)" prio=10 tid=0x00007fec90024800 nid=0x4b96 runnable [junit4:junit4] [junit4:junit4] "GC task thread#5 (ParallelGC)" prio=10 tid=0x00007fec90026800 nid=0x4b97 runnable [junit4:junit4] [junit4:junit4] "GC task thread#6 (ParallelGC)" prio=10 tid=0x00007fec90028000 nid=0x4b98 runnable [junit4:junit4] [junit4:junit4] "GC task thread#7 (ParallelGC)" prio=10 tid=0x00007fec9002a000 nid=0x4b99 runnable [junit4:junit4] [junit4:junit4] "VM Periodic Task Thread" prio=10 tid=0x00007fec900a6800 nid=0x4ba1 waiting on condition [junit4:junit4] [junit4:junit4] JNI global references: 1751 [junit4:junit4] [junit4:junit4] Heap [junit4:junit4] PSYoungGen total 74816K, used 40079K [0x00000000f5560000, 0x00000000fa8e0000, 0x0000000100000000) [junit4:junit4] eden space 64128K, 62% used [0x00000000f5560000,0x00000000f7c83f40,0x00000000f9400000) [junit4:junit4] from space 10688K, 0% used [0x00000000f9e70000,0x00000000f9e70000,0x00000000fa8e0000) [junit4:junit4] to space 10688K, 0% used [0x00000000f9400000,0x00000000f9400000,0x00000000f9e70000) [junit4:junit4] PSOldGen total 171008K, used 0K [0x00000000e0000000, 0x00000000ea700000, 0x00000000f5560000) [junit4:junit4] object space 171008K, 0% used [0x00000000e0000000,0x00000000e0000000,0x00000000ea700000) [junit4:junit4] PSPermGen total 21248K, used 12691K [0x00000000dae00000, 0x00000000dc2c0000, 0x00000000e0000000) [junit4:junit4] object space 21248K, 59% used [0x00000000dae00000,0x00000000dba64de0,0x00000000dc2c0000)
        Hide
        Simon Willnauer added a comment -

        here is a new patch. Now I have a fix plus an explanation. When we do a full flush we swap in a new DeleteQueue. All subsequent adds / updates / deletes go into the new queue and flushes that are associated with that queue will be blocked until the full flush (commit) is finished. Now imagine you add a document while a full flush is running and that document fills up the in memory buffers such that the DWPT needs to be flushed. Usually the thread adding the document helps flushing segments associated with the full flush until its done and then tries to flush the blocked segments. Yet, there is a window that allows to miss that window since the full flush is still "running" while already done ie. the switch is not flipped yet. This means the add thread returns leaving the DWPT in a "checked out & flushable" state. If we don't prune those DWPT during rollback / abort we can hit the hang since there are still segments in the pipeline that need to be flushed. If there is no thread adding or committing (which is impossible since we hold the commit log etc) we hang.

        This patch fixes the issue. I asked robert to beast it and it seems fine. Robert, can you confirm?

        Show
        Simon Willnauer added a comment - here is a new patch. Now I have a fix plus an explanation. When we do a full flush we swap in a new DeleteQueue. All subsequent adds / updates / deletes go into the new queue and flushes that are associated with that queue will be blocked until the full flush (commit) is finished. Now imagine you add a document while a full flush is running and that document fills up the in memory buffers such that the DWPT needs to be flushed. Usually the thread adding the document helps flushing segments associated with the full flush until its done and then tries to flush the blocked segments. Yet, there is a window that allows to miss that window since the full flush is still "running" while already done ie. the switch is not flipped yet. This means the add thread returns leaving the DWPT in a "checked out & flushable" state. If we don't prune those DWPT during rollback / abort we can hit the hang since there are still segments in the pipeline that need to be flushed. If there is no thread adding or committing (which is impossible since we hold the commit log etc) we hang. This patch fixes the issue. I asked robert to beast it and it seems fine. Robert, can you confirm?
        Hide
        Simon Willnauer added a comment -

        patch including a changes.txt entry.

        Show
        Simon Willnauer added a comment - patch including a changes.txt entry.
        Hide
        Robert Muir added a comment -

        I beasted the seed 50 times @ that revision.
        Also removed the seed and did 50 more times.

        no hangs/no fails. Thanks Simon!

        Show
        Robert Muir added a comment - I beasted the seed 50 times @ that revision. Also removed the seed and did 50 more times. no hangs/no fails. Thanks Simon!
        Hide
        Simon Willnauer added a comment -

        committed to trunk in revision r1368157
        merged to 4.x in revision r1368183

        Show
        Simon Willnauer added a comment - committed to trunk in revision r1368157 merged to 4.x in revision r1368183

          People

          • Assignee:
            Simon Willnauer
            Reporter:
            Dawid Weiss
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development