Solr
  1. Solr
  2. SOLR-6920

During replication use checksums to verify if files are the same

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 5.0, 6.0
    • Component/s: replication (java)
    • Labels:
      None

      Description

      Currently we check if an index file on the master and slave is the same by checking if it's name and file length match.

      With LUCENE-2446 we now have a checksums for each index file in the segment. We should leverage this to verify if two files are the same.

      Places like SnapPuller.isIndexStale and SnapPuller.downloadIndexFiles should check against the checksum also.

      1. SOLR-6920.patch
        16 kB
        Varun Thacker
      2. SOLR-6920.patch
        17 kB
        Mark Miller
      3. SOLR-6920.patch
        15 kB
        Mark Miller
      4. SOLR-6920.patch
        9 kB
        Varun Thacker
      5. SOLR-6920.patch
        11 kB
        Varun Thacker
      6. SOLR-6920.patch
        12 kB
        Varun Thacker
      7. SOLR-6920-5x.patch
        19 kB
        Mark Miller
      8. SOLR-6920-5x.patch
        19 kB
        Mark Miller
      9. SOLR-6920-5x.patch
        18 kB
        Mark Miller
      10. SOLR-6920-5x.patch
        18 kB
        Mark Miller

        Issue Links

          Activity

          Hide
          Varun Thacker added a comment -

          Few tests fail with this patch -

             [junit4] Tests with failures:
             [junit4]   - org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch
             [junit4]   - org.apache.solr.cloud.ShardSplitTest.testDistribSearch
             [junit4]   - org.apache.solr.cloud.SyncSliceTest.testDistribSearch
             [junit4]   - org.apache.solr.cloud.RecoveryZkTest.testDistribSearch
          

          I picked ShardSplitTest.testDistribSearch and started investigating why is it failing always with the patch -

          I was seeing the following stack trace -

          53340 T166 C65 P58357 oasc.SolrException.log ERROR SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher
          		at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1603)
          		at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1715)
          		at org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:680)
          		at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:496)
          		at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:340)
          		at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
          		at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
          		at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
          	Caused by: java.io.FileNotFoundException: _1_2.liv in dir=RAMDirectory@2fdfb9bd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@91b3fd9
          		at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:655)
          		at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110)
          		at org.apache.lucene.codecs.lucene50.Lucene50LiveDocsFormat.readLiveDocs(Lucene50LiveDocsFormat.java:84)
          		at org.apache.lucene.codecs.asserting.AssertingLiveDocsFormat.readLiveDocs(AssertingLiveDocsFormat.java:64)
          		at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:101)
          		at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:134)
          		at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:186)
          		at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:94)
          		at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:430)
          		at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268)
          		at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:203)
          		at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1522)
          		... 7 more
          	
          53344 T166 C65 P58357 oasc.SolrException.log ERROR Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed.
          		at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:166)
          		at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
          		at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
          

          This looked very similar to the problem we were trying to solve in SOLR-6640. So I applied the patch from SOLR-6640 along with this patch and now I am seeing these -

          392008 T11 oasc.Diagnostics.logThreadDumps ERROR Gave up waiting for recovery to finish.  THREAD DUMP:
          	"qtp2004060070-174" Id=174 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"RecoveryThread-collection1_shard1_1_replica2" Id=168 TIMED_WAITING on java.lang.Object@1398321c
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Object@1398321c
          		at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:209)
          		-  locked org.apache.solr.update.DefaultSolrCoreState@50d256b5
          		at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:469)
          		at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:340)
          		at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
          		at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
          		at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
          	
          		Number of locked synchronizers = 1
          		- java.util.concurrent.locks.ReentrantLock$NonfairSync@213aae87
          	
          	"searcherExecutor-61-thread-1" Id=166 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d01d650
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d01d650
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"RecoveryThread-collection1_shard1_0_replica2" Id=163 TIMED_WAITING on java.lang.Object@50e7ed31
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Object@50e7ed31
          		at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:209)
          		-  locked org.apache.solr.update.DefaultSolrCoreState@3e7f8d5e
          		at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:469)
          		at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:340)
          		at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163)
          		at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)
          		at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235)
          	
          		Number of locked synchronizers = 1
          		- java.util.concurrent.locks.ReentrantLock$NonfairSync@281a3660
          	
          	"searcherExecutor-56-thread-1" Id=161 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a855576
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a855576
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"searcherExecutor-52-thread-1" Id=156 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17adcbc9
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17adcbc9
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"searcherExecutor-48-thread-1" Id=154 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@131e0544
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@131e0544
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"OverseerThreadFactory-5-thread-4" Id=152 WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
          		at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          		at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		...
          	
          	"OverseerThreadFactory-5-thread-3" Id=146 WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
          		at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          		at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		...
          	
          	"OverseerThreadFactory-5-thread-2" Id=145 WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
          		at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          		at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		...
          	
          	"OverseerThreadFactory-5-thread-1" Id=144 WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)
          		at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
          		at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		...
          	
          	"coreZkRegister-39-thread-1" Id=136 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75d01caa
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75d01caa
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"Thread-57" Id=134 WAITING on java.lang.Object@320a1953
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Object@320a1953
          		at java.lang.Object.wait(Object.java:503)
          		at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
          	
          	"searcherExecutor-44-thread-1" Id=133 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3504b1cf
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3504b1cf
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=129 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4735a83a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4735a83a
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=128 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@7b7a6620
          		-  locked java.util.Collections$UnmodifiableSet@685b5a62
          		-  locked sun.nio.ch.KQueueSelectorImpl@2c205e57
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
          		at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
          	
          	"qtp860063164-124" Id=124 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp860063164-123" Id=123 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp860063164-122" Id=122 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp860063164-121" Id=121 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp860063164-120" Id=120 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp860063164-119" Id=119 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp860063164-118 Acceptor0 SelectChannelConnector@127.0.0.1:57992" Id=118 RUNNABLE (in native)
          		at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
          		at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
          		-  locked java.lang.Object@32606c21
          		at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
          		at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp860063164-117 Selector0" Id=117 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@3f7ba2fc
          		-  locked java.util.Collections$UnmodifiableSet@4d533b8b
          		-  locked sun.nio.ch.KQueueSelectorImpl@27365e0f
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
          		at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		...
          	
          	"HashSessionScavenger-4" Id=116 TIMED_WAITING on java.util.TaskQueue@62593ee5
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.util.TaskQueue@62593ee5
          		at java.util.TimerThread.mainLoop(Timer.java:552)
          		at java.util.TimerThread.run(Timer.java:505)
          	
          	"coreZkRegister-30-thread-1" Id=113 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51305ed4
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51305ed4
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"Thread-45" Id=111 WAITING on java.lang.Object@320a1953
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Object@320a1953
          		at java.lang.Object.wait(Object.java:503)
          		at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
          	
          	"searcherExecutor-35-thread-1" Id=110 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ea1d9f0
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ea1d9f0
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=107 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73eb3baf
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73eb3baf
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=106 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@4f823fdd
          		-  locked java.util.Collections$UnmodifiableSet@19498b68
          		-  locked sun.nio.ch.KQueueSelectorImpl@41550ef
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
          		at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
          	
          	"qtp695935058-102" Id=102 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp695935058-101" Id=101 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp695935058-100" Id=100 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp695935058-99" Id=99 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp695935058-98" Id=98 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp695935058-97" Id=97 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp695935058-96 Acceptor0 SelectChannelConnector@127.0.0.1:57988" Id=96 RUNNABLE (in native)
          		at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
          		at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
          		-  locked java.lang.Object@44bad6a1
          		at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
          		at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp695935058-95 Selector0" Id=95 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@c24ea6a
          		-  locked java.util.Collections$UnmodifiableSet@6211a266
          		-  locked sun.nio.ch.KQueueSelectorImpl@7f8f54f6
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
          		at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		...
          	
          	"HashSessionScavenger-3" Id=94 TIMED_WAITING on java.util.TaskQueue@1b1323e7
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.util.TaskQueue@1b1323e7
          		at java.util.TimerThread.mainLoop(Timer.java:552)
          		at java.util.TimerThread.run(Timer.java:505)
          	
          	"coreZkRegister-21-thread-1" Id=93 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59a2094
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59a2094
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"Thread-34" Id=91 WAITING on java.lang.Object@320a1953
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Object@320a1953
          		at java.lang.Object.wait(Object.java:503)
          		at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
          	
          	"searcherExecutor-26-thread-1" Id=90 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@49d0d76b
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@49d0d76b
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=87 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2deed7a9
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2deed7a9
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=86 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@38060b85
          		-  locked java.util.Collections$UnmodifiableSet@14d97f8d
          		-  locked sun.nio.ch.KQueueSelectorImpl@5a431ae9
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
          		at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
          	
          	"qtp104014779-82" Id=82 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp104014779-81" Id=81 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp104014779-80" Id=80 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp104014779-79" Id=79 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp104014779-78" Id=78 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp104014779-77" Id=77 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp104014779-76 Acceptor0 SelectChannelConnector@127.0.0.1:57984" Id=76 RUNNABLE (in native)
          		at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
          		at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
          		-  locked java.lang.Object@47ea9477
          		at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
          		at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp104014779-75 Selector0" Id=75 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@6e6869e3
          		-  locked java.util.Collections$UnmodifiableSet@5a90f357
          		-  locked sun.nio.ch.KQueueSelectorImpl@3a7d5d
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
          		at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		...
          	
          	"HashSessionScavenger-2" Id=74 TIMED_WAITING on java.util.TaskQueue@7ffc64c8
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.util.TaskQueue@7ffc64c8
          		at java.util.TimerThread.mainLoop(Timer.java:552)
          		at java.util.TimerThread.run(Timer.java:505)
          	
          	"coreZkRegister-12-thread-1" Id=73 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133cc81d
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133cc81d
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"Thread-23" Id=71 WAITING on java.lang.Object@320a1953
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Object@320a1953
          		at java.lang.Object.wait(Object.java:503)
          		at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
          	
          	"searcherExecutor-17-thread-1" Id=70 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b75f1c8
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b75f1c8
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=67 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c54c0ca
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c54c0ca
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=66 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@18adf136
          		-  locked java.util.Collections$UnmodifiableSet@2ac95c29
          		-  locked sun.nio.ch.KQueueSelectorImpl@657d478
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
          		at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
          	
          	"qtp2004060070-62" Id=62 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp2004060070-61" Id=61 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp2004060070-59" Id=59 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp2004060070-58" Id=58 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp2004060070-57" Id=57 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp2004060070-56 Acceptor0 SelectChannelConnector@127.0.0.1:57977" Id=56 RUNNABLE (in native)
          		at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
          		at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
          		-  locked java.lang.Object@703e1588
          		at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
          		at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp2004060070-55 Selector0" Id=55 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@49c6b334
          		-  locked java.util.Collections$UnmodifiableSet@fd4cc87
          		-  locked sun.nio.ch.KQueueSelectorImpl@89a39b8
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
          		at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		...
          	
          	"HashSessionScavenger-1" Id=54 TIMED_WAITING on java.util.TaskQueue@45436371
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.util.TaskQueue@45436371
          		at java.util.TimerThread.mainLoop(Timer.java:552)
          		at java.util.TimerThread.run(Timer.java:505)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=48 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4b35b33d
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4b35b33d
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=47 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@270c253e
          		-  locked java.util.Collections$UnmodifiableSet@9d5fa4f
          		-  locked sun.nio.ch.KQueueSelectorImpl@6c7bbfee
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
          		at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
          	
          	"coreZkRegister-2-thread-1" Id=46 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3075d9e6
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3075d9e6
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"Thread-12" Id=44 WAITING on java.lang.Object@320a1953
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Object@320a1953
          		at java.lang.Object.wait(Object.java:503)
          		at org.apache.solr.core.CloserThread.run(CoreContainer.java:929)
          	
          	"searcherExecutor-8-thread-1" Id=43 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@934a3e5
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@934a3e5
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
          		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
          		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"OverseerHdfsCoreFailoverThread-93101408575684611-127.0.0.1:57971__%2Fze-n_0000000000" Id=41 TIMED_WAITING
          		at java.lang.Thread.sleep(Native Method)
          		at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"OverseerCollectionProcessor-93101408575684611-127.0.0.1:57971__%2Fze-n_0000000000" Id=40 TIMED_WAITING on java.lang.Object@5ef4141b
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Object@5ef4141b
          		at org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:273)
          		at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:317)
          		at org.apache.solr.cloud.DistributedQueue.peekTopN(DistributedQueue.java:444)
          		at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:326)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"OverseerStateUpdate-93101408575684611-127.0.0.1:57971__%2Fze-n_0000000000" Id=39 TIMED_WAITING on java.lang.Object@6a542950
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Object@6a542950
          		at org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:273)
          		at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:317)
          		at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:591)
          		at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:569)
          		at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:215)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b6479b9
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b6479b9
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=36 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@5a077b5
          		-  locked java.util.Collections$UnmodifiableSet@93b2d3b
          		-  locked sun.nio.ch.KQueueSelectorImpl@197dcb62
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349)
          		at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
          	
          	"qtp1641950186-32" Id=32 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp1641950186-31" Id=31 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp1641950186-30" Id=30 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp1641950186-29" Id=29 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp1641950186-28" Id=28 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp1641950186-27" Id=27 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01
          		at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
          		at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp1641950186-26 Acceptor0 SelectChannelConnector@127.0.0.1:57971" Id=26 RUNNABLE (in native)
          		at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
          		at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
          		-  locked java.lang.Object@26c70ebb
          		at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109)
          		at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"qtp1641950186-25 Selector0" Id=25 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@2027048a
          		-  locked java.util.Collections$UnmodifiableSet@b593a85
          		-  locked sun.nio.ch.KQueueSelectorImpl@22ee5eed
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569)
          		at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		...
          	
          	"HashSessionScavenger-0" Id=24 TIMED_WAITING on java.util.TaskQueue@4f9a2b85
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.util.TaskQueue@4f9a2b85
          		at java.util.TimerThread.mainLoop(Timer.java:552)
          		at java.util.TimerThread.run(Timer.java:505)
          	
          	"ProcessThread(sid:0 cport:-1):" Id=16 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@129f5197
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@129f5197
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120)
          	
          	"SyncThread:0" Id=15 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@19a59a0e
          		at sun.misc.Unsafe.park(Native Method)
          		-  waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@19a59a0e
          		at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
          		at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
          		at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
          		at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127)
          	
          	"SessionTracker" Id=14 TIMED_WAITING on org.apache.zookeeper.server.SessionTrackerImpl@47251a49
          		at java.lang.Object.wait(Native Method)
          		-  waiting on org.apache.zookeeper.server.SessionTrackerImpl@47251a49
          		at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146)
          	
          	"NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0" Id=13 RUNNABLE (in native)
          		at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
          		at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200)
          		at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
          		at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
          		-  locked sun.nio.ch.Util$2@54e0d31c
          		-  locked java.util.Collections$UnmodifiableSet@7443fecc
          		-  locked sun.nio.ch.KQueueSelectorImpl@1c0d3eb6
          		at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
          		at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"Thread-1" Id=12 WAITING on java.lang.Thread@29902d1d
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Thread@29902d1d
          		at java.lang.Thread.join(Thread.java:1281)
          		at java.lang.Thread.join(Thread.java:1355)
          		at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302)
          		at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:303)
          		at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:452)
          	
          	"TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]" Id=11 RUNNABLE
          		at sun.management.ThreadImpl.dumpThreads0(Native Method)
          		at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446)
          		at org.apache.solr.core.Diagnostics.logThreadDumps(Diagnostics.java:46)
          		at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:176)
          		at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:137)
          		at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:132)
          		at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:823)
          		at org.apache.solr.cloud.ShardSplitTest.splitByUniqueKeyTest(ShardSplitTest.java:240)
          		...
          	
          	"SUITE-ShardSplitTest-seed#[8490788943A65A5D]" Id=10 TIMED_WAITING on java.lang.Thread@6ed9b2f9
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.Thread@6ed9b2f9
          		at java.lang.Thread.join(Thread.java:1289)
          		at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:803)
          		at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:401)
          		at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:642)
          		at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:129)
          		at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:559)
          	
          	"Monitor Ctrl-Break" Id=8 RUNNABLE (in native)
          		at java.net.PlainSocketImpl.socketAccept(Native Method)
          		at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
          		at java.net.ServerSocket.implAccept(ServerSocket.java:530)
          		at java.net.ServerSocket.accept(ServerSocket.java:498)
          		at com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:85)
          		at java.lang.Thread.run(Thread.java:745)
          	
          	"Signal Dispatcher" Id=4 RUNNABLE
          	
          	"Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3881e03c
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.ref.ReferenceQueue$Lock@3881e03c
          		at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
          		at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
          		at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)
          	
          	"Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@56a8a01f
          		at java.lang.Object.wait(Native Method)
          		-  waiting on java.lang.ref.Reference$Lock@56a8a01f
          		at java.lang.Object.wait(Object.java:503)
          		at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
          	
          	"main" Id=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$1@3058b2a9
          		at java.lang.Object.wait(Native Method)
          		-  waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$1@3058b2a9
          		at java.lang.Thread.join(Thread.java:1281)
          		at java.lang.Thread.join(Thread.java:1355)
          		at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:569)
          		at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:435)
          		at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
          		at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
          		at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
          		...
          

          Looking into why are the RecoveryThreads timing out.

          Show
          Varun Thacker added a comment - Few tests fail with this patch - [junit4] Tests with failures: [junit4] - org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch [junit4] - org.apache.solr.cloud.ShardSplitTest.testDistribSearch [junit4] - org.apache.solr.cloud.SyncSliceTest.testDistribSearch [junit4] - org.apache.solr.cloud.RecoveryZkTest.testDistribSearch I picked ShardSplitTest.testDistribSearch and started investigating why is it failing always with the patch - I was seeing the following stack trace - 53340 T166 C65 P58357 oasc.SolrException.log ERROR SnapPull failed :org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1603) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1715) at org.apache.solr.handler.SnapPuller.openNewSearcherAndUpdateCommitPoint(SnapPuller.java:680) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:496) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:340) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235) Caused by: java.io.FileNotFoundException: _1_2.liv in dir=RAMDirectory@2fdfb9bd lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@91b3fd9 at org.apache.lucene.store.MockDirectoryWrapper.openInput(MockDirectoryWrapper.java:655) at org.apache.lucene.store.Directory.openChecksumInput(Directory.java:110) at org.apache.lucene.codecs.lucene50.Lucene50LiveDocsFormat.readLiveDocs(Lucene50LiveDocsFormat.java:84) at org.apache.lucene.codecs.asserting.AssertingLiveDocsFormat.readLiveDocs(AssertingLiveDocsFormat.java:64) at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:101) at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:134) at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:186) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:94) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:430) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:203) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1522) ... 7 more 53344 T166 C65 P58357 oasc.SolrException.log ERROR Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed. at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:166) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235) This looked very similar to the problem we were trying to solve in SOLR-6640 . So I applied the patch from SOLR-6640 along with this patch and now I am seeing these - 392008 T11 oasc.Diagnostics.logThreadDumps ERROR Gave up waiting for recovery to finish. THREAD DUMP: "qtp2004060070-174" Id=174 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "RecoveryThread-collection1_shard1_1_replica2" Id=168 TIMED_WAITING on java.lang. Object @1398321c at java.lang. Object .wait(Native Method) - waiting on java.lang. Object @1398321c at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:209) - locked org.apache.solr.update.DefaultSolrCoreState@50d256b5 at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:469) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:340) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235) Number of locked synchronizers = 1 - java.util.concurrent.locks.ReentrantLock$NonfairSync@213aae87 "searcherExecutor-61-thread-1" Id=166 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d01d650 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6d01d650 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) "RecoveryThread-collection1_shard1_0_replica2" Id=163 TIMED_WAITING on java.lang. Object @50e7ed31 at java.lang. Object .wait(Native Method) - waiting on java.lang. Object @50e7ed31 at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:209) - locked org.apache.solr.update.DefaultSolrCoreState@3e7f8d5e at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:469) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:340) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:163) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:235) Number of locked synchronizers = 1 - java.util.concurrent.locks.ReentrantLock$NonfairSync@281a3660 "searcherExecutor-56-thread-1" Id=161 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a855576 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a855576 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) "searcherExecutor-52-thread-1" Id=156 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17adcbc9 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17adcbc9 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) "searcherExecutor-48-thread-1" Id=154 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@131e0544 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@131e0544 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) "OverseerThreadFactory-5-thread-4" Id=152 WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... "OverseerThreadFactory-5-thread-3" Id=146 WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... "OverseerThreadFactory-5-thread-2" Id=145 WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... "OverseerThreadFactory-5-thread-1" Id=144 WAITING on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.SynchronousQueue$TransferStack@17ebbd2a at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359) at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:925) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ... "coreZkRegister-39-thread-1" Id=136 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75d01caa at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@75d01caa at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) " Thread -57" Id=134 WAITING on java.lang. Object @320a1953 at java.lang. Object .wait(Native Method) - waiting on java.lang. Object @320a1953 at java.lang. Object .wait( Object .java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:929) "searcherExecutor-44-thread-1" Id=133 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3504b1cf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3504b1cf at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=129 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4735a83a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4735a83a at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=128 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@7b7a6620 - locked java.util.Collections$UnmodifiableSet@685b5a62 - locked sun.nio.ch.KQueueSelectorImpl@2c205e57 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) "qtp860063164-124" Id=124 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp860063164-123" Id=123 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp860063164-122" Id=122 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp860063164-121" Id=121 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp860063164-120" Id=120 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp860063164-119" Id=119 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@111b863 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp860063164-118 Acceptor0 SelectChannelConnector@127.0.0.1:57992" Id=118 RUNNABLE (in native ) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) - locked java.lang. Object @32606c21 at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang. Thread .run( Thread .java:745) "qtp860063164-117 Selector0" Id=117 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@3f7ba2fc - locked java.util.Collections$UnmodifiableSet@4d533b8b - locked sun.nio.ch.KQueueSelectorImpl@27365e0f at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) ... "HashSessionScavenger-4" Id=116 TIMED_WAITING on java.util.TaskQueue@62593ee5 at java.lang. Object .wait(Native Method) - waiting on java.util.TaskQueue@62593ee5 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "coreZkRegister-30-thread-1" Id=113 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51305ed4 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@51305ed4 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) " Thread -45" Id=111 WAITING on java.lang. Object @320a1953 at java.lang. Object .wait(Native Method) - waiting on java.lang. Object @320a1953 at java.lang. Object .wait( Object .java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:929) "searcherExecutor-35-thread-1" Id=110 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ea1d9f0 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ea1d9f0 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=107 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73eb3baf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@73eb3baf at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=106 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@4f823fdd - locked java.util.Collections$UnmodifiableSet@19498b68 - locked sun.nio.ch.KQueueSelectorImpl@41550ef at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) "qtp695935058-102" Id=102 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp695935058-101" Id=101 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp695935058-100" Id=100 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp695935058-99" Id=99 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp695935058-98" Id=98 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp695935058-97" Id=97 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@13747d01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp695935058-96 Acceptor0 SelectChannelConnector@127.0.0.1:57988" Id=96 RUNNABLE (in native ) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) - locked java.lang. Object @44bad6a1 at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang. Thread .run( Thread .java:745) "qtp695935058-95 Selector0" Id=95 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@c24ea6a - locked java.util.Collections$UnmodifiableSet@6211a266 - locked sun.nio.ch.KQueueSelectorImpl@7f8f54f6 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) ... "HashSessionScavenger-3" Id=94 TIMED_WAITING on java.util.TaskQueue@1b1323e7 at java.lang. Object .wait(Native Method) - waiting on java.util.TaskQueue@1b1323e7 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "coreZkRegister-21-thread-1" Id=93 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59a2094 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59a2094 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) " Thread -34" Id=91 WAITING on java.lang. Object @320a1953 at java.lang. Object .wait(Native Method) - waiting on java.lang. Object @320a1953 at java.lang. Object .wait( Object .java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:929) "searcherExecutor-26-thread-1" Id=90 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@49d0d76b at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@49d0d76b at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=87 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2deed7a9 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2deed7a9 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=86 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@38060b85 - locked java.util.Collections$UnmodifiableSet@14d97f8d - locked sun.nio.ch.KQueueSelectorImpl@5a431ae9 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) "qtp104014779-82" Id=82 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp104014779-81" Id=81 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp104014779-80" Id=80 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp104014779-79" Id=79 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp104014779-78" Id=78 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp104014779-77" Id=77 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6b9be54a at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp104014779-76 Acceptor0 SelectChannelConnector@127.0.0.1:57984" Id=76 RUNNABLE (in native ) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) - locked java.lang. Object @47ea9477 at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang. Thread .run( Thread .java:745) "qtp104014779-75 Selector0" Id=75 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@6e6869e3 - locked java.util.Collections$UnmodifiableSet@5a90f357 - locked sun.nio.ch.KQueueSelectorImpl@3a7d5d at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) ... "HashSessionScavenger-2" Id=74 TIMED_WAITING on java.util.TaskQueue@7ffc64c8 at java.lang. Object .wait(Native Method) - waiting on java.util.TaskQueue@7ffc64c8 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "coreZkRegister-12-thread-1" Id=73 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133cc81d at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@133cc81d at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) " Thread -23" Id=71 WAITING on java.lang. Object @320a1953 at java.lang. Object .wait(Native Method) - waiting on java.lang. Object @320a1953 at java.lang. Object .wait( Object .java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:929) "searcherExecutor-17-thread-1" Id=70 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b75f1c8 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b75f1c8 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=67 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c54c0ca at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c54c0ca at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=66 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@18adf136 - locked java.util.Collections$UnmodifiableSet@2ac95c29 - locked sun.nio.ch.KQueueSelectorImpl@657d478 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) "qtp2004060070-62" Id=62 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp2004060070-61" Id=61 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp2004060070-59" Id=59 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp2004060070-58" Id=58 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp2004060070-57" Id=57 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42da41bf at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp2004060070-56 Acceptor0 SelectChannelConnector@127.0.0.1:57977" Id=56 RUNNABLE (in native ) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) - locked java.lang. Object @703e1588 at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang. Thread .run( Thread .java:745) "qtp2004060070-55 Selector0" Id=55 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@49c6b334 - locked java.util.Collections$UnmodifiableSet@fd4cc87 - locked sun.nio.ch.KQueueSelectorImpl@89a39b8 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) ... "HashSessionScavenger-1" Id=54 TIMED_WAITING on java.util.TaskQueue@45436371 at java.lang. Object .wait(Native Method) - waiting on java.util.TaskQueue@45436371 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=48 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4b35b33d at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4b35b33d at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=47 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@270c253e - locked java.util.Collections$UnmodifiableSet@9d5fa4f - locked sun.nio.ch.KQueueSelectorImpl@6c7bbfee at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) "coreZkRegister-2-thread-1" Id=46 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3075d9e6 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3075d9e6 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) " Thread -12" Id=44 WAITING on java.lang. Object @320a1953 at java.lang. Object .wait(Native Method) - waiting on java.lang. Object @320a1953 at java.lang. Object .wait( Object .java:503) at org.apache.solr.core.CloserThread.run(CoreContainer.java:929) "searcherExecutor-8-thread-1" Id=43 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@934a3e5 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@934a3e5 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang. Thread .run( Thread .java:745) "OverseerHdfsCoreFailoverThread-93101408575684611-127.0.0.1:57971__%2Fze-n_0000000000" Id=41 TIMED_WAITING at java.lang. Thread .sleep(Native Method) at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:136) at java.lang. Thread .run( Thread .java:745) "OverseerCollectionProcessor-93101408575684611-127.0.0.1:57971__%2Fze-n_0000000000" Id=40 TIMED_WAITING on java.lang. Object @5ef4141b at java.lang. Object .wait(Native Method) - waiting on java.lang. Object @5ef4141b at org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:273) at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:317) at org.apache.solr.cloud.DistributedQueue.peekTopN(DistributedQueue.java:444) at org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:326) at java.lang. Thread .run( Thread .java:745) "OverseerStateUpdate-93101408575684611-127.0.0.1:57971__%2Fze-n_0000000000" Id=39 TIMED_WAITING on java.lang. Object @6a542950 at java.lang. Object .wait(Native Method) - waiting on java.lang. Object @6a542950 at org.apache.solr.cloud.DistributedQueue$LatchWatcher.await(DistributedQueue.java:273) at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:317) at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:591) at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:569) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:215) at java.lang. Thread .run( Thread .java:745) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-EventThread" Id=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b6479b9 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b6479b9 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]-SendThread(127.0.0.1:57967)" Id=36 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@5a077b5 - locked java.util.Collections$UnmodifiableSet@93b2d3b - locked sun.nio.ch.KQueueSelectorImpl@197dcb62 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:349) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081) "qtp1641950186-32" Id=32 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp1641950186-31" Id=31 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp1641950186-30" Id=30 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp1641950186-29" Id=29 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp1641950186-28" Id=28 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp1641950186-27" Id=27 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11b39c01 at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:342) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:526) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:44) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572) at java.lang. Thread .run( Thread .java:745) "qtp1641950186-26 Acceptor0 SelectChannelConnector@127.0.0.1:57971" Id=26 RUNNABLE (in native ) at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241) - locked java.lang. Object @26c70ebb at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:109) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:938) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang. Thread .run( Thread .java:745) "qtp1641950186-25 Selector0" Id=25 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@2027048a - locked java.util.Collections$UnmodifiableSet@b593a85 - locked sun.nio.ch.KQueueSelectorImpl@22ee5eed at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:569) at org.eclipse.jetty.io.nio.SelectorManager$1.run(SelectorManager.java:290) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) ... "HashSessionScavenger-0" Id=24 TIMED_WAITING on java.util.TaskQueue@4f9a2b85 at java.lang. Object .wait(Native Method) - waiting on java.util.TaskQueue@4f9a2b85 at java.util.TimerThread.mainLoop(Timer.java:552) at java.util.TimerThread.run(Timer.java:505) "ProcessThread(sid:0 cport:-1):" Id=16 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@129f5197 at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@129f5197 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:120) "SyncThread:0" Id=15 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@19a59a0e at sun.misc.Unsafe.park(Native Method) - waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@19a59a0e at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:127) "SessionTracker" Id=14 TIMED_WAITING on org.apache.zookeeper.server.SessionTrackerImpl@47251a49 at java.lang. Object .wait(Native Method) - waiting on org.apache.zookeeper.server.SessionTrackerImpl@47251a49 at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) "NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0" Id=13 RUNNABLE (in native ) at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:200) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked sun.nio.ch.Util$2@54e0d31c - locked java.util.Collections$UnmodifiableSet@7443fecc - locked sun.nio.ch.KQueueSelectorImpl@1c0d3eb6 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:178) at java.lang. Thread .run( Thread .java:745) " Thread -1" Id=12 WAITING on java.lang. Thread @29902d1d at java.lang. Object .wait(Native Method) - waiting on java.lang. Thread @29902d1d at java.lang. Thread .join( Thread .java:1281) at java.lang. Thread .join( Thread .java:1355) at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:302) at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:303) at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:452) "TEST-ShardSplitTest.testDistribSearch-seed#[8490788943A65A5D]" Id=11 RUNNABLE at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:446) at org.apache.solr.core.Diagnostics.logThreadDumps(Diagnostics.java:46) at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:176) at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:137) at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:132) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:823) at org.apache.solr.cloud.ShardSplitTest.splitByUniqueKeyTest(ShardSplitTest.java:240) ... "SUITE-ShardSplitTest-seed#[8490788943A65A5D]" Id=10 TIMED_WAITING on java.lang. Thread @6ed9b2f9 at java.lang. Object .wait(Native Method) - waiting on java.lang. Thread @6ed9b2f9 at java.lang. Thread .join( Thread .java:1289) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:803) at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:401) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:642) at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:129) at com.carrotsearch.randomizedtesting.RandomizedRunner$1.run(RandomizedRunner.java:559) "Monitor Ctrl-Break" Id=8 RUNNABLE (in native ) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:530) at java.net.ServerSocket.accept(ServerSocket.java:498) at com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:85) at java.lang. Thread .run( Thread .java:745) "Signal Dispatcher" Id=4 RUNNABLE "Finalizer" Id=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3881e03c at java.lang. Object .wait(Native Method) - waiting on java.lang.ref.ReferenceQueue$Lock@3881e03c at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189) "Reference Handler" Id=2 WAITING on java.lang.ref.Reference$Lock@56a8a01f at java.lang. Object .wait(Native Method) - waiting on java.lang.ref.Reference$Lock@56a8a01f at java.lang. Object .wait( Object .java:503) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133) "main" Id=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$1@3058b2a9 at java.lang. Object .wait(Native Method) - waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$1@3058b2a9 at java.lang. Thread .join( Thread .java:1281) at java.lang. Thread .join( Thread .java:1355) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:569) at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:435) at org.junit.runner.JUnitCore.run(JUnitCore.java:157) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211) ... Looking into why are the RecoveryThreads timing out.
          Hide
          Varun Thacker added a comment -

          When using a try-finally block code in SOLR-6440 the threads don't hang anymore.

          The test still fails because of errors like this -

          391305 T11 C196 P59014 oasu.SolrIndexWriter.close ERROR Error closing IndexWriter java.lang.AssertionError: file _2_2.liv does not exist; files=[segments_2, _1.cfs, _3.si, _2.cfe, _1.si, _1.cfe, _3.cfe, _0.cfs, _0.cfe, _2.si, _0.si, _3.cfs, _2.cfs]
          	at org.apache.lucene.index.IndexWriter.filesExist(IndexWriter.java:4232)
          	at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4303)
          	at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2785)
          	at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2888)
          	at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:965)
          	at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1008)
          	at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:129)
          	at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:804)
          	at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:68)
          	at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:359)
          	at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:72)
          	at org.apache.solr.core.SolrCore.close(SolrCore.java:1110)
          	at org.apache.solr.core.SolrCores.close(SolrCores.java:117)
          	at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:366)
          	at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:194)
          	at org.apache.solr.cloud.ChaosMonkey.stopJettySolrRunner(ChaosMonkey.java:197)
          	at org.apache.solr.cloud.ChaosMonkey.stop(ChaosMonkey.java:550)
          	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.destroyServers(AbstractFullDistribZkTestBase.java:1568)
          	at org.apache.solr.BaseDistributedSearchTestCase.tearDown(BaseDistributedSearchTestCase.java:283)
          	at org.apache.solr.cloud.AbstractDistribZkTestBase.tearDown(AbstractDistribZkTestBase.java:231)
          	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.tearDown(AbstractFullDistribZkTestBase.java:1546)
          	at org.apache.solr.cloud.BasicDistributedZkTest.tearDown(BasicDistributedZkTest.java:1166)
          	at org.apache.solr.cloud.ShardSplitTest.tearDown(ShardSplitTest.java:91)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          	at java.lang.reflect.Method.invoke(Method.java:606)
          	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618)
          	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:885)
          	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
          	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
          	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
          	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
          	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
          	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
          	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
          	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
          	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
          	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
          	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
          	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836)
          	at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738)
          	at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772)
          	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783)
          	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
          	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
          	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
          	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
          	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
          	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
          	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
          	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
          	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
          	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
          	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
          	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
          	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
          	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
          	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
          	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
          	at java.lang.Thread.run(Thread.java:745)
          

          No .liv files get included with the logic change in ReplicationHandler.getFileList with the patch. Looking into that now.

          Show
          Varun Thacker added a comment - When using a try-finally block code in SOLR-6440 the threads don't hang anymore. The test still fails because of errors like this - 391305 T11 C196 P59014 oasu.SolrIndexWriter.close ERROR Error closing IndexWriter java.lang.AssertionError: file _2_2.liv does not exist; files=[segments_2, _1.cfs, _3.si, _2.cfe, _1.si, _1.cfe, _3.cfe, _0.cfs, _0.cfe, _2.si, _0.si, _3.cfs, _2.cfs] at org.apache.lucene.index.IndexWriter.filesExist(IndexWriter.java:4232) at org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4303) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2785) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2888) at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:965) at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1008) at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:129) at org.apache.solr.update.DirectUpdateHandler2.closeWriter(DirectUpdateHandler2.java:804) at org.apache.solr.update.DefaultSolrCoreState.closeIndexWriter(DefaultSolrCoreState.java:68) at org.apache.solr.update.DefaultSolrCoreState.close(DefaultSolrCoreState.java:359) at org.apache.solr.update.SolrCoreState.decrefSolrCoreState(SolrCoreState.java:72) at org.apache.solr.core.SolrCore.close(SolrCore.java:1110) at org.apache.solr.core.SolrCores.close(SolrCores.java:117) at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:366) at org.apache.solr.servlet.SolrDispatchFilter.destroy(SolrDispatchFilter.java:194) at org.apache.solr.cloud.ChaosMonkey.stopJettySolrRunner(ChaosMonkey.java:197) at org.apache.solr.cloud.ChaosMonkey.stop(ChaosMonkey.java:550) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.destroyServers(AbstractFullDistribZkTestBase.java:1568) at org.apache.solr.BaseDistributedSearchTestCase.tearDown(BaseDistributedSearchTestCase.java:283) at org.apache.solr.cloud.AbstractDistribZkTestBase.tearDown(AbstractDistribZkTestBase.java:231) at org.apache.solr.cloud.AbstractFullDistribZkTestBase.tearDown(AbstractFullDistribZkTestBase.java:1546) at org.apache.solr.cloud.BasicDistributedZkTest.tearDown(BasicDistributedZkTest.java:1166) at org.apache.solr.cloud.ShardSplitTest.tearDown(ShardSplitTest.java:91) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1618) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:885) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:738) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:772) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:783) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang. Thread .run( Thread .java:745) No .liv files get included with the logic change in ReplicationHandler.getFileList with the patch. Looking into that now.
          Hide
          Varun Thacker added a comment -

          Updated patch. Tests pass.

          The difference from the earlier patch is the API usage of SegmentInfos

          For reference, here is the link to the question I asked a question on the lucene user mailing list about the SegmentInfos API usage - http://mail-archives.apache.org/mod_mbox/lucene-java-user/201501.mbox/%3CCAEH2wZDm%2BEXEhWEyp9RoQDVffb7jJSG31A3WVGxV_TNCE%3D12zA%40mail.gmail.com%3E

          Show
          Varun Thacker added a comment - Updated patch. Tests pass. The difference from the earlier patch is the API usage of SegmentInfos For reference, here is the link to the question I asked a question on the lucene user mailing list about the SegmentInfos API usage - http://mail-archives.apache.org/mod_mbox/lucene-java-user/201501.mbox/%3CCAEH2wZDm%2BEXEhWEyp9RoQDVffb7jJSG31A3WVGxV_TNCE%3D12zA%40mail.gmail.com%3E
          Hide
          Mark Miller added a comment -

          This looks good. We should not need the back compat < 5.0 code AFAICT though.

          Show
          Mark Miller added a comment - This looks good. We should not need the back compat < 5.0 code AFAICT though.
          Hide
          Varun Thacker added a comment -

          Hi Mark Miller,

          Thanks for the review!

          We should not need the back compat < 5.0 code AFAICT though.

          From what I understand an index built pre LUCENE-2446 would not have the checksum , so CodecUtil.retrieveChecksum(in) would throw an error as the validation would fail. I guess the check should be 4_8 and not 5_0 though.

          Show
          Varun Thacker added a comment - Hi Mark Miller , Thanks for the review! We should not need the back compat < 5.0 code AFAICT though. From what I understand an index built pre LUCENE-2446 would not have the checksum , so CodecUtil.retrieveChecksum(in) would throw an error as the validation would fail. I guess the check should be 4_8 and not 5_0 though.
          Hide
          Mark Miller added a comment -

          But isnt this just checking what solrconfig is set to? And that doesn't have to match what the index was built with?

          Show
          Mark Miller added a comment - But isnt this just checking what solrconfig is set to? And that doesn't have to match what the index was built with?
          Hide
          Varun Thacker added a comment -

          Updated patch. This handles the back compat check correctly

          While running the tests I got a failure which can be reproduced with

          ant test  -Dtestcase=SyncSliceTest -Dtests.method=test -Dtests.seed=588DD6F3A8F57A44 -Dtests.slow=true -Dtests.locale=no_NO_NY -Dtests.timezone=America/Bahia -Dtests.asserts=true -Dtests.file.encoding=UTF-8

          The exception thrown is -

          131990 T79 C17 P63349 oasc.SolrException.log ERROR java.lang.ArrayIndexOutOfBoundsException: -8
          		at org.apache.lucene.store.RAMInputStream.readByte(RAMInputStream.java:73)
          		at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
          		at org.apache.lucene.store.MockIndexInputWrapper.readInt(MockIndexInputWrapper.java:159)
          		at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:414)
          		at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:401)
          		at org.apache.solr.handler.ReplicationHandler.getFileList(ReplicationHandler.java:445)
          		at org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:212)
          		at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
          		at org.apache.solr.core.SolrCore.execute(SolrCore.java:2006)
          		at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
          		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
          		at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:204)
          		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
          		at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:142)
          		at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
          		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
          		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
          		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
          		at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301)
          		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077)
          		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
          		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
          		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
          		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
          		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
          		at org.eclipse.jetty.server.Server.handle(Server.java:368)
          		at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
          		at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
          		at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
          		at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
          		at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
          		at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
          		at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
          		at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
          		at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
          		at java.lang.Thread.run(Thread.java:745)
          

          On debugging I found out the file which was causing it -
          _0_MockRandom_0.sd . This is a MockRandomPostingsFormat.SEED_EXT fille.

          Adding this to SyncSliceTest fixed the fail -

          @LuceneTestCase.SuppressCodecs({ "MockRandom" })

          but any other test could end up using it causing a failure. Any idea on how to tackle it

          Show
          Varun Thacker added a comment - Updated patch. This handles the back compat check correctly While running the tests I got a failure which can be reproduced with ant test -Dtestcase=SyncSliceTest -Dtests.method=test -Dtests.seed=588DD6F3A8F57A44 -Dtests.slow=true -Dtests.locale=no_NO_NY -Dtests.timezone=America/Bahia -Dtests.asserts=true -Dtests.file.encoding=UTF-8 The exception thrown is - 131990 T79 C17 P63349 oasc.SolrException.log ERROR java.lang.ArrayIndexOutOfBoundsException: -8 at org.apache.lucene.store.RAMInputStream.readByte(RAMInputStream.java:73) at org.apache.lucene.store.DataInput.readInt(DataInput.java:98) at org.apache.lucene.store.MockIndexInputWrapper.readInt(MockIndexInputWrapper.java:159) at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:414) at org.apache.lucene.codecs.CodecUtil.retrieveChecksum(CodecUtil.java:401) at org.apache.solr.handler.ReplicationHandler.getFileList(ReplicationHandler.java:445) at org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:212) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144) at org.apache.solr.core.SolrCore.execute(SolrCore.java:2006) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:204) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:142) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:301) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1077) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:368) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang. Thread .run( Thread .java:745) On debugging I found out the file which was causing it - _0_MockRandom_0.sd . This is a MockRandomPostingsFormat.SEED_EXT fille. Adding this to SyncSliceTest fixed the fail - @LuceneTestCase.SuppressCodecs({ "MockRandom" }) but any other test could end up using it causing a failure. Any idea on how to tackle it
          Hide
          Mark Miller added a comment -

          Varun Thacker, can't we just calculate the checksum ourselves if we don't find it? So that we treat old segments right as well? This issue is critical for avoiding corrupted indexes on replication.

          Show
          Mark Miller added a comment - Varun Thacker , can't we just calculate the checksum ourselves if we don't find it? So that we treat old segments right as well? This issue is critical for avoiding corrupted indexes on replication.
          Hide
          Mark Miller added a comment -

          Any idea on how to tackle it

          It seems like we can't be sure every file will have a checksum (at least in tests) and we have to recover properly if we find that it doesn't?

          Show
          Mark Miller added a comment - Any idea on how to tackle it It seems like we can't be sure every file will have a checksum (at least in tests) and we have to recover properly if we find that it doesn't?
          Hide
          Mark Miller added a comment -

          can't we just calculate the checksum ourselves if we don't find it?

          I guess it would be fairly expensive for large files. For critical small files (the ones that always seem to come as the problem), perhaps we can do the checksum ourselves, or more likely, just always down .si and segments_n files regardless.

          Show
          Mark Miller added a comment - can't we just calculate the checksum ourselves if we don't find it? I guess it would be fairly expensive for large files. For critical small files (the ones that always seem to come as the problem), perhaps we can do the checksum ourselves, or more likely, just always down .si and segments_n files regardless.
          Hide
          Varun Thacker added a comment -

          I guess it would be fairly expensive for large files. For critical small files (the ones that always seem to come as the problem), perhaps we can do the checksum ourselves, or more likely, just always down .si and segments_n files regardless.

          Nice idea! So .si/segments_n/.liv should be downloaded again regardless.

          Show
          Varun Thacker added a comment - I guess it would be fairly expensive for large files. For critical small files (the ones that always seem to come as the problem), perhaps we can do the checksum ourselves, or more likely, just always down .si and segments_n files regardless. Nice idea! So .si/segments_n/.liv should be downloaded again regardless.
          Hide
          Mark Miller added a comment -

          Here is a first pass at making the patch suitable for trunk.

          Show
          Mark Miller added a comment - Here is a first pass at making the patch suitable for trunk.
          Hide
          Mark Miller added a comment -

          For 5x, we will want to put the version checks back in.

          Show
          Mark Miller added a comment - For 5x, we will want to put the version checks back in.
          Hide
          Varun Thacker added a comment -

          Mark,

          In your patch if the file fails to retrieve a checksum and if its a .si/segments_n file then we will have to re-download the complete index. the reason being isStaleIndex will return true.

          Instead if we moved to the check into the if statement in downloadIndexFiles() to always re-download those files won't it be better?

          I think you have an extra file from HdfsTestUtil in the patch.

          Show
          Varun Thacker added a comment - Mark, In your patch if the file fails to retrieve a checksum and if its a .si/segments_n file then we will have to re-download the complete index. the reason being isStaleIndex will return true. Instead if we moved to the check into the if statement in downloadIndexFiles() to always re-download those files won't it be better? I think you have an extra file from HdfsTestUtil in the patch.
          Hide
          Mark Miller added a comment -

          In your patch if the file fails to retrieve a checksum...

          Feel free to put up a follow on patch - this one is just a first draft. I'll be taking another pass when I get a chance as well.

          I think you have an extra file from HdfsTestUtil in the patch.

          No, that's fixes an error I see in the test that shalin mentions in SOLR-6640 and led me back to this issue.

          Show
          Mark Miller added a comment - In your patch if the file fails to retrieve a checksum... Feel free to put up a follow on patch - this one is just a first draft. I'll be taking another pass when I get a chance as well. I think you have an extra file from HdfsTestUtil in the patch. No, that's fixes an error I see in the test that shalin mentions in SOLR-6640 and led me back to this issue.
          Hide
          Mark Miller added a comment -

          Another patch incorporating Varun's feedback and minor tweaks.

          Show
          Mark Miller added a comment - Another patch incorporating Varun's feedback and minor tweaks.
          Hide
          Mark Miller added a comment -

          That patch did pick up cruft (TopGroupsShardResponseProcessor) from fooling around trying to fix a Solr fail triggered by a Lucene change. I'll put up a clean one in a moment.

          Show
          Mark Miller added a comment - That patch did pick up cruft (TopGroupsShardResponseProcessor) from fooling around trying to fix a Solr fail triggered by a Lucene change. I'll put up a clean one in a moment.
          Hide
          Varun Thacker added a comment -

          1. Changed the log message in SnapPuller.compareFile() since checksum won't be present in case of exception.
          From - LOG.warn("File {} did not match. expected checksum is {} and actual is checksum {}. " + "expected length is {} and actual length is {}", filename, backupIndexFileChecksum indexFileChecksum, backupIndexFileLen, indexFileLen);
          To - LOG.warn("File {} did not match. expected length is {} and actual length is {}", filename,backupIndexFileLen, indexFileLen);

          2. In SnapPuller.downloadIndexFiles made (String) file.get(NAME)) into a variable for better readibility.

          3. The if condition still needs tweaking? We could still have a non checksummed/checksum threw error .si/.liv/segments_n file and be equal in length and we wouldn't re-download?

          Maybe the condition could be the check you initially proposed -
          1. If file is a .si/.liv/segments_n file download regardless
          2. else if if (!compareResult.equal || downloadCompleteIndex) then re-download?

          Show
          Varun Thacker added a comment - 1. Changed the log message in SnapPuller.compareFile() since checksum won't be present in case of exception. From - LOG.warn("File {} did not match. expected checksum is {} and actual is checksum {}. " + "expected length is {} and actual length is {}", filename, backupIndexFileChecksum indexFileChecksum, backupIndexFileLen, indexFileLen); To - LOG.warn("File {} did not match. expected length is {} and actual length is {}", filename,backupIndexFileLen, indexFileLen); 2. In SnapPuller.downloadIndexFiles made (String) file.get(NAME)) into a variable for better readibility. 3. The if condition still needs tweaking? We could still have a non checksummed/checksum threw error .si/.liv/segments_n file and be equal in length and we wouldn't re-download? Maybe the condition could be the check you initially proposed - 1. If file is a .si/.liv/segments_n file download regardless 2. else if if (!compareResult.equal || downloadCompleteIndex) then re-download?
          Hide
          Mark Miller added a comment -

          3

          I'm not following - what are you seeing exactly? I read it as 'if a file was not checksummed, download anyway if it matches this file list'.

          || (!compareResult.checkSummed && (((String) file.get(NAME)).endsWith(".si") || ((String) file.get(NAME)).endsWith(".liv") || ((String) file.get(NAME)).startsWith("segments_")))

          If the file is checksummed, we can rely on the checksum. What am I missing that you are seeing?

          Show
          Mark Miller added a comment - 3 I'm not following - what are you seeing exactly? I read it as 'if a file was not checksummed, download anyway if it matches this file list'. || (!compareResult.checkSummed && (((String) file.get(NAME)).endsWith(".si") || ((String) file.get(NAME)).endsWith(".liv") || ((String) file.get(NAME)).startsWith("segments_"))) If the file is checksummed, we can rely on the checksum. What am I missing that you are seeing?
          Hide
          Varun Thacker added a comment -

          You're right. I got confused a bit with this portion !compareResult.equal in the statement and replied too early.

          Show
          Varun Thacker added a comment - You're right. I got confused a bit with this portion !compareResult.equal in the statement and replied too early.
          Hide
          Mark Miller added a comment -

          Okay - I think we should put this in trunk and 5x to start jenkins on it.

          I've moved my local machine to only running the 9 chaosmonkey jobs I have.

          If anyone is following along, more review would be good.

          I'll commit to trunk and let's work on a 5x patch.

          Show
          Mark Miller added a comment - Okay - I think we should put this in trunk and 5x to start jenkins on it. I've moved my local machine to only running the 9 chaosmonkey jobs I have. If anyone is following along, more review would be good. I'll commit to trunk and let's work on a 5x patch.
          Hide
          ASF subversion and git services added a comment -

          Commit 1657969 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1657969 ]

          SOLR-6920, SOLR-6640: A replicated index can end up corrupted when small files end up with the same file name and size.

          Show
          ASF subversion and git services added a comment - Commit 1657969 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1657969 ] SOLR-6920 , SOLR-6640 : A replicated index can end up corrupted when small files end up with the same file name and size.
          Hide
          Mark Miller added a comment -

          First pass at the 5x version.

          Show
          Mark Miller added a comment - First pass at the 5x version.
          Hide
          Mark Miller added a comment -

          New 5x patch to fix where Version is parsed in SnapPuller.

          Show
          Mark Miller added a comment - New 5x patch to fix where Version is parsed in SnapPuller.
          Hide
          Varun Thacker added a comment -

          Hi Mark,

          In the 5.x patch, should we wrap fileMeta.put(CHECKSUM, CodecUtil.retrieveChecksum(in)); for the segments_n file in a try-catch block?

          Show
          Varun Thacker added a comment - Hi Mark, In the 5.x patch, should we wrap fileMeta.put(CHECKSUM, CodecUtil.retrieveChecksum(in)); for the segments_n file in a try-catch block?
          Hide
          Mark Miller added a comment -

          Yeah, good catch. I'd almost like to do that on trunk as well just for consistency.

          Show
          Mark Miller added a comment - Yeah, good catch. I'd almost like to do that on trunk as well just for consistency.
          Hide
          Mark Miller added a comment -

          New patch - cleaned it up a bit, handled Varun's comment.

          Show
          Mark Miller added a comment - New patch - cleaned it up a bit, handled Varun's comment.
          Hide
          Mark Miller added a comment - - edited

          Been thinking about adding this since last night - should go in the trunk version too - this patch will also download any files under 100kb if there is no checksum - more future proof for new small files, alternate codec proof, etc.

          Show
          Mark Miller added a comment - - edited Been thinking about adding this since last night - should go in the trunk version too - this patch will also download any files under 100kb if there is no checksum - more future proof for new small files, alternate codec proof, etc.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658078 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1658078 ]

          SOLR-6920, SOLR-6640: When we so not have a checksum for a file, always download files under 100kb and other small improvements.

          Show
          ASF subversion and git services added a comment - Commit 1658078 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1658078 ] SOLR-6920 , SOLR-6640 : When we so not have a checksum for a file, always download files under 100kb and other small improvements.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658083 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1658083 ]

          SOLR-6920, SOLR-6640: A replicated index can end up corrupted when small files end up with the same file name and size.

          Show
          ASF subversion and git services added a comment - Commit 1658083 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658083 ] SOLR-6920 , SOLR-6640 : A replicated index can end up corrupted when small files end up with the same file name and size.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658089 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1658089 ]

          SOLR-6920, SOLR-6640: Do not log an error when a file does not exist for comparison.

          Show
          ASF subversion and git services added a comment - Commit 1658089 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1658089 ] SOLR-6920 , SOLR-6640 : Do not log an error when a file does not exist for comparison.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658090 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1658090 ]

          SOLR-6920, SOLR-6640: Do not log an error when a file does not exist for comparison.

          Show
          ASF subversion and git services added a comment - Commit 1658090 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658090 ] SOLR-6920 , SOLR-6640 : Do not log an error when a file does not exist for comparison.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658129 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1658129 ]

          SOLR-6920, SOLR-6640: Rename method to filesToAlwaysDownloadIfNoChecksums.

          Show
          ASF subversion and git services added a comment - Commit 1658129 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1658129 ] SOLR-6920 , SOLR-6640 : Rename method to filesToAlwaysDownloadIfNoChecksums.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658130 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1658130 ]

          SOLR-6920, SOLR-6640: Rename method to filesToAlwaysDownloadIfNoChecksums.

          Show
          ASF subversion and git services added a comment - Commit 1658130 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658130 ] SOLR-6920 , SOLR-6640 : Rename method to filesToAlwaysDownloadIfNoChecksums.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658240 from Mark Miller in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1658240 ]

          SOLR-6920, SOLR-6640: A replicated index can end up corrupted when small files end up with the same file name and size.

          Show
          ASF subversion and git services added a comment - Commit 1658240 from Mark Miller in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1658240 ] SOLR-6920 , SOLR-6640 : A replicated index can end up corrupted when small files end up with the same file name and size.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658519 from Mark Miller in branch 'dev/trunk'
          [ https://svn.apache.org/r1658519 ]

          SOLR-6920, SOLR-6640: Make constant and fix logging.

          Show
          ASF subversion and git services added a comment - Commit 1658519 from Mark Miller in branch 'dev/trunk' [ https://svn.apache.org/r1658519 ] SOLR-6920 , SOLR-6640 : Make constant and fix logging.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658524 from Mark Miller in branch 'dev/branches/branch_5x'
          [ https://svn.apache.org/r1658524 ]

          SOLR-6920, SOLR-6640: Make constant and fix logging.

          Show
          ASF subversion and git services added a comment - Commit 1658524 from Mark Miller in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1658524 ] SOLR-6920 , SOLR-6640 : Make constant and fix logging.
          Hide
          Mark Miller added a comment -

          Thanks Varun, great job.

          Show
          Mark Miller added a comment - Thanks Varun, great job.
          Hide
          ASF subversion and git services added a comment -

          Commit 1658526 from Mark Miller in branch 'dev/branches/lucene_solr_5_0'
          [ https://svn.apache.org/r1658526 ]

          SOLR-6920, SOLR-6640: Make constant and fix logging.

          Show
          ASF subversion and git services added a comment - Commit 1658526 from Mark Miller in branch 'dev/branches/lucene_solr_5_0' [ https://svn.apache.org/r1658526 ] SOLR-6920 , SOLR-6640 : Make constant and fix logging.
          Hide
          Anshum Gupta added a comment -

          Bulk close after 5.0 release.

          Show
          Anshum Gupta added a comment - Bulk close after 5.0 release.

            People

            • Assignee:
              Mark Miller
              Reporter:
              Varun Thacker
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development