[2019-11-11 12:11:17,167][ERROR][tcp-disco-msg-worker-[6f8dd6fc 127.0.0.1:47501 crd]-#195%cache.PutAllDeadlockTest0%][G] Blocked system-critical thread has been detected. This can lead to cluster-wide undefined behaviour [threadName=data-streamer-stripe-6, blockedFor=218s] [2019-11-11 12:11:17,167][WARN ][tcp-disco-msg-worker-[6f8dd6fc 127.0.0.1:47501 crd]-#195%cache.PutAllDeadlockTest0%][G] Thread [name="data-streamer-stripe-6-#1429%cache.PutAllDeadlockTest0%", id=1738, state=WAITING, blockCnt=0, waitCnt=352] Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@691e8389, ownerName=data-streamer-stripe-3-#1426%cache.PutAllDeadlockTest0%, ownerId=1735] [2019-11-11 12:11:17,167][ERROR][tcp-disco-msg-worker-[6f8dd6fc 127.0.0.1:47501 crd]-#195%cache.PutAllDeadlockTest0%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=SYSTEM_WORKER_BLOCKED, err=class o.a.i.IgniteException: GridWorker [name=data-streamer-stripe-6, igniteInstanceName=cache.PutAllDeadlockTest0, finished=false, heartbeatTs=1573463258755]]] class org.apache.ignite.IgniteException: GridWorker [name=data-streamer-stripe-6, igniteInstanceName=cache.PutAllDeadlockTest0, finished=false, heartbeatTs=1573463258755] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1803) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1798) at org.apache.ignite.internal.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) at org.apache.ignite.internal.util.worker.GridWorker.onIdle(GridWorker.java:297) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2840) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7731) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2928) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7669) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [2019-11-11 12:11:17,167][WARN ][tcp-disco-msg-worker-[6f8dd6fc 127.0.0.1:47501 crd]-#195%cache.PutAllDeadlockTest0%][FailureProcessor] Deadlocked threads detected (see thread dump below) [deadlockedThreadsCnt=3] [2019-11-11 12:11:17,210][WARN ][tcp-disco-msg-worker-[6f8dd6fc 127.0.0.1:47501 crd]-#195%cache.PutAllDeadlockTest0%][FailureProcessor] Thread dump at 2019/11/11 12:11:17 MSK Thread [name="sys-#1631%cache.PutAllDeadlockTest1%", id=1976, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20dc5765, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1630%cache.PutAllDeadlockTest2%", id=1975, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2bb68217, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1629%cache.PutAllDeadlockTest0%", id=1974, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d234bea, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1628%cache.PutAllDeadlockTest2%", id=1973, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2bb68217, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1627%cache.PutAllDeadlockTest0%", id=1972, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d234bea, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1626%cache.PutAllDeadlockTest0%", id=1971, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d234bea, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1625%cache.PutAllDeadlockTest1%", id=1970, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20dc5765, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1624%cache.PutAllDeadlockTest2%", id=1969, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2bb68217, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1623%cache.PutAllDeadlockTest1%", id=1968, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20dc5765, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1622%cache.PutAllDeadlockTest2%", id=1967, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2bb68217, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1621%cache.PutAllDeadlockTest0%", id=1966, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d234bea, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1620%cache.PutAllDeadlockTest0%", id=1965, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d234bea, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1619%cache.PutAllDeadlockTest1%", id=1964, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20dc5765, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1618%cache.PutAllDeadlockTest2%", id=1963, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2bb68217, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1617%cache.PutAllDeadlockTest1%", id=1962, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20dc5765, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1616%cache.PutAllDeadlockTest2%", id=1961, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2bb68217, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1615%cache.PutAllDeadlockTest0%", id=1960, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d234bea, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1614%cache.PutAllDeadlockTest0%", id=1959, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d234bea, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1613%cache.PutAllDeadlockTest1%", id=1958, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20dc5765, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-#1612%cache.PutAllDeadlockTest2%", id=1957, state=TIMED_WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2bb68217, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Thread [name="mgmt-#1579%cache.PutAllDeadlockTest0%", id=1924, state=WAITING, blockCnt=0, waitCnt=120] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) at o.a.i.i.processors.datastreamer.DataStreamerImpl.doFlush(DataStreamerImpl.java:1181) at o.a.i.i.processors.datastreamer.DataStreamerImpl.closeEx(DataStreamerImpl.java:1356) at o.a.i.i.processors.datastreamer.DataStreamerImpl.closeEx(DataStreamerImpl.java:1314) at o.a.i.i.processors.datastreamer.DataStreamerImpl.close(DataStreamerImpl.java:1302) at o.a.i.i.processors.datastreamer.DataStreamerImpl.close(DataStreamerImpl.java:1406) at o.a.i.i.processors.cache.distributed.GridDistributedCacheAdapter$GlobalRemoveAllJob.localExecute(GridDistributedCacheAdapter.java:478) at o.a.i.i.processors.cache.GridCacheAdapter$TopologyVersionAwareJob.execute(GridCacheAdapter.java:6780) at o.a.i.i.processors.job.GridJobWorker$2.call(GridJobWorker.java:569) at o.a.i.i.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6963) at o.a.i.i.processors.job.GridJobWorker.execute0(GridJobWorker.java:563) at o.a.i.i.processors.job.GridJobWorker.body(GridJobWorker.java:492) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.i.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1271) at o.a.i.i.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2089) at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843) at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468) at o.a.i.i.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1365) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Locked synchronizers: java.util.concurrent.ThreadPoolExecutor$Worker@21190fa8 Thread [name="mgmt-#1577%cache.PutAllDeadlockTest0%", id=1922, state=WAITING, blockCnt=0, waitCnt=135] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) at o.a.i.i.processors.datastreamer.DataStreamerImpl.doFlush(DataStreamerImpl.java:1181) at o.a.i.i.processors.datastreamer.DataStreamerImpl.closeEx(DataStreamerImpl.java:1356) at o.a.i.i.processors.datastreamer.DataStreamerImpl.closeEx(DataStreamerImpl.java:1314) at o.a.i.i.processors.datastreamer.DataStreamerImpl.close(DataStreamerImpl.java:1302) at o.a.i.i.processors.datastreamer.DataStreamerImpl.close(DataStreamerImpl.java:1406) at o.a.i.i.processors.cache.distributed.GridDistributedCacheAdapter$GlobalRemoveAllJob.localExecute(GridDistributedCacheAdapter.java:478) at o.a.i.i.processors.cache.GridCacheAdapter$TopologyVersionAwareJob.execute(GridCacheAdapter.java:6780) at o.a.i.i.processors.job.GridJobWorker$2.call(GridJobWorker.java:569) at o.a.i.i.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6963) at o.a.i.i.processors.job.GridJobWorker.execute0(GridJobWorker.java:563) at o.a.i.i.processors.job.GridJobWorker.body(GridJobWorker.java:492) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.i.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1271) at o.a.i.i.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:2089) at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843) at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468) at o.a.i.i.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1365) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Locked synchronizers: java.util.concurrent.ThreadPoolExecutor$Worker@22a1b8c2 Thread [name="cache.PutAllDeadlockTest-4", id=1918, state=WAITING, blockCnt=2, waitCnt=3139] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) at o.a.i.i.processors.cache.distributed.GridDistributedCacheAdapter.removeAll(GridDistributedCacheAdapter.java:189) at o.a.i.i.processors.cache.IgniteCacheProxyImpl.removeAll(IgniteCacheProxyImpl.java:1592) at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.removeAll(GatewayProtectedCacheProxy.java:1120) at o.a.i.cache.PutAllDeadlockTest.lambda$testHashSetRemoveAll$5(PutAllDeadlockTest.java:211) at o.a.i.cache.PutAllDeadlockTest$$Lambda$297/587486101.run(Unknown Source) at o.a.i.testframework.GridTestUtils$7.call(GridTestUtils.java:1416) at o.a.i.testframework.GridTestThread.run(GridTestThread.java:84) Thread [name="cache.PutAllDeadlockTest-3", id=1917, state=WAITING, blockCnt=2, waitCnt=3145] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:634) at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2575) at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2552) at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1302) at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:856) at o.a.i.cache.PutAllDeadlockTest.lambda$testHashSetRemoveAll$5(PutAllDeadlockTest.java:204) at o.a.i.cache.PutAllDeadlockTest$$Lambda$297/587486101.run(Unknown Source) at o.a.i.testframework.GridTestUtils$7.call(GridTestUtils.java:1416) at o.a.i.testframework.GridTestThread.run(GridTestThread.java:84) Thread [name="cache.PutAllDeadlockTest-2", id=1916, state=WAITING, blockCnt=0, waitCnt=3142] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.put0(GridDhtAtomicCache.java:634) at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2575) at o.a.i.i.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2552) at o.a.i.i.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1302) at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:856) at o.a.i.cache.PutAllDeadlockTest.lambda$testHashSetRemoveAll$5(PutAllDeadlockTest.java:204) at o.a.i.cache.PutAllDeadlockTest$$Lambda$297/587486101.run(Unknown Source) at o.a.i.testframework.GridTestUtils$7.call(GridTestUtils.java:1416) at o.a.i.testframework.GridTestThread.run(GridTestThread.java:84) Thread [name="cache.PutAllDeadlockTest-1", id=1915, state=WAITING, blockCnt=2, waitCnt=3137] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:178) at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141) at o.a.i.i.processors.cache.distributed.GridDistributedCacheAdapter.removeAll(GridDistributedCacheAdapter.java:189) at o.a.i.i.processors.cache.IgniteCacheProxyImpl.removeAll(IgniteCacheProxyImpl.java:1592) at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.removeAll(GatewayProtectedCacheProxy.java:1120) at o.a.i.cache.PutAllDeadlockTest.lambda$testHashSetRemoveAll$5(PutAllDeadlockTest.java:211) at o.a.i.cache.PutAllDeadlockTest$$Lambda$297/587486101.run(Unknown Source) at o.a.i.testframework.GridTestUtils$7.call(GridTestUtils.java:1416) at o.a.i.testframework.GridTestThread.run(GridTestThread.java:84) Thread [name="services-deployment-worker-#1573%cache.PutAllDeadlockTest2%", id=1913, state=WAITING, blockCnt=0, waitCnt=3] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4cd14f43, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at o.a.i.i.processors.service.ServiceDeploymentManager$ServicesDeploymentWorker.body(ServiceDeploymentManager.java:466) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-3-#1547%cache.PutAllDeadlockTest2%", id=1880, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@50f002d9 - locked java.util.Collections$UnmodifiableSet@6609806e - locked sun.nio.ch.EPollSelectorImpl@20e470bb at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-2-#1546%cache.PutAllDeadlockTest2%", id=1879, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@6a21cc6a - locked java.util.Collections$UnmodifiableSet@3a17ed3d - locked sun.nio.ch.EPollSelectorImpl@103700e1 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-1-#1545%cache.PutAllDeadlockTest2%", id=1878, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1355dbf4 - locked java.util.Collections$UnmodifiableSet@77731768 - locked sun.nio.ch.EPollSelectorImpl@720872be at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-0-#1544%cache.PutAllDeadlockTest2%", id=1877, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@5a105416 - locked java.util.Collections$UnmodifiableSet@67a7bedb - locked sun.nio.ch.EPollSelectorImpl@8003d10 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="nio-acceptor-client-listener-#1569%cache.PutAllDeadlockTest2%", id=1909, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@61c460fb - locked java.util.Collections$UnmodifiableSet@3b4f5b7e - locked sun.nio.ch.EPollSelectorImpl@322e19ba at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2974) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2922) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="ttl-cleanup-worker-#1555%cache.PutAllDeadlockTest2%", id=1895, state=TIMED_WAITING, blockCnt=0, waitCnt=438] at java.lang.Thread.sleep(Native Method) at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7796) at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:162) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="exchange-worker-#1552%cache.PutAllDeadlockTest2%", id=1892, state=TIMED_WAITING, blockCnt=0, waitCnt=16] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:219) at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160) at o.a.i.i.processors.cache.distributed.dht.preloader.latch.ExchangeLatchManager$CompletableLatch.await(ExchangeLatchManager.java:762) at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:1797) at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1464) at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:882) at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3162) at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3011) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="disco-event-worker-#1551%cache.PutAllDeadlockTest2%", id=1891, state=WAITING, blockCnt=0, waitCnt=437] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5ecc6118, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2807) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2777) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="tcp-disco-ip-finder-cleaner-#215%cache.PutAllDeadlockTest2%", id=1890, state=TIMED_WAITING, blockCnt=0, waitCnt=4] at java.lang.Thread.sleep(Native Method) at o.a.i.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:2111) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-disco-sock-reader-[da077aa8 127.0.0.1:59237]-#214%cache.PutAllDeadlockTest0%", id=1889, state=RUNNABLE, blockCnt=1, waitCnt=0] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked java.io.BufferedInputStream@62ce130d at o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53) at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2620) at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2636) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.(ObjectInputStream.java:349) at o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.(JdkMarshallerObjectInputStream.java:43) at o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:122) at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:93) at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:10180) at o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6738) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-disco-sock-reader-[6f8dd6fc 127.0.0.1:51673]-#213%cache.PutAllDeadlockTest2%", id=1888, state=RUNNABLE, blockCnt=0, waitCnt=0] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked java.io.BufferedInputStream@36273797 at o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53) at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2620) at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2636) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.(ObjectInputStream.java:349) at o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.(JdkMarshallerObjectInputStream.java:43) at o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:122) at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:93) at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:10180) at o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6738) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-disco-srvr-[:47502]-#210%cache.PutAllDeadlockTest2%", id=1885, state=RUNNABLE, blockCnt=0, waitCnt=0] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:6325) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:6248) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-disco-msg-worker-[2906500e 127.0.0.1:47500]-#209%cache.PutAllDeadlockTest2%", id=1884, state=RUNNABLE, blockCnt=0, waitCnt=21271] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7733) at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2928) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7669) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="disco-notifier-worker-#1550%cache.PutAllDeadlockTest2%", id=1883, state=WAITING, blockCnt=0, waitCnt=443] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@14e7cdf4, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2601) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2646) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-data-loader-flusher-#1549%cache.PutAllDeadlockTest2%", id=1882, state=WAITING, blockCnt=0, waitCnt=1] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@623e21e, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="Thread-85", id=1876, state=TIMED_WAITING, blockCnt=0, waitCnt=4] at java.lang.Thread.sleep(Native Method) at o.a.i.i.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$TimeOutWorker.run(SharedPageLockTracker.java:378) Thread [name="grid-sharedfs-timeout-worker-#208%cache.PutAllDeadlockTest2%", id=1875, state=TIMED_WAITING, blockCnt=0, waitCnt=44] Lock [object=java.lang.Object@937e6c2, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at o.a.i.spi.checkpoint.sharedfs.SharedFsTimeoutTask.body(SharedFsTimeoutTask.java:99) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-comm-worker-#207%cache.PutAllDeadlockTest2%", id=1874, state=TIMED_WAITING, blockCnt=0, waitCnt=2] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@551edfd7, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4589) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.spi.communication.tcp.TcpCommunicationSpi$6.body(TcpCommunicationSpi.java:2356) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="grid-nio-worker-tcp-comm-3-#1542%cache.PutAllDeadlockTest2%", id=1872, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7fbc30ed - locked java.util.Collections$UnmodifiableSet@66161e16 - locked sun.nio.ch.EPollSelectorImpl@50a1f5e0 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-tcp-comm-2-#1541%cache.PutAllDeadlockTest2%", id=1871, state=RUNNABLE, blockCnt=1, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@73e27844 - locked java.util.Collections$UnmodifiableSet@6d47058d - locked sun.nio.ch.EPollSelectorImpl@3bf7409d at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-tcp-comm-1-#1540%cache.PutAllDeadlockTest2%", id=1870, state=RUNNABLE, blockCnt=10, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@64a6bcf6 - locked java.util.Collections$UnmodifiableSet@466c87e - locked sun.nio.ch.EPollSelectorImpl@5158de12 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-tcp-comm-0-#1539%cache.PutAllDeadlockTest2%", id=1869, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@502a9f0c - locked java.util.Collections$UnmodifiableSet@13ad8f3 - locked sun.nio.ch.EPollSelectorImpl@33adc592 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="nio-acceptor-tcp-comm-#1543%cache.PutAllDeadlockTest2%", id=1873, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@241235bb - locked java.util.Collections$UnmodifiableSet@20c40cca - locked sun.nio.ch.EPollSelectorImpl@77eea561 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2974) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2922) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-timeout-worker-#1538%cache.PutAllDeadlockTest2%", id=1868, state=TIMED_WAITING, blockCnt=5, waitCnt=447] Lock [object=java.lang.Object@2ab27f5b, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="jvm-pause-detector-worker", id=1861, state=TIMED_WAITING, blockCnt=0, waitCnt=4238] at java.lang.Thread.sleep(Native Method) at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:117) Thread [name="data-streamer-stripe-7-#1531%cache.PutAllDeadlockTest2%", id=1860, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-6-#1530%cache.PutAllDeadlockTest2%", id=1859, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-5-#1529%cache.PutAllDeadlockTest2%", id=1858, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-4-#1528%cache.PutAllDeadlockTest2%", id=1857, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-3-#1527%cache.PutAllDeadlockTest2%", id=1856, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-2-#1526%cache.PutAllDeadlockTest2%", id=1855, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-1-#1525%cache.PutAllDeadlockTest2%", id=1854, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-0-#1524%cache.PutAllDeadlockTest2%", id=1853, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-7-#1523%cache.PutAllDeadlockTest2%", id=1852, state=WAITING, blockCnt=0, waitCnt=980] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-6-#1522%cache.PutAllDeadlockTest2%", id=1851, state=WAITING, blockCnt=0, waitCnt=1072] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-5-#1521%cache.PutAllDeadlockTest2%", id=1850, state=WAITING, blockCnt=0, waitCnt=980] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-4-#1520%cache.PutAllDeadlockTest2%", id=1849, state=WAITING, blockCnt=0, waitCnt=1008] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-3-#1519%cache.PutAllDeadlockTest2%", id=1848, state=WAITING, blockCnt=0, waitCnt=1139] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-2-#1518%cache.PutAllDeadlockTest2%", id=1847, state=WAITING, blockCnt=0, waitCnt=925] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-1-#1517%cache.PutAllDeadlockTest2%", id=1846, state=WAITING, blockCnt=0, waitCnt=1143] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-0-#1516%cache.PutAllDeadlockTest2%", id=1845, state=WAITING, blockCnt=0, waitCnt=872] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="services-deployment-worker-#1514%cache.PutAllDeadlockTest1%", id=1842, state=WAITING, blockCnt=0, waitCnt=4] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1dddbe05, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at o.a.i.i.processors.service.ServiceDeploymentManager$ServicesDeploymentWorker.body(ServiceDeploymentManager.java:466) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-3-#1494%cache.PutAllDeadlockTest1%", id=1815, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@24ac5242 - locked java.util.Collections$UnmodifiableSet@7a084c6c - locked sun.nio.ch.EPollSelectorImpl@33727582 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-2-#1493%cache.PutAllDeadlockTest1%", id=1814, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@43afd87f - locked java.util.Collections$UnmodifiableSet@5d7ac0a - locked sun.nio.ch.EPollSelectorImpl@3d84ebf0 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-1-#1492%cache.PutAllDeadlockTest1%", id=1813, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1d7a57a7 - locked java.util.Collections$UnmodifiableSet@202b1857 - locked sun.nio.ch.EPollSelectorImpl@182457a4 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-0-#1491%cache.PutAllDeadlockTest1%", id=1812, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@2e5f43b - locked java.util.Collections$UnmodifiableSet@4ded4caf - locked sun.nio.ch.EPollSelectorImpl@357c0ce5 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="nio-acceptor-client-listener-#1513%cache.PutAllDeadlockTest1%", id=1841, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@f5fe499 - locked java.util.Collections$UnmodifiableSet@22b69d65 - locked sun.nio.ch.EPollSelectorImpl@51925a8 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2974) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2922) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="ttl-cleanup-worker-#1502%cache.PutAllDeadlockTest1%", id=1830, state=TIMED_WAITING, blockCnt=0, waitCnt=438] at java.lang.Thread.sleep(Native Method) at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7796) at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:162) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="exchange-worker-#1499%cache.PutAllDeadlockTest1%", id=1827, state=TIMED_WAITING, blockCnt=0, waitCnt=21] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:219) at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160) at o.a.i.i.processors.cache.distributed.dht.preloader.latch.ExchangeLatchManager$CompletableLatch.await(ExchangeLatchManager.java:762) at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:1797) at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1464) at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:882) at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3162) at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3011) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="disco-event-worker-#1498%cache.PutAllDeadlockTest1%", id=1826, state=WAITING, blockCnt=1, waitCnt=437] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3617abae, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2807) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2777) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="tcp-disco-ip-finder-cleaner-#206%cache.PutAllDeadlockTest1%", id=1825, state=TIMED_WAITING, blockCnt=0, waitCnt=4] at java.lang.Thread.sleep(Native Method) at o.a.i.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:2111) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-disco-sock-reader-[2906500e 127.0.0.1:49241]-#204%cache.PutAllDeadlockTest1%", id=1823, state=RUNNABLE, blockCnt=1, waitCnt=0] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at java.io.BufferedInputStream.read(BufferedInputStream.java:345) - locked java.io.BufferedInputStream@7aeffca1 at o.a.i.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53) at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2620) at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2636) at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3113) at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:853) at java.io.ObjectInputStream.(ObjectInputStream.java:349) at o.a.i.marshaller.jdk.JdkMarshallerObjectInputStream.(JdkMarshallerObjectInputStream.java:43) at o.a.i.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:122) at o.a.i.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:93) at o.a.i.i.util.IgniteUtils.unmarshal(IgniteUtils.java:10180) at o.a.i.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:6738) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-disco-srvr-[:47501]-#201%cache.PutAllDeadlockTest1%", id=1820, state=RUNNABLE, blockCnt=0, waitCnt=0] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:6325) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:6248) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-disco-msg-worker-[da077aa8 127.0.0.1:47502]-#200%cache.PutAllDeadlockTest1%", id=1819, state=TIMED_WAITING, blockCnt=2, waitCnt=21288] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1286a87c, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522) at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684) at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7733) at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2928) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7669) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="disco-notifier-worker-#1497%cache.PutAllDeadlockTest1%", id=1818, state=WAITING, blockCnt=0, waitCnt=440] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@50a3a236, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2601) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2646) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-data-loader-flusher-#1496%cache.PutAllDeadlockTest1%", id=1817, state=WAITING, blockCnt=0, waitCnt=2] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5b622957, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="Thread-83", id=1811, state=TIMED_WAITING, blockCnt=0, waitCnt=4] at java.lang.Thread.sleep(Native Method) at o.a.i.i.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$TimeOutWorker.run(SharedPageLockTracker.java:378) Thread [name="grid-sharedfs-timeout-worker-#199%cache.PutAllDeadlockTest1%", id=1810, state=TIMED_WAITING, blockCnt=0, waitCnt=44] Lock [object=java.lang.Object@5eeddc6e, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at o.a.i.spi.checkpoint.sharedfs.SharedFsTimeoutTask.body(SharedFsTimeoutTask.java:99) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-comm-worker-#198%cache.PutAllDeadlockTest1%", id=1809, state=TIMED_WAITING, blockCnt=0, waitCnt=2] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@798f4a87, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4589) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.spi.communication.tcp.TcpCommunicationSpi$6.body(TcpCommunicationSpi.java:2356) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="grid-nio-worker-tcp-comm-3-#1489%cache.PutAllDeadlockTest1%", id=1807, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@34f0e592 - locked java.util.Collections$UnmodifiableSet@7e924aca - locked sun.nio.ch.EPollSelectorImpl@4e2cf7da at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-tcp-comm-2-#1488%cache.PutAllDeadlockTest1%", id=1806, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@4df457ea - locked java.util.Collections$UnmodifiableSet@7e1bf7b3 - locked sun.nio.ch.EPollSelectorImpl@12b326a0 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-tcp-comm-1-#1487%cache.PutAllDeadlockTest1%", id=1805, state=RUNNABLE, blockCnt=2, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@3b08910b - locked java.util.Collections$UnmodifiableSet@54acda87 - locked sun.nio.ch.EPollSelectorImpl@75ab09ce at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-tcp-comm-0-#1486%cache.PutAllDeadlockTest1%", id=1804, state=RUNNABLE, blockCnt=24, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7ea6a7fd - locked java.util.Collections$UnmodifiableSet@159f1eb - locked sun.nio.ch.EPollSelectorImpl@33937ba5 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="nio-acceptor-tcp-comm-#1490%cache.PutAllDeadlockTest1%", id=1808, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@1e5be160 - locked java.util.Collections$UnmodifiableSet@55093419 - locked sun.nio.ch.EPollSelectorImpl@1b9151fa at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2974) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2922) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-timeout-worker-#1485%cache.PutAllDeadlockTest1%", id=1803, state=TIMED_WAITING, blockCnt=4, waitCnt=367] Lock [object=java.lang.Object@33775f7e, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="jvm-pause-detector-worker", id=1796, state=TIMED_WAITING, blockCnt=0, waitCnt=4240] at java.lang.Thread.sleep(Native Method) at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:117) Thread [name="data-streamer-stripe-7-#1478%cache.PutAllDeadlockTest1%", id=1795, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-6-#1477%cache.PutAllDeadlockTest1%", id=1794, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-5-#1476%cache.PutAllDeadlockTest1%", id=1793, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-4-#1475%cache.PutAllDeadlockTest1%", id=1792, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-3-#1474%cache.PutAllDeadlockTest1%", id=1791, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-2-#1473%cache.PutAllDeadlockTest1%", id=1790, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-1-#1472%cache.PutAllDeadlockTest1%", id=1789, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-0-#1471%cache.PutAllDeadlockTest1%", id=1788, state=WAITING, blockCnt=0, waitCnt=2] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-7-#1470%cache.PutAllDeadlockTest1%", id=1787, state=WAITING, blockCnt=0, waitCnt=1690] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-6-#1469%cache.PutAllDeadlockTest1%", id=1786, state=WAITING, blockCnt=0, waitCnt=1656] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-5-#1468%cache.PutAllDeadlockTest1%", id=1785, state=WAITING, blockCnt=0, waitCnt=1614] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-4-#1467%cache.PutAllDeadlockTest1%", id=1784, state=WAITING, blockCnt=0, waitCnt=1629] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-3-#1466%cache.PutAllDeadlockTest1%", id=1783, state=WAITING, blockCnt=0, waitCnt=1619] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-2-#1465%cache.PutAllDeadlockTest1%", id=1782, state=WAITING, blockCnt=0, waitCnt=1680] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-1-#1464%cache.PutAllDeadlockTest1%", id=1781, state=WAITING, blockCnt=0, waitCnt=1561] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-0-#1463%cache.PutAllDeadlockTest1%", id=1780, state=WAITING, blockCnt=0, waitCnt=1585] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="services-deployment-worker-#1462%cache.PutAllDeadlockTest0%", id=1778, state=WAITING, blockCnt=0, waitCnt=5] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6ed3a328, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at o.a.i.i.processors.service.ServiceDeploymentManager$ServicesDeploymentWorker.body(ServiceDeploymentManager.java:466) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-3-#1446%cache.PutAllDeadlockTest0%", id=1759, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@7fdc3189 - locked java.util.Collections$UnmodifiableSet@76d91da3 - locked sun.nio.ch.EPollSelectorImpl@33bec189 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-2-#1445%cache.PutAllDeadlockTest0%", id=1758, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@4087ee35 - locked java.util.Collections$UnmodifiableSet@5e49dfd5 - locked sun.nio.ch.EPollSelectorImpl@6043e5ba at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-1-#1444%cache.PutAllDeadlockTest0%", id=1757, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@2534d2b2 - locked java.util.Collections$UnmodifiableSet@5a3d7667 - locked sun.nio.ch.EPollSelectorImpl@4a9abd82 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-client-listener-0-#1443%cache.PutAllDeadlockTest0%", id=1756, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@358bef5e - locked java.util.Collections$UnmodifiableSet@36290a1f - locked sun.nio.ch.EPollSelectorImpl@716ccf3 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="nio-acceptor-client-listener-#1461%cache.PutAllDeadlockTest0%", id=1777, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@18ff073e - locked java.util.Collections$UnmodifiableSet@7264ea60 - locked sun.nio.ch.EPollSelectorImpl@c636fc1 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2974) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2922) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) ##### DEADLOCKED Thread [name="sys-#1458%cache.PutAllDeadlockTest0%", id=1774, state=WAITING, blockCnt=0, waitCnt=9] Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@49c66c58, ownerName=data-streamer-stripe-6-#1429%cache.PutAllDeadlockTest0%, ownerId=1738] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:5025) at o.a.i.i.processors.cache.GridCacheMapEntry.markObsoleteVersion(GridCacheMapEntry.java:2799) at o.a.i.i.processors.cache.distributed.dht.topology.GridDhtLocalPartition.removeVersionedEntry(GridDhtLocalPartition.java:388) at o.a.i.i.processors.cache.distributed.dht.topology.GridDhtLocalPartition.cleanupRemoveQueue(GridDhtLocalPartition.java:412) at o.a.i.i.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1.run(GridCacheProcessor.java:5595) at o.a.i.i.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:7019) at o.a.i.i.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Locked synchronizers: java.util.concurrent.ThreadPoolExecutor$Worker@15a61a18 Thread [name="ttl-cleanup-worker-#1454%cache.PutAllDeadlockTest0%", id=1770, state=TIMED_WAITING, blockCnt=0, waitCnt=438] at java.lang.Thread.sleep(Native Method) at o.a.i.i.util.IgniteUtils.sleep(IgniteUtils.java:7796) at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:162) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="exchange-worker-#1451%cache.PutAllDeadlockTest0%", id=1767, state=TIMED_WAITING, blockCnt=0, waitCnt=21] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338) at o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:219) at o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:160) at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:1688) at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1464) at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:882) at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3162) at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3011) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="disco-event-worker-#1450%cache.PutAllDeadlockTest0%", id=1766, state=WAITING, blockCnt=0, waitCnt=222] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d057d87, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:2807) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:2777) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="tcp-disco-ip-finder-cleaner-#197%cache.PutAllDeadlockTest0%", id=1765, state=TIMED_WAITING, blockCnt=0, waitCnt=4] at java.lang.Thread.sleep(Native Method) at o.a.i.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:2111) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-disco-srvr-[:47500]-#196%cache.PutAllDeadlockTest0%", id=1764, state=RUNNABLE, blockCnt=0, waitCnt=0] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at o.a.i.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:6325) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.spi.discovery.tcp.ServerImpl$TcpServerThread.body(ServerImpl.java:6248) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-disco-msg-worker-[6f8dd6fc 127.0.0.1:47501 crd]-#195%cache.PutAllDeadlockTest0%", id=1763, state=RUNNABLE, blockCnt=1, waitCnt=21232] at sun.management.ThreadImpl.dumpThreads0(Native Method) at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454) at o.a.i.i.util.IgniteUtils.dumpThreads(IgniteUtils.java:1426) at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:142) - locked o.a.i.i.processors.failure.FailureProcessor@867bbad at o.a.i.i.processors.failure.FailureProcessor.process(FailureProcessor.java:111) at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1801) at o.a.i.i.IgnitionEx$IgniteNamedInstance$2.apply(IgnitionEx.java:1798) at o.a.i.i.worker.WorkersRegistry.onIdle(WorkersRegistry.java:233) at o.a.i.i.util.worker.GridWorker.onIdle(GridWorker.java:297) at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.lambda$new$0(ServerImpl.java:2840) at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker$$Lambda$92/1209874402.run(Unknown Source) at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7731) at o.a.i.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2928) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7669) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="disco-notifier-worker-#1449%cache.PutAllDeadlockTest0%", id=1762, state=WAITING, blockCnt=0, waitCnt=229] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1f6dd007, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2601) at o.a.i.i.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2646) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-data-loader-flusher-#1448%cache.PutAllDeadlockTest0%", id=1761, state=WAITING, blockCnt=0, waitCnt=2] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3836a41b, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.DelayQueue.take(DelayQueue.java:211) at o.a.i.i.processors.datastreamer.DataStreamProcessor$2.body(DataStreamProcessor.java:109) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="Thread-81", id=1755, state=TIMED_WAITING, blockCnt=0, waitCnt=4] at java.lang.Thread.sleep(Native Method) at o.a.i.i.processors.cache.persistence.diagnostic.pagelocktracker.SharedPageLockTracker$TimeOutWorker.run(SharedPageLockTracker.java:378) Thread [name="grid-sharedfs-timeout-worker-#194%cache.PutAllDeadlockTest0%", id=1754, state=TIMED_WAITING, blockCnt=0, waitCnt=44] Lock [object=java.lang.Object@4e291fdf, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at o.a.i.spi.checkpoint.sharedfs.SharedFsTimeoutTask.body(SharedFsTimeoutTask.java:99) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="tcp-comm-worker-#193%cache.PutAllDeadlockTest0%", id=1753, state=TIMED_WAITING, blockCnt=0, waitCnt=2] Lock [object=java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b267f64, ownerName=null, ownerId=-1] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at o.a.i.spi.communication.tcp.TcpCommunicationSpi$CommunicationWorker.body(TcpCommunicationSpi.java:4589) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.spi.communication.tcp.TcpCommunicationSpi$6.body(TcpCommunicationSpi.java:2356) at o.a.i.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) Thread [name="grid-nio-worker-tcp-comm-3-#1441%cache.PutAllDeadlockTest0%", id=1751, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@a3a65ef - locked java.util.Collections$UnmodifiableSet@7210ecf - locked sun.nio.ch.EPollSelectorImpl@1f03e13 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-tcp-comm-2-#1440%cache.PutAllDeadlockTest0%", id=1750, state=RUNNABLE, blockCnt=1, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@49e5bc16 - locked java.util.Collections$UnmodifiableSet@1029add8 - locked sun.nio.ch.EPollSelectorImpl@1f718e64 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-tcp-comm-1-#1439%cache.PutAllDeadlockTest0%", id=1749, state=RUNNABLE, blockCnt=28, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@1ff12b3f - locked java.util.Collections$UnmodifiableSet@7b1031db - locked sun.nio.ch.EPollSelectorImpl@260a140c at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-nio-worker-tcp-comm-0-#1438%cache.PutAllDeadlockTest0%", id=1748, state=RUNNABLE, blockCnt=33, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked o.a.i.i.util.nio.SelectedSelectionKeySet@359bbc07 - locked java.util.Collections$UnmodifiableSet@2325d158 - locked sun.nio.ch.EPollSelectorImpl@5941b20e at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2169) at o.a.i.i.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1815) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="nio-acceptor-tcp-comm-#1442%cache.PutAllDeadlockTest0%", id=1752, state=RUNNABLE, blockCnt=0, waitCnt=0] at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked sun.nio.ch.Util$3@2f9d675e - locked java.util.Collections$UnmodifiableSet@478e9d2e - locked sun.nio.ch.EPollSelectorImpl@4aac2b96 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.accept(GridNioServer.java:2974) at o.a.i.i.util.nio.GridNioServer$GridNioAcceptWorker.body(GridNioServer.java:2922) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="grid-timeout-worker-#1437%cache.PutAllDeadlockTest0%", id=1747, state=TIMED_WAITING, blockCnt=1, waitCnt=346] Lock [object=java.lang.Object@1e29210f, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at o.a.i.i.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:269) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="jvm-pause-detector-worker", id=1740, state=TIMED_WAITING, blockCnt=0, waitCnt=4241] at java.lang.Thread.sleep(Native Method) at o.a.i.i.LongJVMPauseDetector$1.run(LongJVMPauseDetector.java:117) Thread [name="data-streamer-stripe-7-#1430%cache.PutAllDeadlockTest0%", id=1739, state=WAITING, blockCnt=0, waitCnt=321] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) ##### DEADLOCKED Thread [name="data-streamer-stripe-6-#1429%cache.PutAllDeadlockTest0%", id=1738, state=WAITING, blockCnt=0, waitCnt=352] Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@691e8389, ownerName=data-streamer-stripe-3-#1426%cache.PutAllDeadlockTest0%, ownerId=1735] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:5025) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.lockEntries(GridDhtAtomicCache.java:3104) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1823) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1731) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:300) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map(GridNearAtomicUpdateFuture.java:814) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapOnTopology(GridNearAtomicUpdateFuture.java:666) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:249) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.removeAllAsync0(GridDhtAtomicCache.java:1406) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.removeAll0(GridDhtAtomicCache.java:731) at o.a.i.i.processors.cache.GridCacheAdapter.removeAll(GridCacheAdapter.java:3258) at o.a.i.i.processors.cache.IgniteCacheProxyImpl.removeAll(IgniteCacheProxyImpl.java:1570) at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.removeAll(GatewayProtectedCacheProxy.java:1096) at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters.updateAll(DataStreamerCacheUpdaters.java:93) at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Batched.receive(DataStreamerCacheUpdaters.java:162) at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140) at o.a.i.i.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6963) at o.a.i.i.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:550) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Locked synchronizers: java.util.concurrent.locks.ReentrantLock$NonfairSync@49c66c58 Thread [name="data-streamer-stripe-5-#1428%cache.PutAllDeadlockTest0%", id=1737, state=WAITING, blockCnt=0, waitCnt=404] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-4-#1427%cache.PutAllDeadlockTest0%", id=1736, state=WAITING, blockCnt=0, waitCnt=366] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) ##### DEADLOCKED Thread [name="data-streamer-stripe-3-#1426%cache.PutAllDeadlockTest0%", id=1735, state=WAITING, blockCnt=0, waitCnt=406] Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@49c66c58, ownerName=data-streamer-stripe-6-#1429%cache.PutAllDeadlockTest0%, ownerId=1738] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:5025) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.lockEntries(GridDhtAtomicCache.java:3104) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1823) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1731) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.sendSingleRequest(GridNearAtomicAbstractUpdateFuture.java:300) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.map(GridNearAtomicUpdateFuture.java:814) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.mapOnTopology(GridNearAtomicUpdateFuture.java:666) at o.a.i.i.processors.cache.distributed.dht.atomic.GridNearAtomicAbstractUpdateFuture.map(GridNearAtomicAbstractUpdateFuture.java:249) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.removeAllAsync0(GridDhtAtomicCache.java:1406) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.removeAll0(GridDhtAtomicCache.java:731) at o.a.i.i.processors.cache.GridCacheAdapter.removeAll(GridCacheAdapter.java:3258) at o.a.i.i.processors.cache.IgniteCacheProxyImpl.removeAll(IgniteCacheProxyImpl.java:1570) at o.a.i.i.processors.cache.GatewayProtectedCacheProxy.removeAll(GatewayProtectedCacheProxy.java:1096) at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters.updateAll(DataStreamerCacheUpdaters.java:93) at o.a.i.i.processors.datastreamer.DataStreamerCacheUpdaters$Batched.receive(DataStreamerCacheUpdaters.java:162) at o.a.i.i.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140) at o.a.i.i.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6963) at o.a.i.i.processors.closure.GridClosureProcessor$2.body(GridClosureProcessor.java:967) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:550) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Locked synchronizers: java.util.concurrent.locks.ReentrantLock$NonfairSync@691e8389 java.util.concurrent.locks.ReentrantLock$NonfairSync@6ff9a89d java.util.concurrent.locks.ReentrantLock$NonfairSync@15b385b9 java.util.concurrent.locks.ReentrantLock$NonfairSync@27c0da8 java.util.concurrent.locks.ReentrantLock$NonfairSync@4245e5a7 java.util.concurrent.locks.ReentrantLock$NonfairSync@1fe3c716 Thread [name="data-streamer-stripe-2-#1425%cache.PutAllDeadlockTest0%", id=1734, state=WAITING, blockCnt=0, waitCnt=375] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-1-#1424%cache.PutAllDeadlockTest0%", id=1733, state=WAITING, blockCnt=0, waitCnt=335] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="data-streamer-stripe-0-#1423%cache.PutAllDeadlockTest0%", id=1732, state=WAITING, blockCnt=0, waitCnt=414] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-7-#1422%cache.PutAllDeadlockTest0%", id=1731, state=WAITING, blockCnt=0, waitCnt=1109] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-6-#1421%cache.PutAllDeadlockTest0%", id=1730, state=WAITING, blockCnt=0, waitCnt=1131] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-5-#1420%cache.PutAllDeadlockTest0%", id=1729, state=WAITING, blockCnt=0, waitCnt=1044] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-4-#1419%cache.PutAllDeadlockTest0%", id=1728, state=WAITING, blockCnt=0, waitCnt=1072] Lock [object=java.util.concurrent.locks.ReentrantLock$NonfairSync@49c66c58, ownerName=data-streamer-stripe-6-#1429%cache.PutAllDeadlockTest0%, ownerId=1738] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209) at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285) at o.a.i.i.processors.cache.GridCacheMapEntry.lockEntry(GridCacheMapEntry.java:5025) at o.a.i.i.processors.cache.GridCacheMapEntry.obsolete(GridCacheMapEntry.java:2883) at o.a.i.i.processors.cache.GridCacheConcurrentMapImpl.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMapImpl.java:125) at o.a.i.i.processors.cache.GridCacheConcurrentMapImpl.putEntryIfObsoleteOrAbsent(GridCacheConcurrentMapImpl.java:70) at o.a.i.i.processors.cache.distributed.dht.GridCachePartitionedConcurrentMap.putEntryIfObsoleteOrAbsent(GridCachePartitionedConcurrentMap.java:94) at o.a.i.i.processors.cache.GridCacheAdapter.entryEx(GridCacheAdapter.java:1049) at o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.entryEx(GridDhtCacheAdapter.java:545) at o.a.i.i.processors.cache.distributed.dht.GridDhtCacheAdapter.entryExx(GridDhtCacheAdapter.java:565) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.lockEntries(GridDhtAtomicCache.java:3076) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0(GridDhtAtomicCache.java:1823) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal(GridDhtAtomicCache.java:1731) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdateRequest(GridDhtAtomicCache.java:3312) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$400(GridDhtAtomicCache.java:142) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:276) at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCache.java:271) at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142) at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591) at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392) at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318) at o.a.i.i.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109) at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308) at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843) at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468) at o.a.i.i.managers.communication.GridIoManager.access$5200(GridIoManager.java:229) at o.a.i.i.managers.communication.GridIoManager$9.run(GridIoManager.java:1365) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:550) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-3-#1418%cache.PutAllDeadlockTest0%", id=1727, state=WAITING, blockCnt=0, waitCnt=1077] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-2-#1417%cache.PutAllDeadlockTest0%", id=1726, state=WAITING, blockCnt=0, waitCnt=1092] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-1-#1416%cache.PutAllDeadlockTest0%", id=1725, state=WAITING, blockCnt=0, waitCnt=1050] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="sys-stripe-0-#1415%cache.PutAllDeadlockTest0%", id=1724, state=WAITING, blockCnt=0, waitCnt=1074] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304) at o.a.i.i.util.StripedExecutor$StripeConcurrentQueue.take(StripedExecutor.java:713) at o.a.i.i.util.StripedExecutor$Stripe.body(StripedExecutor.java:536) at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:120) at java.lang.Thread.run(Thread.java:748) Thread [name="test-runner-#1414%cache.PutAllDeadlockTest%", id=1723, state=WAITING, blockCnt=2, waitCnt=12] Lock [object=o.a.i.testframework.GridTestSafeThreadFactory$1@66ee7696, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1252) at java.lang.Thread.join(Thread.java:1326) at o.a.i.testframework.GridTestUtils.runMultiThreaded(GridTestUtils.java:1005) at o.a.i.testframework.GridTestUtils.runMultiThreaded(GridTestUtils.java:970) at o.a.i.testframework.GridTestUtils.runMultiThreaded(GridTestUtils.java:888) at o.a.i.testframework.GridTestUtils.runMultiThreaded(GridTestUtils.java:861) at o.a.i.testframework.junits.GridAbstractTest.multithreaded(GridAbstractTest.java:508) at o.a.i.testframework.junits.GridAbstractTest.multithreaded(GridAbstractTest.java:494) at o.a.i.cache.PutAllDeadlockTest.testHashSetRemoveAll(PutAllDeadlockTest.java:196) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at o.a.i.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2090) at java.lang.Thread.run(Thread.java:748) Thread [name="Attach Listener", id=38, state=RUNNABLE, blockCnt=0, waitCnt=0] Thread [name="ignite-clock-for-tests", id=11, state=TIMED_WAITING, blockCnt=0, waitCnt=27560] at java.lang.Thread.sleep(Native Method) at o.a.i.i.util.GridTestClockTimer.run(GridTestClockTimer.java:50) at java.lang.Thread.run(Thread.java:748) Thread [name="Monitor Ctrl-Break", id=5, state=RUNNABLE, blockCnt=0, waitCnt=0] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) at java.net.SocketInputStream.read(SocketInputStream.java:171) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178) - locked java.io.InputStreamReader@6734c62a at java.io.InputStreamReader.read(InputStreamReader.java:184) at java.io.BufferedReader.fill(BufferedReader.java:161) at java.io.BufferedReader.readLine(BufferedReader.java:324) - locked java.io.InputStreamReader@6734c62a at java.io.BufferedReader.readLine(BufferedReader.java:389) at com.intellij.rt.execution.application.AppMainV2$1.run(AppMainV2.java:64) Thread [name="Signal Dispatcher", id=4, state=RUNNABLE, blockCnt=0, waitCnt=0] Thread [name="Finalizer", id=3, state=WAITING, blockCnt=36, waitCnt=26] Lock [object=java.lang.ref.ReferenceQueue$Lock@5478629, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209) Thread [name="Reference Handler", id=2, state=WAITING, blockCnt=29, waitCnt=28] Lock [object=java.lang.ref.Reference$Lock@27586b5c, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at java.lang.ref.Reference.tryHandlePending(Reference.java:191) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) Thread [name="main", id=1, state=TIMED_WAITING, blockCnt=548, waitCnt=735] Lock [object=o.a.i.thread.IgniteThread@61eb55c0, ownerName=null, ownerId=-1] at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1260) at o.a.i.testframework.junits.GridAbstractTest.runTest(GridAbstractTest.java:2102) at o.a.i.testframework.junits.GridAbstractTest.access$600(GridAbstractTest.java:164) at o.a.i.testframework.junits.GridAbstractTest$2.evaluate(GridAbstractTest.java:199) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at o.a.i.testframework.junits.SystemPropertiesRule.lambda$methodStatement$1(SystemPropertiesRule.java:110) at o.a.i.testframework.junits.SystemPropertiesRule$$Lambda$3/1935637221.evaluate(Unknown Source) at o.a.i.testframework.junits.DelegatingJUnitStatement.evaluate(DelegatingJUnitStatement.java:49) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at o.a.i.testframework.junits.GridAbstractTest.evaluateInsideFixture(GridAbstractTest.java:2628) at o.a.i.testframework.junits.GridAbstractTest.access$500(GridAbstractTest.java:164) at o.a.i.testframework.junits.GridAbstractTest$BeforeFirstAndAfterLastTestRule$1.evaluate(GridAbstractTest.java:2608) at o.a.i.testframework.junits.SystemPropertiesRule.lambda$classStatement$0(SystemPropertiesRule.java:94) at o.a.i.testframework.junits.SystemPropertiesRule$$Lambda$1/1348949648.evaluate(Unknown Source) at o.a.i.testframework.junits.DelegatingJUnitStatement.evaluate(DelegatingJUnitStatement.java:49) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runner.JUnitCore.run(JUnitCore.java:160) at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68) at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:67) at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242) at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70) Locked synchronizers: java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@23ff86f3 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4bf4cfbb java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@67c03e21 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4fe2591b java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4451cd1c java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@64cb023e java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@29b55b3a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@22c42fac java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@30c653fc java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@70a046c0 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@4c466018 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2e381a96 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@42229562 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@3d1b6b9e java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@3ea18ade java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@3c7a7745 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@6a257c2d java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@c04bd6c java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@3e412148 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@516eb99a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@106b738e java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@392d4332 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@e517e3e java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@2a57a407 [2019-11-11 12:11:17,213][WARN ][tcp-disco-msg-worker-[6f8dd6fc 127.0.0.1:47501 crd]-#195%cache.PutAllDeadlockTest0%][CacheDiagnosticManager] Page locks dump: Thread=[name=data-streamer-stripe-0-#1423%cache.PutAllDeadlockTest0%, id=1732], state=WAITING Locked pages = [] Locked pages log: name=data-streamer-stripe-0-#1423%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=data-streamer-stripe-1-#1424%cache.PutAllDeadlockTest0%, id=1733], state=WAITING Locked pages = [] Locked pages log: name=data-streamer-stripe-1-#1424%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=data-streamer-stripe-2-#1425%cache.PutAllDeadlockTest0%, id=1734], state=WAITING Locked pages = [] Locked pages log: name=data-streamer-stripe-2-#1425%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=data-streamer-stripe-3-#1426%cache.PutAllDeadlockTest0%, id=1735], state=WAITING Locked pages = [] Locked pages log: name=data-streamer-stripe-3-#1426%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=data-streamer-stripe-4-#1427%cache.PutAllDeadlockTest0%, id=1736], state=WAITING Locked pages = [] Locked pages log: name=data-streamer-stripe-4-#1427%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=data-streamer-stripe-5-#1428%cache.PutAllDeadlockTest0%, id=1737], state=WAITING Locked pages = [] Locked pages log: name=data-streamer-stripe-5-#1428%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=data-streamer-stripe-6-#1429%cache.PutAllDeadlockTest0%, id=1738], state=WAITING Locked pages = [] Locked pages log: name=data-streamer-stripe-6-#1429%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=data-streamer-stripe-7-#1430%cache.PutAllDeadlockTest0%, id=1739], state=WAITING Locked pages = [] Locked pages log: name=data-streamer-stripe-7-#1430%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=exchange-worker-#1451%cache.PutAllDeadlockTest0%, id=1767], state=TIMED_WAITING Locked pages = [] Locked pages log: name=exchange-worker-#1451%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=mgmt-#1577%cache.PutAllDeadlockTest0%, id=1922], state=WAITING Locked pages = [] Locked pages log: name=mgmt-#1577%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=mgmt-#1579%cache.PutAllDeadlockTest0%, id=1924], state=WAITING Locked pages = [] Locked pages log: name=mgmt-#1579%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=sys-stripe-0-#1415%cache.PutAllDeadlockTest0%, id=1724], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-0-#1415%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=sys-stripe-1-#1416%cache.PutAllDeadlockTest0%, id=1725], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-1-#1416%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=sys-stripe-2-#1417%cache.PutAllDeadlockTest0%, id=1726], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-2-#1417%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=sys-stripe-3-#1418%cache.PutAllDeadlockTest0%, id=1727], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-3-#1418%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=sys-stripe-4-#1419%cache.PutAllDeadlockTest0%, id=1728], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-4-#1419%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=sys-stripe-5-#1420%cache.PutAllDeadlockTest0%, id=1729], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-5-#1420%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=sys-stripe-6-#1421%cache.PutAllDeadlockTest0%, id=1730], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-6-#1421%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) Thread=[name=sys-stripe-7-#1422%cache.PutAllDeadlockTest0%, id=1731], state=WAITING Locked pages = [] Locked pages log: name=sys-stripe-7-#1422%cache.PutAllDeadlockTest0% time=(1573463477213, 2019-11-11 12:11:17.213) [2019-11-11 12:11:17,638][INFO ][Thread-86][G] Invoking shutdown hook... [2019-11-11 12:11:17,638][INFO ][Thread-82][G] Invoking shutdown hook... [2019-11-11 12:11:17,638][INFO ][Thread-84][G] Invoking shutdown hook... [2019-11-11 12:11:17,639][WARN ][Thread-84][GridTaskProcessor] Will cancel unfinished tasks due to stopping of the grid [cnt=2] Failure in thread: Thread [id=1918, name=cache.PutAllDeadlockTest-4] javax.cache.CacheException: class org.apache.ignite.compute.ComputeTaskCancelledException: Task cancelled due to stopping of the grid: GridTaskWorker [internal=true, marsh=BinaryMarshaller [], ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], fut=ComputeTaskInternalFuture [ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1492807644]], state=WAITING, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskCls=class org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, task=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask@49694fa9, continuous=false, lockRespProc=false, resCache=true, noFailover=false, affPartId=-1, affCacheName=null, mapTopVer=null, retryAttemptCnt=0, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapper=org.apache.ignite.internal.processors.task.GridTaskWorker$1@52d04fba] at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2069) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.removeAll(IgniteCacheProxyImpl.java:1595) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.removeAll(GatewayProtectedCacheProxy.java:1120) at org.apache.ignite.cache.PutAllDeadlockTest.lambda$testHashSetRemoveAll$5(PutAllDeadlockTest.java:211) at org.apache.ignite.testframework.GridTestUtils$7.call(GridTestUtils.java:1416) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: class org.apache.ignite.compute.ComputeTaskCancelledException: Task cancelled due to stopping of the grid: GridTaskWorker [internal=true, marsh=BinaryMarshaller [], ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], fut=ComputeTaskInternalFuture [ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1492807644]], state=WAITING, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskCls=class org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, task=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask@49694fa9, continuous=false, lockRespProc=false, resCache=true, noFailover=false, affPartId=-1, affCacheName=null, mapTopVer=null, retryAttemptCnt=0, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapper=org.apache.ignite.internal.processors.task.GridTaskWorker$1@52d04fba] at org.apache.ignite.internal.util.IgniteUtils$10.apply(IgniteUtils.java:934) at org.apache.ignite.internal.util.IgniteUtils$10.apply(IgniteUtils.java:932) ... 7 more Caused by: class org.apache.ignite.internal.compute.ComputeTaskCancelledCheckedException: Task cancelled due to stopping of the grid: GridTaskWorker [internal=true, marsh=BinaryMarshaller [], ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], fut=ComputeTaskInternalFuture [ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1492807644]], state=WAITING, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskCls=class org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, task=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask@49694fa9, continuous=false, lockRespProc=false, resCache=true, noFailover=false, affPartId=-1, affCacheName=null, mapTopVer=null, retryAttemptCnt=0, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapper=org.apache.ignite.internal.processors.task.GridTaskWorker$1@52d04fba] at org.apache.ignite.internal.processors.task.GridTaskProcessor.onKernalStop(GridTaskProcessor.java:276) at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2550) at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2498) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2643) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2606) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$6.run(IgnitionEx.java:2103) Failure in thread: Thread [id=1917, name=cache.PutAllDeadlockTest-3] javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for future to complete. at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2069) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1305) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:856) at org.apache.ignite.cache.PutAllDeadlockTest.lambda$testHashSetRemoveAll$5(PutAllDeadlockTest.java:204) at org.apache.ignite.testframework.GridTestUtils$7.call(GridTestUtils.java:1416) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: class org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for future to complete. at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:885) at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:883) ... 7 more Failure in thread: Thread [id=1916, name=cache.PutAllDeadlockTest-2] Failure in thread: Thread [id=1915, name=cache.PutAllDeadlockTest-1] javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for future to complete. at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2069) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1305) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:856) at org.apache.ignite.cache.PutAllDeadlockTest.lambda$testHashSetRemoveAll$5(PutAllDeadlockTest.java:204) at org.apache.ignite.testframework.GridTestUtils$7.call(GridTestUtils.java:1416) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: class org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for future to complete. at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:885) at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:883) ... 7 more javax.cache.CacheException: class org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for future to complete. [2019-11-11 12:11:17,646][INFO ][main][root] >>> Stopping test: PutAllDeadlockTest#testHashSetRemoveAll in 219933 ms <<< [2019-11-11 12:11:17,647][INFO ][main][root] >>> Stopping grid [name=cache.PutAllDeadlockTest2, id=da077aa8-e739-4a05-a68e-fa2c8f600002] at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2069) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.removeAll(IgniteCacheProxyImpl.java:1595) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.removeAll(GatewayProtectedCacheProxy.java:1120) at org.apache.ignite.cache.PutAllDeadlockTest.lambda$testHashSetRemoveAll$5(PutAllDeadlockTest.java:211) at org.apache.ignite.testframework.GridTestUtils$7.call(GridTestUtils.java:1416) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: class org.apache.ignite.IgniteInterruptedException: Got interrupted while waiting for future to complete. at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:885) at org.apache.ignite.internal.util.IgniteUtils$3.apply(IgniteUtils.java:883) ... 7 more [2019-11-11 12:11:17,646][ERROR][main][root] Test failed. javax.cache.CacheException: class org.apache.ignite.compute.ComputeTaskCancelledException: Task cancelled due to stopping of the grid: GridTaskWorker [internal=true, marsh=BinaryMarshaller [], ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], fut=ComputeTaskInternalFuture [ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1492807644]], state=WAITING, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskCls=class org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, task=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask@49694fa9, continuous=false, lockRespProc=false, resCache=true, noFailover=false, affPartId=-1, affCacheName=null, mapTopVer=null, retryAttemptCnt=0, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapper=org.apache.ignite.internal.processors.task.GridTaskWorker$1@52d04fba] at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1317) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2069) at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.removeAll(IgniteCacheProxyImpl.java:1595) at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.removeAll(GatewayProtectedCacheProxy.java:1120) at org.apache.ignite.cache.PutAllDeadlockTest.lambda$testHashSetRemoveAll$5(PutAllDeadlockTest.java:211) at org.apache.ignite.testframework.GridTestUtils$7.call(GridTestUtils.java:1416) at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84) Caused by: class org.apache.ignite.compute.ComputeTaskCancelledException: Task cancelled due to stopping of the grid: GridTaskWorker [internal=true, marsh=BinaryMarshaller [], ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], fut=ComputeTaskInternalFuture [ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1492807644]], state=WAITING, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskCls=class org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, task=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask@49694fa9, continuous=false, lockRespProc=false, resCache=true, noFailover=false, affPartId=-1, affCacheName=null, mapTopVer=null, retryAttemptCnt=0, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapper=org.apache.ignite.internal.processors.task.GridTaskWorker$1@52d04fba] at org.apache.ignite.internal.util.IgniteUtils$10.apply(IgniteUtils.java:934) at org.apache.ignite.internal.util.IgniteUtils$10.apply(IgniteUtils.java:932) ... 7 more Caused by: class org.apache.ignite.internal.compute.ComputeTaskCancelledCheckedException: Task cancelled due to stopping of the grid: GridTaskWorker [internal=true, marsh=BinaryMarshaller [], ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], fut=ComputeTaskInternalFuture [ses=GridTaskSessionImpl [taskName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskClsName=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, sesId=96839b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, startTime=1573463258755, endTime=9223372036854775807, taskNodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, closed=false, cpSpi=null, failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=true, topPred=null, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapFut=IgniteFuture [orig=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=null, hash=1999865023]], execName=null], super=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, hash=1492807644]], state=WAITING, dep=GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=false, usage=2], taskCls=class org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask, task=org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheAdapter$RemoveAllTask@49694fa9, continuous=false, lockRespProc=false, resCache=true, noFailover=false, affPartId=-1, affCacheName=null, mapTopVer=null, retryAttemptCnt=0, subjId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, mapper=org.apache.ignite.internal.processors.task.GridTaskWorker$1@52d04fba] at org.apache.ignite.internal.processors.task.GridTaskProcessor.onKernalStop(GridTaskProcessor.java:276) at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2550) at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2498) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2643) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2606) at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance$6.run(IgnitionEx.java:2103) [2019-11-11 12:11:17,649][WARN ][exchange-worker-#1552%cache.PutAllDeadlockTest2%][GridDhtPartitionsExchangeFuture] Stop waiting for partitions release latch: Got interrupted while waiting for future to complete. [2019-11-11 12:11:17,649][INFO ][exchange-worker-#1552%cache.PutAllDeadlockTest2%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL] [2019-11-11 12:11:17,649][INFO ][exchange-worker-#1552%cache.PutAllDeadlockTest2%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], resVer=null, err=class org.apache.ignite.IgniteCheckedException: Node stopped] [2019-11-11 12:11:17,649][INFO ][exchange-worker-#1552%cache.PutAllDeadlockTest2%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=da077aa8-e739-4a05-a68e-fa2c8f600002, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=2906500e-c99c-4e83-89ed-3f6721500000, consistentId=127.0.0.1:47500, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1573463257960, loc=false, ver=2.8.0#20191111-sha1:00000000, isClient=false], done=true, newCrdFut=null], topVer=null] [2019-11-11 12:11:17,652][INFO ][tcp-disco-sock-reader-[6f8dd6fc 127.0.0.1:51673]-#213%cache.PutAllDeadlockTest2%][TestTcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:51673, rmtPort=51673 [2019-11-11 12:11:17,652][INFO ][disco-notifier-worker-#1449%cache.PutAllDeadlockTest0%][GridClusterStateProcessor] Received activate cluster request with BaselineTopology[id=0] initiator node ID: da077aa8-e739-4a05-a68e-fa2c8f600002 [2019-11-11 12:11:17,652][INFO ][disco-notifier-worker-#1449%cache.PutAllDeadlockTest0%][GridClusterStateProcessor] Started state transition: true [2019-11-11 12:11:17,652][INFO ][disco-notifier-worker-#1449%cache.PutAllDeadlockTest0%][GridClusterStateProcessor] Received state change finish message: true [2019-11-11 12:11:17,652][INFO ][tcp-disco-msg-worker-[da077aa8 127.0.0.1:47502]-#200%cache.PutAllDeadlockTest1%][TestTcpDiscoverySpi] New next node [newNext=TcpDiscoveryNode [id=2906500e-c99c-4e83-89ed-3f6721500000, consistentId=127.0.0.1:47500, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1573463257839, loc=false, ver=2.8.0#20191111-sha1:00000000, isClient=false]] [2019-11-11 12:11:17,652][INFO ][disco-event-worker-#1450%cache.PutAllDeadlockTest0%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=da077aa8-e739-4a05-a68e-fa2c8f600002, consistentId=127.0.0.1:47502, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1573463257939, loc=false, ver=2.8.0#20191111-sha1:00000000, isClient=false] [2019-11-11 12:11:17,653][WARN ][exchange-worker-#1499%cache.PutAllDeadlockTest1%][GridDhtPartitionsExchangeFuture] Stop waiting for partitions release latch: Got interrupted while waiting for future to complete. [2019-11-11 12:11:17,653][INFO ][tcp-disco-srvr-[:47500]-#196%cache.PutAllDeadlockTest0%][TestTcpDiscoverySpi] TCP discovery accepted incoming connection [rmtAddr=/127.0.0.1, rmtPort=39937] [2019-11-11 12:11:17,653][INFO ][tcp-disco-srvr-[:47500]-#196%cache.PutAllDeadlockTest0%][TestTcpDiscoverySpi] TCP discovery spawning a new thread for connection [rmtAddr=/127.0.0.1, rmtPort=39937] [2019-11-11 12:11:17,653][INFO ][disco-event-worker-#1450%cache.PutAllDeadlockTest0%][GridDiscoveryManager] Topology snapshot [ver=4, locNode=2906500e, servers=2, clients=0, state=ACTIVE, CPUs=8, offheap=6.3GB, heap=7.0GB] [2019-11-11 12:11:17,654][INFO ][disco-event-worker-#1450%cache.PutAllDeadlockTest0%][GridDiscoveryManager] ^-- Baseline [id=0, size=2, online=2, offline=0] [2019-11-11 12:11:17,654][INFO ][exchange-worker-#1499%cache.PutAllDeadlockTest1%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL] [2019-11-11 12:11:17,654][INFO ][exchange-worker-#1499%cache.PutAllDeadlockTest1%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], resVer=null, err=class org.apache.ignite.IgniteCheckedException: Node stopped] [2019-11-11 12:11:17,653][INFO ][disco-notifier-worker-#1497%cache.PutAllDeadlockTest1%][GridClusterStateProcessor] Received activate cluster request with BaselineTopology[id=0] initiator node ID: da077aa8-e739-4a05-a68e-fa2c8f600002 [2019-11-11 12:11:17,653][INFO ][Thread-86][GridCacheProcessor] Stopped cache [cacheName=default] [2019-11-11 12:11:17,657][INFO ][disco-notifier-worker-#1497%cache.PutAllDeadlockTest1%][GridClusterStateProcessor] Started state transition: true [2019-11-11 12:11:17,657][INFO ][disco-notifier-worker-#1497%cache.PutAllDeadlockTest1%][GridClusterStateProcessor] Received state change finish message: true [2019-11-11 12:11:17,657][INFO ][exchange-worker-#1499%cache.PutAllDeadlockTest1%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=6f8dd6fc-00ef-487e-8cd2-6834c0900001, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=2906500e-c99c-4e83-89ed-3f6721500000, consistentId=127.0.0.1:47500, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1573463257839, loc=false, ver=2.8.0#20191111-sha1:00000000, isClient=false], done=true, newCrdFut=null], topVer=null] [2019-11-11 12:11:17,657][INFO ][disco-event-worker-#1498%cache.PutAllDeadlockTest1%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=da077aa8-e739-4a05-a68e-fa2c8f600002, consistentId=127.0.0.1:47502, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1573463257950, loc=false, ver=2.8.0#20191111-sha1:00000000, isClient=false] [2019-11-11 12:11:17,658][INFO ][Thread-86][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2019-11-11 12:11:17,658][INFO ][disco-event-worker-#1498%cache.PutAllDeadlockTest1%][GridDiscoveryManager] Topology snapshot [ver=4, locNode=6f8dd6fc, servers=2, clients=0, state=ACTIVE, CPUs=8, offheap=6.3GB, heap=7.0GB] [2019-11-11 12:11:17,658][INFO ][disco-event-worker-#1498%cache.PutAllDeadlockTest1%][GridDiscoveryManager] ^-- Baseline [id=0, size=2, online=2, offline=0] [2019-11-11 12:11:17,657][INFO ][tcp-disco-sock-reader-[]-#216%cache.PutAllDeadlockTest0%][TestTcpDiscoverySpi] Started serving remote node connection [rmtAddr=/127.0.0.1:39937, rmtPort=39937] [2019-11-11 12:11:17,659][INFO ][tcp-disco-sock-reader-[6f8dd6fc 127.0.0.1:39937]-#216%cache.PutAllDeadlockTest0%][TestTcpDiscoverySpi] Initialized connection with remote server node [nodeId=6f8dd6fc-00ef-487e-8cd2-6834c0900001, rmtAddr=/127.0.0.1:39937] [2019-11-11 12:11:17,663][INFO ][Thread-86][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1573463258010, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=fe139b95e61-da077aa8-e739-4a05-a68e-fa2c8f600002, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionHistorySuppliersMap, pendingUndeploy=false, undeployed=true, usage=0] [2019-11-11 12:11:17,665][INFO ][Thread-86][PutAllDeadlockTest2] >>> +-----------------------------------------------------+ >>> Ignite ver. 2.8.0-SNAPSHOT#20191111-sha1:DEV stopped OK >>> +-----------------------------------------------------+ >>> Ignite instance name: cache.PutAllDeadlockTest2 >>> Grid uptime: 00:03:39.591 [2019-11-11 12:11:17,666][INFO ][tcp-disco-sock-reader-[2906500e 127.0.0.1:49241]-#204%cache.PutAllDeadlockTest1%][TestTcpDiscoverySpi] Finished serving remote node connection [rmtAddr=/127.0.0.1:49241, rmtPort=49241 [2019-11-11 12:11:17,667][INFO ][Thread-84][GridCacheProcessor] Stopped cache [cacheName=default] [2019-11-11 12:11:17,667][INFO ][Thread-84][GridCacheProcessor] Stopped cache [cacheName=ignite-sys-cache] [2019-11-11 12:11:17,668][INFO ][disco-notifier-worker-#1449%cache.PutAllDeadlockTest0%][GridClusterStateProcessor] Received activate cluster request with BaselineTopology[id=0] initiator node ID: 6f8dd6fc-00ef-487e-8cd2-6834c0900001 [2019-11-11 12:11:17,668][INFO ][disco-notifier-worker-#1449%cache.PutAllDeadlockTest0%][GridClusterStateProcessor] Started state transition: true [2019-11-11 12:11:17,668][INFO ][disco-notifier-worker-#1449%cache.PutAllDeadlockTest0%][GridClusterStateProcessor] Received state change finish message: true [2019-11-11 12:11:17,668][INFO ][disco-event-worker-#1450%cache.PutAllDeadlockTest0%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=6f8dd6fc-00ef-487e-8cd2-6834c0900001, consistentId=127.0.0.1:47501, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47501], discPort=47501, order=2, intOrder=2, lastExchangeTime=1573463257829, loc=false, ver=2.8.0#20191111-sha1:00000000, isClient=false] [2019-11-11 12:11:17,668][INFO ][disco-event-worker-#1450%cache.PutAllDeadlockTest0%][GridDiscoveryManager] Topology snapshot [ver=5, locNode=2906500e, servers=1, clients=0, state=ACTIVE, CPUs=8, offheap=6.3GB, heap=7.0GB] [2019-11-11 12:11:17,668][INFO ][disco-event-worker-#1450%cache.PutAllDeadlockTest0%][GridDiscoveryManager] ^-- Baseline [id=0, size=1, online=1, offline=0] [2019-11-11 12:11:17,669][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], waitTime=218122ms, futInfo=NA, mode=DISTRIBUTED] [2019-11-11 12:11:17,669][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=CompletableLatchUid [id=exchange, topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1]]]] [2019-11-11 12:11:17,669][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL] [2019-11-11 12:11:17,671][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], crd=true] [2019-11-11 12:11:17,671][INFO ][main][root] >>> Stopping grid [name=cache.PutAllDeadlockTest1, id=6f8dd6fc-00ef-487e-8cd2-6834c0900001] [2019-11-11 12:11:17,671][INFO ][sys-#1626%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=3, minorTopVer=1]] [2019-11-11 12:11:17,671][INFO ][Thread-84][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1573463257869, depMode=SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@18b4aac2, clsLdrId=0a139b95e61-6f8dd6fc-00ef-487e-8cd2-6834c0900001, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.IgniteDhtPartitionsToReloadMap, pendingUndeploy=false, undeployed=true, usage=0] [2019-11-11 12:11:17,673][INFO ][Thread-84][PutAllDeadlockTest1] >>> +-----------------------------------------------------+ >>> Ignite ver. 2.8.0-SNAPSHOT#20191111-sha1:DEV stopped OK >>> +-----------------------------------------------------+ >>> Ignite instance name: cache.PutAllDeadlockTest1 >>> Grid uptime: 00:03:39.772 [2019-11-11 12:11:17,673][INFO ][sys-#1626%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], err=null] [2019-11-11 12:11:17,674][INFO ][sys-#1626%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=2906500e-c99c-4e83-89ed-3f6721500000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=2906500e-c99c-4e83-89ed-3f6721500000, consistentId=127.0.0.1:47500, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1573463477661, loc=true, ver=2.8.0#20191111-sha1:00000000, isClient=false], done=true, newCrdFut=null], topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1]] [2019-11-11 12:11:17,674][INFO ][sys-#1626%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Exchange timings [startVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], stage="Waiting in exchange queue" (0 ms), stage="Exchange parameters initialization" (0 ms), stage="Determine exchange type" (0 ms), stage="Preloading notification" (0 ms), stage="WAL history reservation" (0 ms), stage="Wait partitions release" (218122 ms), stage="Wait partitions release latch" (0 ms), stage="Wait partitions release" (0 ms), stage="After states restored callback" (0 ms), stage="Waiting for all single messages" (1 ms), stage="Affinity recalculation (crd)" (0 ms), stage="Collect update counters and create affinity messages" (0 ms), stage="Validate partitions states" (0 ms), stage="Apply update counters" (0 ms), stage="Full message preparing" (1 ms), stage="Full message sending" (0 ms), stage="Exchange done" (0 ms), stage="Total time" (218124 ms)] [2019-11-11 12:11:17,674][INFO ][sys-#1626%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=3, minorTopVer=1], stage="Affinity change by custom message [grp=ignite-sys-cache]" (0 ms) (parent=Determine exchange type), stage="Affinity change by custom message [grp=default]" (0 ms) (parent=Determine exchange type)] [2019-11-11 12:11:17,675][INFO ][main][root] >>> Stopping grid [name=cache.PutAllDeadlockTest0, id=2906500e-c99c-4e83-89ed-3f6721500000] [2019-11-11 12:11:17,675][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (obsolete exchange ID) [top=AffinityTopologyVersion [topVer=3, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, node=2906500e-c99c-4e83-89ed-3f6721500000] [2019-11-11 12:11:17,675][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], crd=true, evt=NODE_LEFT, evtNode=da077aa8-e739-4a05-a68e-fa2c8f600002, customEvt=null, allowMerge=true] [2019-11-11 12:11:17,675][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=DISTRIBUTED] [2019-11-11 12:11:17,675][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partitions release latch: ServerLatch [permits=0, pendingAcks=HashSet [], super=CompletableLatch [id=CompletableLatchUid [id=exchange, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0]]]] [2019-11-11 12:11:17,675][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], waitTime=0ms, futInfo=NA, mode=LOCAL] [2019-11-11 12:11:17,676][INFO ][sys-#1614%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Coordinator received all messages, try merge [ver=AffinityTopologyVersion [topVer=4, minorTopVer=0]] [2019-11-11 12:11:17,676][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], crd=true] [2019-11-11 12:11:17,676][INFO ][sys-#1614%cache.PutAllDeadlockTest0%][GridCachePartitionExchangeManager] Merge exchange future [curFut=AffinityTopologyVersion [topVer=4, minorTopVer=0], mergedFut=AffinityTopologyVersion [topVer=5, minorTopVer=0], evt=NODE_LEFT, evtNode=6f8dd6fc-00ef-487e-8cd2-6834c0900001, evtNodeClient=false] [2019-11-11 12:11:17,676][INFO ][sys-#1614%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [2019-11-11 12:11:17,676][INFO ][sys-#1614%cache.PutAllDeadlockTest0%][GridAffinityAssignmentCache] Local node affinity assignment distribution is not ideal [cache=default, expectedPrimary=1024,00, actualPrimary=1024, expectedBackups=1024,00, actualBackups=0, warningThreshold=50,00%] [2019-11-11 12:11:17,681][INFO ][sys-#1614%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], err=null] [2019-11-11 12:11:17,681][INFO ][sys-#1614%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=2906500e-c99c-4e83-89ed-3f6721500000, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], evt=NODE_LEFT, evtNode=TcpDiscoveryNode [id=da077aa8-e739-4a05-a68e-fa2c8f600002, consistentId=127.0.0.1:47502, addrs=ArrayList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47502], discPort=47502, order=3, intOrder=3, lastExchangeTime=1573463257939, loc=false, ver=2.8.0#20191111-sha1:00000000, isClient=false], done=true, newCrdFut=null], topVer=AffinityTopologyVersion [topVer=5, minorTopVer=0]] [2019-11-11 12:11:17,681][INFO ][sys-#1614%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Exchange timings [startVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], stage="Waiting in exchange queue" (12 ms), stage="Exchange parameters initialization" (0 ms), stage="Determine exchange type" (0 ms), stage="Preloading notification" (0 ms), stage="WAL history reservation" (0 ms), stage="Wait partitions release" (0 ms), stage="Wait partitions release latch" (0 ms), stage="Wait partitions release" (0 ms), stage="After states restored callback" (0 ms), stage="Finalize update counters" (0 ms), stage="Waiting for all single messages" (0 ms), stage="Exchanges merge" (0 ms), stage="Affinity recalculation (crd)" (2 ms), stage="Collect update counters and create affinity messages" (0 ms), stage="Validate partitions states" (0 ms), stage="Apply update counters" (0 ms), stage="Detect lost partitions" (0 ms), stage="Full message preparing" (0 ms), stage="Full message sending" (0 ms), stage="State finish message sending" (0 ms), stage="Exchange done" (0 ms), stage="Total time" (14 ms)] [2019-11-11 12:11:17,681][INFO ][sys-#1614%cache.PutAllDeadlockTest0%][GridDhtPartitionsExchangeFuture] Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], resVer=AffinityTopologyVersion [topVer=5, minorTopVer=0], stage="Affinity recalculation (partitions availability) [grp=ignite-sys-cache]" (0 ms) (parent=Affinity recalculation (crd)), stage="Affinity recalculation (partitions availability) [grp=default]" (0 ms) (parent=Affinity recalculation (crd)), stage="Affinity initialization (enforced) [grp=ignite-sys-cache]" (0 ms) (parent=Affinity recalculation (crd))] [2019-11-11 12:11:17,684][INFO ][exchange-worker-#1451%cache.PutAllDeadlockTest0%][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=5, minorTopVer=0], force=false, evt=NODE_LEFT, node=da077aa8-e739-4a05-a68e-fa2c8f600002] Process finished with exit code 137 (interrupted by signal 9: SIGKILL)