[IGNITE-5549] Ignite Cache Failover2: test suite hands up from CacheAsyncOperationsFailoverTxTest.testAsyncFailover()

• Type: Test
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.1
• Fix Version/s: 2.3
• Labels: MakeTeamcityGreenAgain test-fail
• Assignee: Alexey Goncharuk
• Reporter: Dmitry Pavlov
• Created: 20/Jun/17 09:02
• Updated: 13/Sep/17 16:20
• Resolved: 13/Sep/17 16:20

Description

Timeout came from CacheAsyncOperationsFailoverTxTest.testAsyncFailover() test:
http://ci.ignite.apache.org/viewLog.html?buildId=756819&buildTypeId=Ignite20Tests_IgniteCacheFailover2&tab=buildResultsDiv

ransaction has been already completed.
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finishDhtLocal(IgniteTxHandler.java:787)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:727)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishRequest(IgniteTxHandler.java:683)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$200(IgniteTxHandler.java:95)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:165)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$3.apply(IgniteTxHandler.java:163)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1042)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:485)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at java.lang.Thread.run(Thread.java:745)
[16:29:30]W:		 [org.apache.ignite:ignite-core] java.lang.AssertionError: Invalid threadId [this=GridCacheMvccCandidate [nodeId=d6f0c17e-68b3-4da7-bd02-914ada700000, ver=GridCacheVersion [topVer=113333427, order=1501856691702, nodeOrder=1], threadId=211336, id=30502322, topVer=AffinityTopologyVersion [topVer=126, minorTopVer=0], reentry=null, otherNodeId=d6f0c17e-68b3-4da7-bd02-914ada700000, otherVer=GridCacheVersion [topVer=113333427, order=1501856691702, nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, serOrder=null, key=org.apache.ignite.internal.processors.cache.distributed.CacheAsyncOperationsFailoverAbstractTest$TestKey [idHash=742616132, hash=-1080846605, key=48], masks=local=1|owner=0|ready=0|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0, prevVer=null, nextVer=null], prev=GridCacheMvccCandidate [nodeId=d6f0c17e-68b3-4da7-bd02-914ada700000, ver=GridCacheVersion [topVer=113333425, order=1501856674174, nodeOrder=1], threadId=216954, id=30502305, topVer=AffinityTopologyVersion [topVer=126, minorTopVer=0], reentry=null, otherNodeId=d6f0c17e-68b3-4da7-bd02-914ada700000, otherVer=GridCacheVersion [topVer=113333425, order=1501856674174, nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, serOrder=null, key=org.apache.ignite.internal.processors.cache.distributed.CacheAsyncOperationsFailoverAbstractTest$TestKey [idHash=950720167, hash=-1566972591, key=146], masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0, prevVer=null, nextVer=GridCacheVersion [topVer=113333427, order=1501856691702, nodeOrder=1]]]
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccCandidate.previous(GridCacheMvccCandidate.java:540)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccManager.addNext(GridCacheMvccManager.java:827)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.addDhtLocal(GridDhtCacheEntry.java:266)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.tmLock(GridDhtCacheEntry.java:284)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.lockMultiple(IgniteTxManager.java:1602)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.prepareTx(IgniteTxManager.java:834)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userPrepare(IgniteTxLocalAdapter.java:425)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareAsyncLocal(GridNearTxLocal.java:3382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareColocatedTx(IgniteTxHandler.java:233)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.proceedPrepare(GridNearOptimisticTxPrepareFuture.java:571)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.proceedPrepare(GridNearOptimisticTxPrepareFuture.java:503)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.prepare(GridNearOptimisticTxPrepareFuture.java:489)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.prepare0(GridNearOptimisticTxPrepareFuture.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepareOnTopology(GridNearOptimisticTxPrepareFutureAdapter.java:137)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepare(GridNearOptimisticTxPrepareFutureAdapter.java:74)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareNearTxLocal(GridNearTxLocal.java:3153)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commitNearTxLocalAsync(GridNearTxLocal.java:3209)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.optimisticPutFuture(GridNearTxLocal.java:2384)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.putAllAsync0(GridNearTxLocal.java:805)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.putAllAsync(GridNearTxLocal.java:364)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$35.op(GridCacheAdapter.java:2814)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOp.op(GridCacheAdapter.java:5086)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOp(GridCacheAdapter.java:4230)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture.execute(GridCacheAdapter.java:4807)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1$1.apply(GridCacheAdapter.java:4837)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1$1.apply(GridCacheAdapter.java:4832)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFinishedFuture.listen(GridFinishedFuture.java:126)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1.apply(GridCacheAdapter.java:4832)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1.apply(GridCacheAdapter.java:4809)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridEmbeddedFuture$4$1.applyx(GridEmbeddedFuture.java:238)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener2.apply(GridEmbeddedFuture.java:326)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener2.apply(GridEmbeddedFuture.java:319)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:81)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:81)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:81)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheCompoundIdentityFuture.onDone(GridCacheCompoundIdentityFuture.java:56)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.onDone(GridNearTxFinishFuture.java:340)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.onDone(GridNearTxFinishFuture.java:69)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:129)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture$FinishMiniFuture.onNearFinishResponse(GridNearTxFinishFuture.java:990)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.onResult(GridNearTxFinishFuture.java:213)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishResponse(IgniteTxHandler.java:587)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$300(IgniteTxHandler.java:95)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$4.apply(IgniteTxHandler.java:171)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$4.apply(IgniteTxHandler.java:169)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1042)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:485)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at java.lang.Thread.run(Thread.java:745)
[16:29:30]W:		 [org.apache.ignite:ignite-core] java.lang.AssertionError: Invalid threadId [this=GridCacheMvccCandidate [nodeId=d6f0c17e-68b3-4da7-bd02-914ada700000, ver=GridCacheVersion [topVer=113333427, order=1501856691820, nodeOrder=1], threadId=211335, id=30502332, topVer=AffinityTopologyVersion [topVer=126, minorTopVer=0], reentry=null, otherNodeId=d6f0c17e-68b3-4da7-bd02-914ada700000, otherVer=GridCacheVersion [topVer=113333427, order=1501856691820, nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, serOrder=null, key=org.apache.ignite.internal.processors.cache.distributed.CacheAsyncOperationsFailoverAbstractTest$TestKey [idHash=2121857807, hash=1371048546, key=450], masks=local=1|owner=0|ready=0|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0, prevVer=null, nextVer=null], prev=GridCacheMvccCandidate [nodeId=d6f0c17e-68b3-4da7-bd02-914ada700000, ver=GridCacheVersion [topVer=113333425, order=1501856673990, nodeOrder=1], threadId=216950, id=30502306, topVer=AffinityTopologyVersion [topVer=126, minorTopVer=0], reentry=null, otherNodeId=d6f0c17e-68b3-4da7-bd02-914ada700000, otherVer=GridCacheVersion [topVer=113333425, order=1501856673990, nodeOrder=1], mappedDhtNodes=null, mappedNearNodes=null, ownerVer=null, serOrder=null, key=org.apache.ignite.internal.processors.cache.distributed.CacheAsyncOperationsFailoverAbstractTest$TestKey [idHash=1954814914, hash=1614111539, key=273], masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0, prevVer=null, nextVer=GridCacheVersion [topVer=113333427, order=1501856691820, nodeOrder=1]]]
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccCandidate.previous(GridCacheMvccCandidate.java:540)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccManager.addNext(GridCacheMvccManager.java:827)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.addDhtLocal(GridDhtCacheEntry.java:266)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.tmLock(GridDhtCacheEntry.java:284)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.lockMultiple(IgniteTxManager.java:1602)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.prepareTx(IgniteTxManager.java:834)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userPrepare(IgniteTxLocalAdapter.java:425)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareAsyncLocal(GridNearTxLocal.java:3382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareColocatedTx(IgniteTxHandler.java:233)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.proceedPrepare(GridNearOptimisticTxPrepareFuture.java:571)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.proceedPrepare(GridNearOptimisticTxPrepareFuture.java:503)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.prepare(GridNearOptimisticTxPrepareFuture.java:489)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFuture.prepare0(GridNearOptimisticTxPrepareFuture.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepareOnTopology(GridNearOptimisticTxPrepareFutureAdapter.java:137)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepare(GridNearOptimisticTxPrepareFutureAdapter.java:74)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareNearTxLocal(GridNearTxLocal.java:3153)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.commitNearTxLocalAsync(GridNearTxLocal.java:3209)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.optimisticPutFuture(GridNearTxLocal.java:2384)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.putAllAsync0(GridNearTxLocal.java:805)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.putAllAsync(GridNearTxLocal.java:364)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$35.op(GridCacheAdapter.java:2814)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOp.op(GridCacheAdapter.java:5086)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.asyncOp(GridCacheAdapter.java:4230)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture.execute(GridCacheAdapter.java:4807)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1$1.apply(GridCacheAdapter.java:4837)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1$1.apply(GridCacheAdapter.java:4832)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFinishedFuture.listen(GridFinishedFuture.java:126)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1.apply(GridCacheAdapter.java:4832)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$AsyncOpRetryFuture$1.apply(GridCacheAdapter.java:4809)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridEmbeddedFuture$4$1.applyx(GridEmbeddedFuture.java:238)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener2.apply(GridEmbeddedFuture.java:326)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridEmbeddedFuture$AsyncListener2.apply(GridEmbeddedFuture.java:319)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:81)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:81)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:81)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheCompoundIdentityFuture.onDone(GridCacheCompoundIdentityFuture.java:56)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.onDone(GridNearTxFinishFuture.java:340)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.onDone(GridNearTxFinishFuture.java:69)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:129)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:382)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:346)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:334)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:494)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:473)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture$FinishMiniFuture.onNearFinishResponse(GridNearTxFinishFuture.java:990)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.onResult(GridNearTxFinishFuture.java:213)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processNearTxFinishResponse(IgniteTxHandler.java:587)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$300(IgniteTxHandler.java:95)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$4.apply(IgniteTxHandler.java:171)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$4.apply(IgniteTxHandler.java:169)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1042)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:561)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:485)
[16:29:30]W:		 [org.apache.ignite:ignite-core] 	at java.lang.Thread.run(Thread.java:745)

Test reproduces locally.

Issue Links

• links to: GitHub Pull Request #2626