[IGNITE-10750] MVCC: Unexpected tx state exception in multithreaded tests

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8
• Component/s: mvcc
• Labels: MakeTeamcityGreenAgain failover
• Assignee: Andrey Mashenkov
• Reporter: Roman Kondakov
• Created: 20/Dec/18 11:25
• Updated: 11/Mar/19 06:32
• Resolved: 30/Jan/19 14:12

Description

 There is a number of tests in MVCC suite sporadically fail with the similar IgniteTxMvccVersionCheckedException. This problem occurs both on stable and unstable topologies.

These tests were muted on TC:

• CacheMvccPartitionedCoordinatorFailoverTest
  • testUpdate_N_Objects_ClientServer_Backups1__PutGet_CoordinatorFails
  • testAccountsTxScan_ClientServer_Backups2_CoordinatorFails

• CacheMvccReplicatedCoordinatorFailoverTest
  • testUpdate_N_Objects_Servers_Backups0__PutGet_CoordinatorFails_Persistence
  • testAccountsTxScan_Server_Backups0_CoordinatorFails

• CacheMvccTransactionsTest
  • testUpdate_N_Objects_ClientServer_Backups2_Get

• IgniteCacheGroupsTest.testScanQueryMvccTxReplicated
• CacheMvccReplicatedSqlCoordinatorFailoverTest.testAccountsTxSql_Server_Backups0_CoordinatorFails
• CacheMvccPartitionedSqlCoordinatorFailoverTest
  • testAccountsTxSql_ClientServer_Backups2_CoordinatorFails
     

    junit.framework.AssertionFailedError: Unexpected error: junit.framework.AssertionFailedError: Unexpected tx exception. javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Failed to update backup node: [localNodeId=fd28ba5c-6216-4aaa-a496-ba24cee00001, remoteNodeId=4198d600-48ee-4627-9802-0d1573800000]
     at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
     at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
     at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1171)
     at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.putAll(GatewayProtectedCacheProxy.java:868)
     at org.apache.ignite.internal.processors.cache.mvcc.CacheMvccAbstractTest.writeAllByMode(CacheMvccAbstractTest.java:2021)
     at org.apache.ignite.internal.processors.cache.mvcc.CacheMvccAbstractTest$7.apply(CacheMvccAbstractTest.java:1143)
     at org.apache.ignite.internal.processors.cache.mvcc.CacheMvccAbstractTest$7.apply(CacheMvccAbstractTest.java:1116)
     at org.apache.ignite.internal.processors.cache.mvcc.CacheMvccAbstractTest$9.call(CacheMvccAbstractTest.java:1328)
     at org.apache.ignite.internal.processors.cache.mvcc.CacheMvccAbstractTest$9.call(CacheMvccAbstractTest.java:1323)
     at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84)
    Caused by: class org.apache.ignite.IgniteCheckedException: Failed to update backup node: [localNodeId=fd28ba5c-6216-4aaa-a496-ba24cee00001, remoteNodeId=4198d600-48ee-4627-9802-0d1573800000]
     at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.onResult(GridDhtTxAbstractEnlistFuture.java:1012)
     at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistResponse(GridDhtTransactionalCacheAdapter.java:2342)
     at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$1200(GridDhtTransactionalCacheAdapter.java:112)
     at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$18.apply(GridDhtTransactionalCacheAdapter.java:257)
     at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$18.apply(GridDhtTransactionalCacheAdapter.java:255)
     at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1127)
     at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:592)
     at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:391)
     at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:317)
     at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:108)
     at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:307)
     at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1568)
     at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1196)
     at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
     at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1092)
     at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
     at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
     at java.lang.Thread.run(Thread.java:748)
    Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []]
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1070)
     at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:1966)
     at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:544)
     at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.mvccSet(GridCacheMapEntry.java:1142)
     at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.mvccEnlistBatch(IgniteTxHandler.java:1948)
     at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistRequest(GridDhtTransactionalCacheAdapter.java:2291)
     at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$1100(GridDhtTransactionalCacheAdapter.java:112)
     at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$17.apply(GridDhtTransactionalCacheAdapter.java:250)
     at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$17.apply(GridDhtTransactionalCacheAdapter.java:248)
     ... 13 more
    Caused by: class org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException: Unexpected state: [state=0, rowVer=1545032895009:306:1, txVer=1545032895009:310:1, localNodeId=4198d600-48ee-4627-9802-0d1573800000]
     at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:317)
     at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:304)
     at org.apache.ignite.internal.processors.cache.tree.mvcc.data.MvccUpdateDataRow.visit(MvccUpdateDataRow.java:416)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3066)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.init(BPlusTree.java:3035)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.notFound(BPlusTree.java:2991)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:308)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5659)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:273)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5644)
     at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:168)
     at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:349)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read(BPlusTree.java:5846)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2630)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doVisit(BPlusTree.java:2597)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$7900(BPlusTree.java:92)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3193)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.access$4900(BPlusTree.java:2953)
     at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1067)
     ... 21 more
    

Issue Links

• duplicates: IGNITE-10752 MVCC: Tests invariants are violated sometimes
• is duplicated by: IGNITE-9892 MVCC: Exchange hangs on mvcc coordinator fail
• is duplicated by: IGNITE-10786 MVCC: Flaky test IgniteClusterActivateDeactivateTestWithPersistence#testDeactivateDuringEvictionAndRebalance
• is duplicated by: IGNITE-10788 MVCC: Get operation may hang in some cases
• is related to: IGNITE-10786 MVCC: Flaky test IgniteClusterActivateDeactivateTestWithPersistence#testDeactivateDuringEvictionAndRebalance
• links to: GitHub Pull Request #5900