Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-11623

MVCC: testRebalancingDuringLoad_N tests are flacky in master

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None

      Description

      The following tests are flacky in master:
      IgnitePdsMvccTestSuite3: IgnitePdsContinuousRestartTest.testRebalancingDuringLoad_10_10_1_1 ( Test history )
      testRebalancingDuringLoad_1000_20000_8_16 ( Test history)
      testRebalancingDuringLoad_1000_500_8_1 (Test history)
      IgniteClusterActivateDeactivateTestWithPersistence.testDeactivateDuringEvictionAndRebalance (Test history)

      According to the logs, the cause of the fall: IgniteTxUnexpectedStateCheckedException: Unexpected state

      java.lang.AssertionError: Unexpected exception: javax.cache.CacheException: class org.apache.ignite.transactions.TransactionRollbackException: Transaction has been rolled back: be9b08fa961-00000000-09d4-4a7d-0000-000000000001
       at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1318)
       at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2064)
       at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1371)
       at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.putAll(GatewayProtectedCacheProxy.java:866)
       at org.apache.ignite.internal.processors.cache.persistence.IgnitePdsContinuousRestartTest$1.call(IgnitePdsContinuousRestartTest.java:280)
       at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84)
      Caused by: class org.apache.ignite.transactions.TransactionRollbackException: Transaction has been rolled back: be9b08fa961-00000000-09d4-4a7d-0000-000000000001
       at org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:935)
       at org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:933)
       ... 6 more
      Caused by: class org.apache.ignite.internal.transactions.IgniteTxRollbackCheckedException: Transaction has been rolled back: be9b08fa961-00000000-09d4-4a7d-0000-000000000001
       at org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4297)
       at org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll0(GridCacheAdapter.java:3000)
       at org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll(GridCacheAdapter.java:2989)
       at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1368)
       ... 3 more
      Caused by: class org.apache.ignite.IgniteCheckedException: Failed to update backup node: [localNodeId=44bb5db4-5317-4ab1-8379-16f3b9200003, remoteNodeId=08a44c02-ebe8-46e8-a7ca-890c62500001]
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.onResult(GridDhtTxAbstractEnlistFuture.java:999)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistResponse(GridDhtTransactionalCacheAdapter.java:2350)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$1200(GridDhtTransactionalCacheAdapter.java:112)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$18.apply(GridDhtTransactionalCacheAdapter.java:257)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$18.apply(GridDhtTransactionalCacheAdapter.java:255)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1126)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
       at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1561)
       at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1189)
       at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
       at org.apache.ignite.internal.managers.communication.GridIoManager$8.run(GridIoManager.java:1086)
       at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:550)
       at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
       at java.lang.Thread.run(Thread.java:748)
      Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []]
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1069)
       at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:1938)
       at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.mvccUpdate(GridCacheOffheapManager.java:2027)
       at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:523)
       at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.mvccSet(GridCacheMapEntry.java:1115)
       at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.mvccEnlistBatch(IgniteTxHandler.java:1937)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processDhtTxQueryEnlistRequest(GridDhtTransactionalCacheAdapter.java:2299)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$1100(GridDhtTransactionalCacheAdapter.java:112)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$17.apply(GridDhtTransactionalCacheAdapter.java:250)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$17.apply(GridDhtTransactionalCacheAdapter.java:248)
       ... 13 more
      Caused by: class org.apache.ignite.internal.transactions.IgniteTxUnexpectedStateCheckedException: Unexpected state: [state=0, rowVer=1553427560308:27927:1, txVer=1553427560308:29840:1, localNodeId=08a44c02-ebe8-46e8-a7ca-890c62500001]
       at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:328)
       at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:315)
       at org.apache.ignite.internal.processors.cache.tree.mvcc.data.MvccUpdateDataRow.visit(MvccUpdateDataRow.java:416)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3060)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.init(BPlusTree.java:3030)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.notFound(BPlusTree.java:2986)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:308)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5643)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:273)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5629)
       at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:168)
       at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:349)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read(BPlusTree.java:5830)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2626)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2635)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doVisit(BPlusTree.java:2593)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$7800(BPlusTree.java:92)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3187)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.access$4900(BPlusTree.java:2948)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1066)
       ... 22 more
      
      java.lang.AssertionError: Unexpected exception: javax.cache.CacheException: class org.apache.ignite.transactions.TransactionRollbackException: Transaction has been rolled back: 2dbd8eba961-00000000-09d3-5d15-0000-000000000001
       at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1318)
       at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:2064)
       at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1371)
       at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.putAll(GatewayProtectedCacheProxy.java:866)
       at org.apache.ignite.internal.processors.cache.persistence.IgnitePdsContinuousRestartTest$1.call(IgnitePdsContinuousRestartTest.java:280)
       at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:84)
      Caused by: class org.apache.ignite.transactions.TransactionRollbackException: Transaction has been rolled back: 2dbd8eba961-00000000-09d3-5d15-0000-000000000001
       at org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:935)
       at org.apache.ignite.internal.util.IgniteUtils$11.apply(IgniteUtils.java:933)
       ... 6 more
      Caused by: class org.apache.ignite.internal.transactions.IgniteTxRollbackCheckedException: Transaction has been rolled back: 2dbd8eba961-00000000-09d3-5d15-0000-000000000001
       at org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4283)
       at org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll0(GridCacheAdapter.java:2986)
       at org.apache.ignite.internal.processors.cache.GridCacheAdapter.putAll(GridCacheAdapter.java:2975)
       at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.putAll(IgniteCacheProxyImpl.java:1368)
       ... 3 more
      Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on bounds: [lower=MvccMaxSearchRow [], upper=MvccMinSearchRow []]
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1069)
       at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:1958)
       at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.mvccUpdate(GridCacheOffheapManager.java:2026)
       at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccUpdate(IgniteCacheOffheapManagerImpl.java:536)
       at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.mvccSet(GridCacheMapEntry.java:1117)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.continueLoop(GridDhtTxAbstractEnlistFuture.java:461)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxAbstractEnlistFuture.init(GridDhtTxAbstractEnlistFuture.java:363)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.processNearTxEnlistRequest(GridDhtTransactionalCacheAdapter.java:2070)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter.access$900(GridDhtTransactionalCacheAdapter.java:112)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:229)
       at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$14.apply(GridDhtTransactionalCacheAdapter.java:227)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1126)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
       at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1$2$1.run(GridCacheIoManager.java:288)
       at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:550)
       at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
       at java.lang.Thread.run(Thread.java:748)
      Caused by: class org.apache.ignite.internal.transactions.IgniteTxUnexpectedStateCheckedException: Unexpected state: [state=0, rowVer=1553366856829:189821:1, txVer=1553366856829:189838:1, localNodeId=aa9e5187-4b3e-456c-8eaa-7ccf6a500001]
       at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:328)
       at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:315)
       at org.apache.ignite.internal.processors.cache.tree.mvcc.data.MvccUpdateDataRow.visit(MvccUpdateDataRow.java:416)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3060)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.init(BPlusTree.java:3030)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.notFound(BPlusTree.java:2986)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:308)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5643)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:273)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5629)
       at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:168)
       at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:349)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.read(BPlusTree.java:5830)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2626)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visitDown(BPlusTree.java:2635)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doVisit(BPlusTree.java:2593)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$7800(BPlusTree.java:92)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.visit(BPlusTree.java:3187)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$TreeVisitor.access$4900(BPlusTree.java:2948)
       at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.visit(BPlusTree.java:1066)
       ... 19 more
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                NSAmelchev Amelchev Nikita
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: