[IGNITE-10561] MVCC: Fix MVCC cache rebalance.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.8
• Component/s: mvcc
• Labels: mvcc_stability
• Assignee: Andrey Mashenkov
• Reporter: Roman Kondakov
• Created: 05/Dec/18 18:27
• Updated: 25/Mar/19 11:04
• Resolved: 07/Mar/19 11:06

Description

Unexpected transaction state may be observed on some nodes after rebalance. Reproducer: GridCacheRebalancingSyncSelfTest#testComplexRebalancing

Stacktrace:

javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]

	at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1756)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931)
	at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640)
	at org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:251)
	at org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.checkData(GridCacheRebalancingSyncSelfTest.java:213)
	at org.apache.ignite.internal.processors.cache.distributed.rebalancing.GridCacheRebalancingSyncSelfTest.testComplexRebalancing(GridCacheRebalancingSyncSelfTest.java:588)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at junit.framework.TestCase.runTest(TestCase.java:176)
	at org.apache.ignite.testframework.junits.GridAbstractTest.access$001(GridAbstractTest.java:149)
	at org.apache.ignite.testframework.junits.GridAbstractTest$6.evaluate(GridAbstractTest.java:2106)
	at org.apache.ignite.testframework.junits.GridAbstractTest$7.run(GridAbstractTest.java:2123)
	at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on bounds: [lower=MvccSnapshotSearchRow [res=null, snapshot=MvccSnapshotResponse [futId=149236, crdVer=1544033736224, cntr=800063, opCntr=1073741823, txs=null, cleanupVer=0, tracking=0]], upper=MvccMinSearchRow []]
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1035)
	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.mvccFind(IgniteCacheOffheapManagerImpl.java:2849)
	at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.mvccRead(IgniteCacheOffheapManagerImpl.java:695)
	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.getAllAsync0(GridCacheAdapter.java:2024)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtAllAsync(GridDhtCacheAdapter.java:807)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.getAsync(GridDhtGetSingleFuture.java:399)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map0(GridDhtGetSingleFuture.java:277)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.map(GridDhtGetSingleFuture.java:259)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtGetSingleFuture.init(GridDhtGetSingleFuture.java:182)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.getDhtSingleAsync(GridDhtCacheAdapter.java:918)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetRequest(GridDhtCacheAdapter.java:933)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$2.apply(GridDhtTransactionalCacheAdapter.java:152)
	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTransactionalCacheAdapter$2.apply(GridDhtTransactionalCacheAdapter.java:150)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1059)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:584)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:383)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:309)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:100)
	at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:299)
	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1568)
	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1196)
	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1092)
	at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:505)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
	... 1 more
Caused by: class org.apache.ignite.internal.transactions.IgniteTxMvccVersionCheckedException: Unexpected state: [state=0, rowVer=1544033736224:446552:1, txVer=1544033736224:800063:1073741823, localNodeId=e60be4d5-b1d7-4c09-8b88-9b8964400002]
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:317)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.unexpectedStateException(MvccUtils.java:304)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.isVisible(MvccUtils.java:271)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.isVisible(MvccUtils.java:210)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils$GetVisibleState.apply(MvccUtils.java:906)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils$GetVisibleState.apply(MvccUtils.java:890)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.invoke(MvccUtils.java:618)
	at org.apache.ignite.internal.processors.cache.mvcc.MvccUtils.getVisibleState(MvccUtils.java:362)
	at org.apache.ignite.internal.processors.cache.tree.mvcc.search.MvccSnapshotSearchRow.apply(MvccSnapshotSearchRow.java:88)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ClosureCursor.fillFromBuffer0(BPlusTree.java:5446)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.fillFromBuffer(BPlusTree.java:5316)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.init(BPlusTree.java:5241)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetCursor.notFound(BPlusTree.java:2935)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:306)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5651)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5636)
	at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:159)
	at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1309)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1318)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind(BPlusTree.java:1276)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$8700(BPlusTree.java:90)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$AbstractForwardCursor.find(BPlusTree.java:5325)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ClosureCursor.iterate(BPlusTree.java:5475)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$ClosureCursor.access$5000(BPlusTree.java:5401)
	at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.iterate(BPlusTree.java:1032)
	... 25 more

 

Issue Links

• blocks: IGNITE-11088 Flacky LocalWalModeChangeDuringRebalancingSelfTest.testWithExchangesMerge
• Dependent: IGNITE-10693 MVCC TX: Random server restart tests became failed
• is blocked by: IGNITE-10604 MVCC: mvcc history can be missed during remove operation.
• is duplicated by: IGNITE-10560 MVCC: Data loss after rebalance
• is duplicated by: IGNITE-11088 Flacky LocalWalModeChangeDuringRebalancingSelfTest.testWithExchangesMerge
• is related to: IGNITE-10769 MVCC: CacheMvccContinuousQueryClientTest.testNodeJoinsRestartQuery fails sometimes
• is related to: IGNITE-11623 MVCC: testRebalancingDuringLoad_N tests are flacky in master
• relates to: IGNITE-9949 MVCC TX: Scan query can return wrong result on unstable topology
• links to: GitHub Pull Request #5799