[IGNITE-10241] MVCC TX: orphaned remote transactions

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 2.8
• Component/s: mvcc
• Labels: failover
• Assignee: Igor Seliverstov
• Reporter: Igor Seliverstov
• Created: 13/Nov/18 14:02
• Updated: 11/Mar/19 06:32
• Resolved: 04/Feb/19 11:51

Description

CacheMvccPartitionedSqlCoordinatorFailoverTest#testUpdate_N_Objects_ClientServer_Backups2_Sql occasionally fails with following error:

[2018-11-13 15:03:44,240][ERROR][sys-#19392%mvcc.CacheMvccPartitionedSqlCoordinatorFailoverTest0%][GridDhtPartitionsExchangeFuture] Failed to notify listener: o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2@22ae2fbd
java.lang.ClassCastException: org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxRemote cannot be cast to org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal
 at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.rollbackMvccTxOnCoordinatorChange(IgniteTxManager.java:327)
 at org.apache.ignite.internal.processors.cache.mvcc.MvccProcessorImpl.onExchangeDone(MvccProcessorImpl.java:394)
 at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:2095)
 at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.finishExchangeOnCoordinator(GridDhtPartitionsExchangeFuture.java:3391)
 at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAllReceived(GridDhtPartitionsExchangeFuture.java:3157)
 at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processSingleMessage(GridDhtPartitionsExchangeFuture.java:2773)
 at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:139)
 at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2578)
 at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:2566)
 at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:395)
 at org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:355)
 at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceiveSingleMessage(GridDhtPartitionsExchangeFuture.java:2566)
 at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(GridCachePartitionExchangeManager.java:1747)
 at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1200(GridCachePartitionExchangeManager.java:144)
 at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:377)
 at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:335)
 at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3194)
 at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3173)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:100)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:294)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1568)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1196)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1092)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

Seems there is an issue with committing/rolling back transactions on topology changes because we expect all transactions holding topology locks to be finished at this time. Mvcc updates lock topology and produce remote txs at once, in other words existence of active remote txs implies pending updates and locked topology which cannot be true because of being finished exchange (all old txs are finished, all new txs are waiting for topology).

Need to be investigated.

Issue Links

• Is contained by: IGNITE-10468 MVCC TX: Failover
• is part of: IGNITE-10434 MVCC: Transaction asynchronous rollback bug.