[IGNITE-15777] Specific issues when transaction stuck

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.13
• Labels: ise.lts
• Release Note: Fixed PME hanging on client due to implicit transaction committing.
• Ignite Flags: Release Notes Required
• Assignee: Vladislav Pyatkov
• Reporter: Vladislav Pyatkov
• Reviewer: Slava Koptilin
• Created: 19/Oct/21 16:36
• Updated: 13/May/22 09:11
• Resolved: 21/Oct/21 18:07

Description

1) Exchange hangs on client node, because implicit transaction stay in the deadlock.

sys-#111494%GRID%" #139376 prio=5 os_prio=0 cpu=838.82ms elapsed=2482.84s allocated=165M defined_classes=0 tid=0x00007fe8ac30d000 nid=0x9b19 sleeping [0x00007fe892eec000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(java.base@11.0.10/Native Method)
    at org.apache.ignite.internal.util.IgniteUtils.sleep(IgniteUtils.java:8156)
    at org.apache.ignite.internal.processors.cache.GridCacheGateway.onStopped(GridCacheGateway.java:323)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.stopGateway(GridCacheProcessor.java:2601)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.lambda$processCacheStopRequestOnExchangeDone$629e8679$1(GridCacheProcessor.java:2797)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor$$Lambda$1602/0x0000000840a49040.apply(Unknown Source)
    at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11572)
    at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11474)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.processCacheStopRequestOnExchangeDone(GridCacheProcessor.java:2776)
    at org.apache.ignite.internal.processors.cache.GridCacheProcessor.onExchangeDone(GridCacheProcessor.java:2906)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onDone(GridDhtPartitionsExchangeFuture.java:2514)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processFullMessage(GridDhtPartitionsExchangeFuture.java:4765)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$1500(GridDhtPartitionsExchangeFuture.java:160)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$4.apply(GridDhtPartitionsExchangeFuture.java:4434)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$4.apply(GridDhtPartitionsExchangeFuture.java:4422)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:407)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:362)
    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceiveFullMessage(GridDhtPartitionsExchangeFuture.java:4422)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processFullPartitionUpdate(GridCachePartitionExchangeManager.java:2017)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$3.onMessage(GridCachePartitionExchangeManager.java:472)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$3.onMessage(GridCachePartitionExchangeManager.java:459)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3875)
    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3854)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
    at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
    at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1719)
    at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1326)
    at org.apache.ignite.internal.managers.communication.GridIoManager.access$4600(GridIoManager.java:157)
    at org.apache.ignite.internal.managers.communication.GridIoManager$8.execute(GridIoManager.java:1211)
    at org.apache.ignite.internal.managers.communication.TraceRunnable.run(TraceRunnable.java:54)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.10/ThreadPoolExecutor.java:1128)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.10/ThreadPoolExecutor.java:628)
    at java.lang.Thread.run(java.base@11.0.10/Thread.java:834)
	
"http-nio-8080-exec-13" #27351 daemon prio=5 os_prio=0 cpu=958477.38ms elapsed=204274.71s allocated=261G defined_classes=126 tid=0x00007fe5dc009800 nid=0x220 waiting on condition  [0x00007fe95b154000]
   java.lang.Thread.State: WAITING (parking)
    at jdk.internal.misc.Unsafe.park(java.base@11.0.10/Native Method)
    at java.util.concurrent.locks.LockSupport.park(java.base@11.0.10/LockSupport.java:323)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:186)
    at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:149)
    at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.op(GridCacheAdapter.java:2096)
    at org.apache.ignite.internal.processors.cache.GridCacheAdapter$16.op(GridCacheAdapter.java:2094)
    at org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:3878)
    at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2094)
    at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2074)
    at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2049)
    at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1310)
    at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:817)
    ....

2) Transaction hangs on recovery, because its timeout is exceeded in the same time.

[2021-09-08T12:03:01,418][INFO ][sys-stripe-2-#3][IgniteTxManager] Finishing prepared transaction [commit=true, tx=GridDhtTxLocal [nearNodeId=cfe23f0e-d878-4bc4-b109-59c8f24ddf96, nearFutId=99f3e32cb71-750eb39d-e776-4732-a8a0-0d86a9f0636a, nearMiniId=2, nearFinFutId=null, nearFinMiniId=0, nearXidVer=GridCacheVersion [topVer=242527905, order=1631058338043, nodeOrder=27],..., nodeId=32123b8d-1750-4e11-bb84-88bba692a6ff, isolation=SERIALIZABLE, concurrency=OPTIMISTIC, timeout=60000, sysInvalidate=false, sys=false, plc=2, commitVer=null, finalizing=RECOVERY_FINISH, invalidParts=null, state=PREPARED, timedOut=false, topVer=AffinityTopologyVersion [topVer=35, minorTopVer=0], mvccSnapshot=null, skipCompletedVers=false, parentTx=null, duration=44291ms, onePhaseCommit=false], size=1]]]]

Issue Links

• links to: GitHub Pull Request #9502