[IGNITE-8435] Inconsistent data after StopNodeFailureHandler handles process termination

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.5
• Assignee: Anton Kalashnikov
• Reporter: Anton Kalashnikov
• Created: 04/May/18 08:54
• Updated: 01/Aug/19 22:13
• Resolved: 08/May/18 14:08

Description

Steps to reproduce:

1. Cluster started with StopNodeFailureHandler in config.
2. Upload data to the caches and run accounts task (transfer the amount from one account to another for some cache).
3. Terminate one thread (ttl-cleanup-worker for example for one node).
4. Wait until accounts task finished the work.
5. Check the sum for caches.
Expected result:
The sum is the same as before thread rermination.
Actual result:
The sum is not the same for some caches.

Also, I see these exception in the logs for node with terminated thread:

[13:08:37,709][SEVERE][sys-stripe-1-#2][GridNearTxLocal] Commit failed.
class org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException: Commit produced a runtime exception (all transaction entries will be invalidated): GridDhtTxRemote[xid=8e4906dd261-00000000-0815-47fa-0000-000000000004, xidVersion=GridCacheVersion [topVer=135612410, order=1524132517096, nod
eOrder=4], concurrency=OPTIMISTIC, isolation=REPEATABLE_READ, state=COMMITTING, invalidate=false, rollbackOnly=false, nodeId=6f5954fa-e283-4506-9344-7a8914774681, timeout=4990, duration=41]
 at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:742)
 at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:815)
 at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1316)
 at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxFinishRequest(IgniteTxHandler.java:1228)
 at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$600(IgniteTxHandler.java:97)
 at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:213)
 at org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$7.apply(IgniteTxHandler.java:211)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1054)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
 at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
 at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556)
 at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184)
 at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:125)
 at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1091)
 at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:511)
 at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Runtime failure on search row: org.apache.ignite.internal.processors.cache.tree.SearchRow@1aa35c0b
 at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.invoke(BPlusTree.java:1668)
 at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:1263)
 at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1782)
 at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:359)
 at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3604)
 at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3580)
 at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerSet(GridCacheMapEntry.java:1040)
 at org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:611)
 ... 18 more

 

Issue Links

• links to: GitHub Pull Request #3944
• links to: GitHub Pull Request #3944