[IGNITE-7324] AssertionError Entry cannot become obsolete when DHT local candidate is added in deadlock detection suite

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3
• Fix Version/s: 2.5
• Labels: MakeTeamcityGreenAgain
• Assignee: Alexey Scherbakov
• Reporter: Alexey Goncharuk
• Created: 28/Dec/17 11:09
• Updated: 26/Apr/18 08:46
• Resolved: 26/Apr/18 08:46

Description

I observed this assertion in deadlock detection suite on TC:

[12:11:30]W:		 [org.apache.ignite:ignite-core] java.lang.AssertionError: Entry cannot become obsolete when DHT local candidate is added [e=GridDhtCacheEntry [rdrs=[], part=39, super=GridDistributedCacheEntry [super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=39, val=39, hasValBytes=true], val=null, startVer=1514452286919, ver=GridCacheVersion [topVer=125932251, order=1514452286919, nodeOrder=2], hash=39, extras=GridCacheObsoleteEntryExtras [obsoleteVer=GridCacheVersion [topVer=2147483647, order=0, nodeOrder=0]], flags=2]]], ex=org.apache.ignite.internal.processors.cache.GridCacheEntryRemovedException]
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.map(GridDhtLockFuture.java:902)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtLockFuture.onOwnerChanged(GridDhtLockFuture.java:655)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccManager.notifyOwnerChanged(GridCacheMvccManager.java:226)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccManager.access$200(GridCacheMvccManager.java:80)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMvccManager$3.onOwnerChanged(GridCacheMvccManager.java:163)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.checkOwnerChanged(GridCacheMapEntry.java:3709)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheEntry.removeLock(GridDistributedCacheEntry.java:400)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.removeLock(GridDhtCacheEntry.java:329)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.GridDistributedCacheEntry.txUnlock(GridDistributedCacheEntry.java:665)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.unlockMultiple(IgniteTxManager.java:1672)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxManager.rollbackTx(IgniteTxManager.java:1287)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.userRollback(IgniteTxLocalAdapter.java:980)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.localFinish(GridNearTxLocal.java:3080)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.finish(GridNearTxFinishFuture.java:418)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.rollbackNearTxLocalAsync(GridNearTxLocal.java:3316)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.access$4900(GridNearTxLocal.java:122)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$26.run(GridNearTxLocal.java:4129)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6673)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[12:11:30]W:		 [org.apache.ignite:ignite-core] 	at java.lang.Thread.run(Thread.java:745)

It looks like the lock future has acquired one lock and was waiting for another lock. When the future timed-out, the previous lock was already removed, but the next lock was successfully acquired, which led to the assertion above.

My suggestion would be to give up in the mapping code if the future has timed out or if we got entry removed exception

Issue Links

• relates to: IGNITE-6827 Configurable rollback for long running transactions before partition exchange