[IGNITE-14078] Deadlock on GridCacheSharedTtlCleanupManager#mgrs if cache is created when ttl cleanup is running

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.1
• Fix Version/s: 2.10
• Assignee: Mirza Aliev
• Reporter: Mirza Aliev
• Created: 28/Jan/21 10:02
• Updated: 24/Feb/21 16:55
• Resolved: 15/Feb/21 19:10

Description

ttl-cleanup-worker does a block of work in ConcurrentHashMap.compute() and tries to acquire checkpoint read lock:

Thread [name="ttl-cleanup-worker-#120%1%", id=225, state=WAITING, blockCnt=0, waitCnt=81486]
    Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@35608c45, ownerName=null, ownerId=-1]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at o.a.i.i.processors.cache.persistence.GridCacheDatabaseSharedManager.checkpointReadLock(GridCacheDatabaseSharedManager.java:1730)
        at o.a.i.i.processors.cache.IgniteCacheOffheapManagerImpl.expireInternal(IgniteCacheOffheapManagerImpl.java:1346)
        at o.a.i.i.processors.cache.IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:1323)
        at o.a.i.i.processors.cache.GridCacheTtlManager.expire(GridCacheTtlManager.java:242)
        at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.lambda$body$0(GridCacheSharedTtlCleanupManager.java:178)
        at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker$$Lambda$619/1960552474.apply(Unknown Source)
        at java.util.concurrent.ConcurrentHashMap.computeIfPresent(ConcurrentHashMap.java:1769)
        - locked java.util.concurrent.ConcurrentHashMap$Node@4f66c754
        at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:177)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:119)
        at java.lang.Thread.run(Thread.java:748)

Meanwhile, exchange thread is waiting on the same ConcurrentHashMap node:

Thread [name="exchange-worker-#93%1%", id=193, state=BLOCKED, blockCnt=8, waitCnt=1669]
    Lock [object=java.util.concurrent.ConcurrentHashMap$Node@4f66c754, ownerName=ttl-cleanup-worker-#120%1%, ownerId=225]
        at java.util.concurrent.ConcurrentHashMap.transfer(ConcurrentHashMap.java:2426)
        at java.util.concurrent.ConcurrentHashMap.addCount(ConcurrentHashMap.java:2288)
        at java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1070)
        at java.util.concurrent.ConcurrentHashMap.put(ConcurrentHashMap.java:1006)
        at o.a.i.i.processors.cache.GridCacheSharedTtlCleanupManager.register(GridCacheSharedTtlCleanupManager.java:68)
        at o.a.i.i.processors.cache.GridCacheTtlManager.start0(GridCacheTtlManager.java:107)
        at o.a.i.i.processors.cache.GridCacheManagerAdapter.start(GridCacheManagerAdapter.java:49)
        at o.a.i.i.processors.cache.GridCacheProcessor.initCacheContext(GridCacheProcessor.java:2176)
        at o.a.i.i.processors.cache.GridCacheProcessor.prepareCacheContext(GridCacheProcessor.java:1964)
        at o.a.i.i.processors.cache.GridCacheProcessor.prepareCacheStart(GridCacheProcessor.java:1883)
        at o.a.i.i.processors.cache.GridCacheProcessor.lambda$prepareStartCaches$55a0e703$1(GridCacheProcessor.java:1758)
        at o.a.i.i.processors.cache.GridCacheProcessor$$Lambda$527/649205444.apply(Unknown Source)
        at o.a.i.i.processors.cache.GridCacheProcessor.lambda$prepareStartCachesIfPossible$14(GridCacheProcessor.java:1728)
        at o.a.i.i.processors.cache.GridCacheProcessor$$Lambda$526/1117407359.handle(Unknown Source)
        at o.a.i.i.processors.cache.GridCacheProcessor.prepareStartCaches(GridCacheProcessor.java:1755)
        at o.a.i.i.processors.cache.GridCacheProcessor.prepareStartCachesIfPossible(GridCacheProcessor.java:1726)
        at o.a.i.i.processors.cache.CacheAffinitySharedManager.processCacheStartRequests(CacheAffinitySharedManager.java:1005)
        at o.a.i.i.processors.cache.CacheAffinitySharedManager.onCacheChangeRequest(CacheAffinitySharedManager.java:891)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onCacheChangeRequest(GridDhtPartitionsExchangeFuture.java:1459)
        at o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:967)
        at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3376)
        at o.a.i.i.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3195)
        at o.a.i.i.util.worker.GridWorker.run(GridWorker.java:119)
        at java.lang.Thread.run(Thread.java:748)

Issue Links

• links to: GitHub Pull Request #8718