[IGNITE-5527] GridDeferredAckMessageSender can cause starvation in stripe pool on unstable topology.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1
• Component/s: clients, general
• Assignee: Andrey Mashenkov
• Reporter: Andrey Mashenkov
• Created: 16/Jun/17 16:50
• Updated: 26/Jul/17 09:07
• Resolved: 10/Jul/17 08:10

Description

Slow or failed client can cause starvation in stripe-pool.

GridDeferredAckMessageSender.sendDeferredAckMessage() can stucks when add message to buffer while other thread trying to send this buffer.

Seems, we shouldn't wait for readLock in DeferredAckMessageBuffer.add() method and return false immediately.

Stacktrace:

Thread name="sys-stripe-3-#4%null%", id=16, state=WAITING, blockCnt=3087, waitCnt=204846
Lock [object=java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync@54ef60c9, ownerName=sys-#12106%null%, ownerId=12326]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
at o.a.i.i.processors.cache.GridDeferredAckMessageSender$DeferredAckMessageBuffer.add(GridDeferredAckMessageSender.java:176)
at o.a.i.i.processors.cache.GridDeferredAckMessageSender.sendDeferredAckMessage(GridDeferredAckMessageSender.java:100)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.sendDeferredUpdateResponse(GridDhtAtomicCache.java:3371)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processDhtAtomicUpdateRequest(GridDhtAtomicCache.java:3353)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1800(GridDhtAtomicCache.java:130)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$8.apply(GridDhtAtomicCache.java:379)
at o.a.i.i.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$8.apply(GridDhtAtomicCache.java:374)
at o.a.i.i.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:827)
at o.a.i.i.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:369)
at o.a.i.i.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:293)
at o.a.i.i.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:95)
at o.a.i.i.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:238)
at o.a.i.i.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1217)
at o.a.i.i.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:845)
at o.a.i.i.managers.communication.GridIoManager.access$2100(GridIoManager.java:109)
at o.a.i.i.managers.communication.GridIoManager$6.run(GridIoManager.java:785)
at o.a.i.i.util.StripedExecutor$Stripe.run(StripedExecutor.java:428)
at java.lang.Thread.run(Thread.java:745)

Issue Links

• links to: GitHub Pull Request #2150