[IGNITE-1977] IgniteSemaphore's failover related tests lead to the deadlock or fail

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.8, 1.9
• Fix Version/s: 2.0
• Component/s: data structures
• Assignee: Vladisav Jelisavcic
• Reporter: Denis A. Magda
• Created: 20/Nov/15 15:53
• Updated: 01/Aug/19 21:51
• Resolved: 11/Apr/17 11:10

Description

All IgniteSemaphore related tests from GridCacheAbstractDataStructuresFailoverSelfTest may cause a deadlock which leads to the whole suite hanging.

The threads are waiting for the following condition:

"topology-change-thread-3" prio=6 tid=0x000000001d98d800 nid=0x2b20 waiting on condition [0x000000002066f000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000798149948> (a org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
	at org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl.acquire(GridCacheSemaphoreImpl.java:538)
	at org.apache.ignite.internal.processors.datastructures.GridCacheSemaphoreImpl.acquire(GridCacheSemaphoreImpl.java:525)
	at org.apache.ignite.internal.processors.cache.datastructures.GridCacheAbstractDataStructuresFailoverSelfTest$7.apply(GridCacheAbstractDataStructuresFailoverSelfTest.java:571)
	at org.apache.ignite.internal.util.lang.GridAbsClosure.run(GridAbsClosure.java:50)
	at org.apache.ignite.testframework.GridTestUtils$7.call(GridTestUtils.java:967)
	at org.apache.ignite.testframework.GridTestThread.run(GridTestThread.java:86)

Probably the semaphore is not properly released when a node leaves the topology abruptly.

In addition the tests should be rewritten to the way which is followed by other data structures and atomics from this suite: using ConstantTopologyChangeWorker and its descendants.

Issue Links

• is related to: IGNITE-4173 IgniteSemaphore with failoverSafe enabled doesn't release permits in case permits owner node left topology
• is related to: IGNITE-4209 Semaphore gets broken when node finishes normally and leaves topology
• links to: GitHub Pull Request #1751
• links to: GitHub Pull Request #1751