[IGNITE-9675] Deadlock on Ignite:active() and stopping grid simultaneously calling

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8
• Labels: MakeTeamcityGreenAgain
• Assignee: Alexey Platonov
• Reporter: Alexey Platonov
• Created: 24/Sep/18 15:04
• Updated: 05/Oct/18 09:49
• Resolved: 05/Oct/18 09:46

Description

There is deadlock on Ignite:active() and stopping grid simultaneously calling

1. Trying to stop client node.

"main-ScalaTest-running-VisorInProcDriverSpec" #1 prio=5 os_prio=0 tid=0x00007f267800e800 nid=0x6574 sleeping[0x00007f2681b7e000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.ignite.internal.util.GridSpinReadWriteLock.writeLock(GridSpinReadWriteLock.java:206)
    at org.apache.ignite.internal.processors.task.GridTaskProcessor.onKernalStop(GridTaskProcessor.java:190)
    at org.apache.ignite.internal.IgniteKernal.stop0(IgniteKernal.java:2135)
    at org.apache.ignite.internal.IgniteKernal.stop(IgniteKernal.java:2083)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop0(IgnitionEx.java:2590)
    - locked <0x0000000797ecf7f8> (a org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.stop(IgnitionEx.java:2553)
    at org.apache.ignite.internal.IgnitionEx.stop(IgnitionEx.java:374)
    at org.apache.ignite.Ignition.stop(Ignition.java:225)

 

and
2. Execute task that tries to get ignite.active() state, which also executes task GridClusterStateProcessor.sendComputeCheckGlobalState(GridClusterStateProcessor.java:1086)

"mgmt-#2470" #2965 prio=5 os_prio=0 tid=0x00007f23fc001000 nid=0x730b waiting on condition [0x00007f221f0ee000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
	at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
	at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
	at org.apache.ignite.internal.AsyncSupportAdapter.saveOrGet(AsyncSupportAdapter.java:112)
	at org.apache.ignite.internal.IgniteComputeImpl.call(IgniteComputeImpl.java:786)
	at org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.sendComputeCheckGlobalState(GridClusterStateProcessor.java:1086)
	at org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.publicApiActiveState(GridClusterStateProcessor.java:177)
	at org.apache.ignite.internal.cluster.IgniteClusterImpl.active(IgniteClusterImpl.java:300)
	at org.apache.ignite.internal.visor.node.VisorNodeDataCollectorTask.reduce(VisorNodeDataCollectorTask.java:75)
	at org.gridgain.grid.internal.visor.node.VisorGridGainNodeDataCollectorTask.reduce0(VisorGridGainNodeDataCollectorTask.java:39)
	at org.gridgain.grid.internal.visor.node.VisorGridGainNodeDataCollectorTask.reduce0(VisorGridGainNodeDataCollectorTask.java:26)
	at org.apache.ignite.internal.visor.VisorMultiNodeTask.reduce(VisorMultiNodeTask.java:139)
	at org.apache.ignite.internal.processors.task.GridTaskWorker$6.call(GridTaskWorker.java:1139)
	at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6765)
	at org.apache.ignite.internal.processors.task.GridTaskWorker.reduce(GridTaskWorker.java:1137)
	at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:964)
	at org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:1081)
	at org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1316)
	at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
	at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
	at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
	at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Issue Links

• links to: GitHub Pull Request #4822
• links to: TC Run ALL
• links to: Upsource review