[IGNITE-10899] Service Grid: disconnecting during node stop may lead to deadlock

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7
• Fix Version/s: 2.8
• Component/s: managed services
• Labels: MakeTeamcityGreenAgain
• Assignee: Vyacheslav Daradur
• Reporter: Vyacheslav Daradur
• Created: 11/Jan/19 14:46
• Updated: 16/Jan/19 17:31
• Resolved: 16/Jan/19 17:31

Description

In a rare case onDisconneced may be called during node stopping and deadlock may occur because of  ServiceDeploymentManage#stopProcessong blocks busyLock and not release it intentionally.

The issue has been found on TeamCity in Zookeeper's suite with the following stack trace:

disco-notifier-worker-#569118%client4%" 
 #609288
 prio=5 os_prio=0 tid=0x00007f905b440800 nid=0x3f6fbd sleeping[0x00007f9383efd000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.ignite.internal.util.GridSpinReadWriteLock.writeLock(GridSpinReadWriteLock.java:204)
at org.apache.ignite.internal.util.GridSpinBusyLock.block(GridSpinBusyLock.java:76)
at org.apache.ignite.internal.processors.service.ServiceDeploymentManager.stopProcessing(ServiceDeploymentManager.java:137)
at org.apache.ignite.internal.processors.service.IgniteServiceProcessor.stopProcessor(IgniteServiceProcessor.java:261)
at org.apache.ignite.internal.processors.service.IgniteServiceProcessor.onDisconnected(IgniteServiceProcessor.java:429)
at org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:4010)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:819)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:602)
 - locked <0x00000000f7ecdfa0> (a java.lang.Object)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4$$Lambda$25/2087171109.run(Unknown Source)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2696)
at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2734)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
at java.lang.Thread.run(Thread.java:748)

Issue Links

• links to: GitHub Pull Request #5812
• links to: GitHub Pull Request #5812
• links to: PR Tests | RunAll
• links to: PR Tests | ZooKeeper (Discovery) 2