Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-10899

Service Grid: disconnecting during node stop may lead to deadlock

    XMLWordPrintableJSON

    Details

      Description

      In a rare case onDisconneced may be called during node stopping and deadlock may occur because of  ServiceDeploymentManage#stopProcessong blocks busyLock and not release it intentionally.

      The issue has been found on TeamCity in Zookeeper's suite with the following stack trace:

      disco-notifier-worker-#569118%client4%" 
       #609288
       prio=5 os_prio=0 tid=0x00007f905b440800 nid=0x3f6fbd sleeping[0x00007f9383efd000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
      at java.lang.Thread.sleep(Native Method)
      at org.apache.ignite.internal.util.GridSpinReadWriteLock.writeLock(GridSpinReadWriteLock.java:204)
      at org.apache.ignite.internal.util.GridSpinBusyLock.block(GridSpinBusyLock.java:76)
      at org.apache.ignite.internal.processors.service.ServiceDeploymentManager.stopProcessing(ServiceDeploymentManager.java:137)
      at org.apache.ignite.internal.processors.service.IgniteServiceProcessor.stopProcessor(IgniteServiceProcessor.java:261)
      at org.apache.ignite.internal.processors.service.IgniteServiceProcessor.onDisconnected(IgniteServiceProcessor.java:429)
      at org.apache.ignite.internal.IgniteKernal.onDisconnected(IgniteKernal.java:4010)
      at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.onDiscovery0(GridDiscoveryManager.java:819)
      at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4.lambda$onDiscovery$0(GridDiscoveryManager.java:602)
       - locked <0x00000000f7ecdfa0> (a java.lang.Object)
      at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$4$$Lambda$25/2087171109.run(Unknown Source)
      at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body0(GridDiscoveryManager.java:2696)
      at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryMessageNotifierWorker.body(GridDiscoveryManager.java:2734)
      at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
      at java.lang.Thread.run(Thread.java:748)
      

        Attachments

          Activity

            People

            • Assignee:
              daradurvs Vyacheslav Daradur
              Reporter:
              daradurvs Vyacheslav Daradur
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 10m
                10m