Uploaded image for project: 'ActiveMQ .Net'
  1. ActiveMQ .Net
  2. AMQNET-487

Deadlock in Failover Transport when reconnecting

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.5.2, 1.6.2, 1.6.3
    • Fix Version/s: 1.6.4, 1.7.0
    • Component/s: ActiveMQ
    • Labels:
      None
    • Environment:

      ActiveMQ 5.5.1 on Windows Server 2008 failover with networked brokers.
      Apache.NMS.ActiveMQ 1.6.2 running on .net 4.5/Windows 7

    • Patch Info:
      Patch Available

      Description

      Lock ordering issue with reconnectMutex and ((ICollection)connectionState.RecoveringPullConsumers).SyncRoot. Results in complete deadlock of NMS library.

      The two thread stacks do correctly take the reconnectMutex first but then in FailoverTransport.Oneway the reconnectMutex is Waited.

      The two stacks are:
      ========================================================================

      ntdll.dll!ZwWaitForMultipleObjects()
      KERNELBASE.dll!WaitForMultipleObjectsEx()
      kernel32.dll!WaitForMultipleObjectsExImplementation()
      [Managed to Native Transition]
      mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.Oneway(Apache.NMS.ActiveMQ.Commands.Command command)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.ConnectionInterruptProcessingComplete(Apache.NMS.ActiveMQ.Transport.ITransport transport, Apache.NMS.ActiveMQ.Commands.ConnectionId connectionId)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.ConnectionInterruptProcessingComplete(Apache.NMS.ActiveMQ.Commands.ConnectionId connectionId)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Connection.SignalInterruptionProcessingComplete()
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.MessageConsumer.ClearMessagesInProgress()
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Threads.TimerEx.TimerImpl.Run()
      mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
      mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
      mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
      mscorlib.dll!System.Threading.ThreadHelper.ThreadStart()
      [Native to Managed Transition]
      kernel32.dll!BaseThreadInitThunk()
      ntdll.dll!RtlUserThreadStart()

      ========================================================================

      ntdll.dll!ZwWaitForMultipleObjects()
      KERNELBASE.dll!WaitForMultipleObjectsEx()
      kernel32.dll!WaitForMultipleObjectsExImplementation()
      [Managed to Native Transition]
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.DoRestoreConsumers(Apache.NMS.ActiveMQ.Transport.ITransport transport, Apache.NMS.ActiveMQ.State.SessionState sessionState)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.DoRestoreSessions(Apache.NMS.ActiveMQ.Transport.ITransport transport, Apache.NMS.ActiveMQ.State.ConnectionState connectionState)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.DoRestore(Apache.NMS.ActiveMQ.Transport.ITransport transport)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.RestoreTransport(Apache.NMS.ActiveMQ.Transport.ITransport t)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.DoConnect()
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.FailoverTask.Iterate()
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Threads.DedicatedTaskRunner.Run()
      mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
      mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
      mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
      mscorlib.dll!System.Threading.ThreadHelper.ThreadStart()
      [Native to Managed Transition]
      kernel32.dll!BaseThreadInitThunk()
      ntdll.dll!RtlUserThreadStart()

        Attachments

          Activity

            People

            • Assignee:
              tabish Timothy A. Bish
              Reporter:
              lscpike Laurence Pike
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: