Uploaded image for project: 'ActiveMQ .Net'
  1. ActiveMQ .Net
  2. AMQNET-487

Deadlock in Failover Transport when reconnecting

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.5.2, 1.6.2, 1.6.3
    • 1.6.4, 1.7.0
    • ActiveMQ
    • None
    • ActiveMQ 5.5.1 on Windows Server 2008 failover with networked brokers.
      Apache.NMS.ActiveMQ 1.6.2 running on .net 4.5/Windows 7

    • Patch Available

    Description

      Lock ordering issue with reconnectMutex and ((ICollection)connectionState.RecoveringPullConsumers).SyncRoot. Results in complete deadlock of NMS library.

      The two thread stacks do correctly take the reconnectMutex first but then in FailoverTransport.Oneway the reconnectMutex is Waited.

      The two stacks are:
      ========================================================================

      ntdll.dll!ZwWaitForMultipleObjects()
      KERNELBASE.dll!WaitForMultipleObjectsEx()
      kernel32.dll!WaitForMultipleObjectsExImplementation()
      [Managed to Native Transition]
      mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.Oneway(Apache.NMS.ActiveMQ.Commands.Command command)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.ConnectionInterruptProcessingComplete(Apache.NMS.ActiveMQ.Transport.ITransport transport, Apache.NMS.ActiveMQ.Commands.ConnectionId connectionId)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.ConnectionInterruptProcessingComplete(Apache.NMS.ActiveMQ.Commands.ConnectionId connectionId)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Connection.SignalInterruptionProcessingComplete()
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.MessageConsumer.ClearMessagesInProgress()
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Threads.TimerEx.TimerImpl.Run()
      mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
      mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
      mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
      mscorlib.dll!System.Threading.ThreadHelper.ThreadStart()
      [Native to Managed Transition]
      kernel32.dll!BaseThreadInitThunk()
      ntdll.dll!RtlUserThreadStart()

      ========================================================================

      ntdll.dll!ZwWaitForMultipleObjects()
      KERNELBASE.dll!WaitForMultipleObjectsEx()
      kernel32.dll!WaitForMultipleObjectsExImplementation()
      [Managed to Native Transition]
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.DoRestoreConsumers(Apache.NMS.ActiveMQ.Transport.ITransport transport, Apache.NMS.ActiveMQ.State.SessionState sessionState)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.DoRestoreSessions(Apache.NMS.ActiveMQ.Transport.ITransport transport, Apache.NMS.ActiveMQ.State.ConnectionState connectionState)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.DoRestore(Apache.NMS.ActiveMQ.Transport.ITransport transport)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.RestoreTransport(Apache.NMS.ActiveMQ.Transport.ITransport t)
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.DoConnect()
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.FailoverTask.Iterate()
      Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Threads.DedicatedTaskRunner.Run()
      mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
      mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
      mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
      mscorlib.dll!System.Threading.ThreadHelper.ThreadStart()
      [Native to Managed Transition]
      kernel32.dll!BaseThreadInitThunk()
      ntdll.dll!RtlUserThreadStart()

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tabish Timothy A. Bish
            lscpike Laurence Pike
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment