[AMQNET-487] Deadlock in Failover Transport when reconnecting

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.5.2, 1.6.2, 1.6.3
• Fix Version/s: 1.6.4, 1.7.0
• Component/s: ActiveMQ

Environment

ActiveMQ 5.5.1 on Windows Server 2008 failover with networked brokers.
Apache.NMS.ActiveMQ 1.6.2 running on .net 4.5/Windows 7

• Patch Info: Patch Available
• Assignee: Timothy A. Bish
• Reporter: Laurence Pike
• Created: 07/Aug/14 13:39
• Updated: 11/Aug/14 17:58
• Resolved: 11/Aug/14 17:57

Description

Lock ordering issue with reconnectMutex and ((ICollection)connectionState.RecoveringPullConsumers).SyncRoot. Results in complete deadlock of NMS library.

The two thread stacks do correctly take the reconnectMutex first but then in FailoverTransport.Oneway the reconnectMutex is Waited.

The two stacks are:
========================================================================

ntdll.dll!ZwWaitForMultipleObjects()
KERNELBASE.dll!WaitForMultipleObjectsEx()
kernel32.dll!WaitForMultipleObjectsExImplementation()
[Managed to Native Transition]
mscorlib.dll!System.Threading.Monitor.Wait(object obj, int millisecondsTimeout)
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.Oneway(Apache.NMS.ActiveMQ.Commands.Command command)
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.ConnectionInterruptProcessingComplete(Apache.NMS.ActiveMQ.Transport.ITransport transport, Apache.NMS.ActiveMQ.Commands.ConnectionId connectionId)
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.ConnectionInterruptProcessingComplete(Apache.NMS.ActiveMQ.Commands.ConnectionId connectionId)
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Connection.SignalInterruptionProcessingComplete()
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.MessageConsumer.ClearMessagesInProgress()
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Threads.TimerEx.TimerImpl.Run()
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart()
[Native to Managed Transition]
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()

========================================================================

ntdll.dll!ZwWaitForMultipleObjects()
KERNELBASE.dll!WaitForMultipleObjectsEx()
kernel32.dll!WaitForMultipleObjectsExImplementation()
[Managed to Native Transition]
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.DoRestoreConsumers(Apache.NMS.ActiveMQ.Transport.ITransport transport, Apache.NMS.ActiveMQ.State.SessionState sessionState)
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.DoRestoreSessions(Apache.NMS.ActiveMQ.Transport.ITransport transport, Apache.NMS.ActiveMQ.State.ConnectionState connectionState)
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.State.ConnectionStateTracker.DoRestore(Apache.NMS.ActiveMQ.Transport.ITransport transport)
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.RestoreTransport(Apache.NMS.ActiveMQ.Transport.ITransport t)
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.DoConnect()
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransport.FailoverTask.Iterate()
Apache.NMS.ActiveMQ.dll!Apache.NMS.ActiveMQ.Threads.DedicatedTaskRunner.Run()
mscorlib.dll!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state, bool preserveSyncCtx)
mscorlib.dll!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, object state)
mscorlib.dll!System.Threading.ThreadHelper.ThreadStart()
[Native to Managed Transition]
kernel32.dll!BaseThreadInitThunk()
ntdll.dll!RtlUserThreadStart()