[AMQNET-366] Deadlock when TCP connection breaks

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.2
• Fix Version/s: 1.5.3, 1.6.0
• Component/s: ActiveMQ
• Assignee: Timothy A. Bish
• Reporter: Mikael Finstad
• Created: 22/Dec/11 14:46
• Updated: 28/Dec/11 19:09
• Resolved: 28/Dec/11 19:06

Description

When an ActiveMQ TCP connection gets broken (for instance when the ActiveMQ server goes down), the AsyncCallExceptionListener is used by helper classes like SingleConnectionFactory and SimpleMessageListenerContainer from springframework.net to reconnect when connection becomes available again.
From this thread, these classes will call Close() on the connection which in turn calls Shutdown on the ThreadPoolExecutor (from the final block in Connection.cs:Close()).
Inside Shutdown(), the execution hangs on this.executionComplete.WaitOne(), and there is a deadlock because the AsyncCallExceptionListener call never returns, and nobody else triggers the executionComplete event

Simple test case to reproduce the problem:

using System;
using System.Threading;
using Apache.NMS.ActiveMQ;
using NUnit.Framework;
using Spring.Messaging.Nms.Listener;

[TestFixture]
public class ActiveMQTests
{
[Test]
[Ignore]
public void TestConnection()
{
var connectionFactory = new ConnectionFactory("tcp://localhost:61616")

{UserName = "qwe", Password = "asd"}

;

var messageListenerContainer = new SimpleMessageListenerContainer

{ ConnectionFactory = connectionFactory, DestinationName = "test.test.in" }

;

messageListenerContainer.Start();

Thread.Sleep(TimeSpan.FromMinutes(10));
}
}

Start debugging. Wait until it connects tothe ActiveMQ server, shutdown the ActiveMQ server, pause execution and look at the Threads.

Example call stack:

[In a sleep, wait, or join]
[External Code]
Apache.NMS.ActiveMQ.DLL!Apache.NMS.ActiveMQ.Threads.ThreadPoolExecutor.Shutdown() Line 121 + 0xe bytes
Apache.NMS.ActiveMQ.DLL!Apache.NMS.ActiveMQ.Connection.Close() Line 592 + 0xd bytes
Spring.Messaging.Nms.DLL!Spring.Messaging.Nms.Connections.ConnectionFactoryUtils.ReleaseConnection(Apache.NMS.IConnection connection, Apache.NMS.IConnectionFactory cf, bool started) Line 74 + 0x9 bytes
Spring.Messaging.Nms.DLL!Spring.Messaging.Nms.Listener.AbstractListenerContainer.RefreshSharedConnection() Line 432
Spring.Messaging.Nms.DLL!Spring.Messaging.Nms.Listener.SimpleMessageListenerContainer.RefreshConnectionUntilSuccessful() Line 242
Spring.Messaging.Nms.DLL!Spring.Messaging.Nms.Listener.SimpleMessageListenerContainer.OnException(System.Exception exception) Line 210
Apache.NMS.ActiveMQ.DLL!Apache.NMS.ActiveMQ.Connection.AsyncCallExceptionListener(object error) Line 945 + 0x11 bytes
Apache.NMS.ActiveMQ.DLL!Apache.NMS.ActiveMQ.Threads.ThreadPoolExecutor.Future.Run() Line 63 + 0x14 bytes
Apache.NMS.ActiveMQ.DLL!Apache.NMS.ActiveMQ.Threads.ThreadPoolExecutor.QueueProcessor(object unused) Line 150 + 0xa bytes
[External Code]

Should it not be possible to Close the connection from the AsyncCallExceptionListener thread?

I've tried running executor.Shutdown() in Connection.Close (under finally block) in a new thread and recompiling the NMS.ActiveMQ DLL, Like so:

if(executor != null)
{
var thread = new Thread(delegate()

{ executor.Shutdown(); }

);
thread.Start();
}

This seems to solve the problem, however I am not certain that this will not break anything else.

I am using Spring framework.net version 1.3.2 and apache.NMS version 1.5.0