[AMQ-5262] ActiveMQ hangs on shutdown when JMS Bridge is created

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 5.9.0, 5.9.1, 5.10.0
• Fix Version/s: 5.10.1, 5.11.0
• Component/s: Broker
• Assignee: Unassigned
• Reporter: Peter Minearo
• Created: 07/Jul/14 23:15
• Updated: 16/Dec/14 00:08
• Resolved: 08/Jul/14 20:21

Description

We are having a problem with ActiveMQ hanging on shutdown. Here is the scenario, we have a stand alone application that runs an embedded ActiveMQ which creates a JMS Queue Bridge via Spring configs. When we call shutdown, the TCPTransport that connects the JMS Queue Bridge does not shutdown, it hangs on java.net.SocketInputStream.socketRead0().

java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:129)
	at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
	at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:604)
	at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:58)
	at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:589)
	at java.io.DataInputStream.readInt(DataInputStream.java:370)
	at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:275)
	at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
	at java.lang.Thread.run(Thread.java:662)

Digging around on the forums and the issue tracker, the work around seems to add a parameter to the URI (Ex - tcp://localhost:60606?daemon=true).

According to this StackOverflow posting (http://stackoverflow.com/questions/2213340/what-is-daemon-thread-in-java) which quotes from Java Concurrency in Practice

• When a new thread is created it inherits the daemon status of its parent.
• Normal thread and daemon threads differ in what happens when they exit. When the JVM halts any remaining daemon threads are abandoned: *finally blocks are not executed*, stacks are not unwound - JVM just exits. Due to this reason daemon threads should be used sparingly and it is dangerous to use them for tasks that might perform any sort of I/O.

So, making the Socket that connects the JMS Queue Bridge a Daemon thread, seems to be the wrong solution.

I was trying to debug the initialization of ActiveMQ, and noticed the org.apache.activemq.network.jms.JmsConnector class has a stop() method on it. I believe this class creates the connection for the JMS Bridge, right? If so, the stop method does not seem to shutdown the connection properly.

public void stop() throws Exception {

        if (started.compareAndSet(true, false)) {

            ThreadPoolUtils.shutdown(connectionSerivce);
            connectionSerivce = null;

            for (DestinationBridge bridge : inboundBridges) {
                bridge.stop();
            }
            for (DestinationBridge bridge : outboundBridges) {
                bridge.stop();
            }
            LOG.info("JMS Connector {} stopped", getName());
        }
        
}

The question I have is why is the stop() method relying on the ThreadPoolUtils.shutdown(connectionSerivce) and NOT calling close() on the Connections first? For example:

public void stop() throws Exception {

        if (started.compareAndSet(true, false)) {

			foreignConnection.get().close();
			localConnection.get().close();
			
            ThreadPoolUtils.shutdown(connectionSerivce);
            connectionSerivce = null;

            for (DestinationBridge bridge : inboundBridges) {
                bridge.stop();
            }
            for (DestinationBridge bridge : outboundBridges) {
                bridge.stop();
            }
            LOG.info("JMS Connector {} stopped", getName());
        }
        
}

It was a little difficult to follow the code, so I may be missing something. BUT shouldn't the connections close first? Or am I looking in the wrong place.

I have created a small project that creates this scenario.
https://github.com/pminearo/activemq-shutdown-bug.git

This was done with ActiveMQ 5.9. Though, since this bug has been around for quite some time, it most likely is still in 5.10, 5.11, and 6.0.