[QPIDJMS-255] AmqpProducer survives JmsConnection.close()

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.20.0
• Assignee: Unassigned
• Reporter: Milan Nikl
• Created: 18/Jan/17 06:46
• Updated: 09/Mar/17 14:52
• Resolved: 09/Mar/17 14:52

Description

Hi, I'm using ActiveMQ 5.14.3 with Qpid JMS client (0.20.0 RC maven artifact) in QueueBridge setup when local and remote queues are connected via JmsConnector. Some of my setup is described in QPIDJMS-230.

On my device I have an osgi bundle sending and receiving messages. ActiveMQ servers as persistence manager (KahaDB) and as entry point for the system, while Qpid JMS client as actual communication client for connection to MS Azure's IoT Hub.

This device suffers from quite common losses of network connection. In such case my ServerConnector thread which handles connection as a state machine tries to close my implementation of JmsConnector and clean all resources. Basically three different things can happen when closing connection:

1. There is no pending message at the moment, everything goes fine and is done instantly.
2. There is a pending message, network stays disconnected for a longer period of time. The foreign connection of JmsConnector is closed, the local connection close() gets stuck and cleanup is interrupted. After 120000 ms (my amqp.idleTimeout) Amqp finds an idle AmqpProvider and shuts it down.
3. There is a pending message, but network connection is renewed under 2 minutes. Then the IdleTimeoutCheck doesn't close the old AmqpProvider before my ServerConnector thread starts brand new JmsConnectionFactory as it tries to recover from connection failure.
   After 2 minutes the IdleTimeoutCheck finds there is probably something wrong with AmqpProvider. An exception is thrown and my thread closes current connection and tries to open a new one. There is one issue though because somehow two instances of my JmsConnector are somehow running. With some AmqpProvider instances working fine and some problematic ones. I can only hope that idle AmqpProviders are eliminated in background. But I still don't like the messy situation I caused by few connector restarts.

I'd like to point out that most of the mess is most probably caused by my code. There are few things I yet have to handle properly. E.g. I can't leave the cleaning thread running indefinitely because the device shutdown could suffer from this.

While I have 5 sec CloseTimeout on both ConnectionFactory instances and 2 sec SendTimeout, close() call on local connection still hangs longer. This might be some ActiveMQ problem as well.

The thing is there are still some AmqpProvider instances left behind running even though the JmsConnection instance which produced them is closed. Is it possible to (forcibly) clean all resources tied to JmsConnectionFactory? Or is this caused by something else and I'm getting the problem all wrong?

Thanks