Problem with message dispatch after a while

• Key: AMQ-2009
• Type: Bug
• Status: Reopened
• Priority: Major
• Assignee: Rob Davies
• Reporter: Rajani Chennamaneni
• Component/s: Broker
• Affects Version/s: 5.1.0, 5.2.0
• Fix Version/s: 5.4.0

Description

Messages are not getting dispatched after a while (although it accepts new incoming messages) until restart of the broker. This problem is described in several posts.

http://www.nabble.com/Pending-Messages-are-shown-in-ActiveMQ-td20241332.html

http://www.nabble.com/Consumer-Listener-stop-receving-message-until-ActiveMQ-restart-td20355247.html

http://www.nabble.com/Stuck-messages---Dispatch-issues-td20467949.html

There was also an issue opened in Spring project for this thinking it was Spring problem.

http://jira.springframework.org/browse/SPR-5110

I am not able to reproduce with Junit test case having BrokerService started with in the test case. I guess I am not hitting the right stress conditions this way. But when I run the test case against an externally running ActiveMQ instance backed with oracle database persistence, it is reproducible most of the times. This is not a every time failure situation, it takes more time once than the other.

I was able to hit this situation of stuck messages on queue using following scenario most of the times:

1) Start 2 concurrent consumers for the queue using Spring's DefaultMessageListenerContainer using cacheLevelName as CACHE_CONSUMER
2) Send messages using JMETER 2.3.2 to the queue on ActiveMQ stand alone broker instance with 50 threads looping 20 times.
3) After a while, you will notice that Spring logs that no messages are being received but the messages are shown jconsole of ActiveMQ and the database backing it for persistence.

But in 5.2 RC3, the problem is that it dispatches duplicate messages and does not remove them from broker's database after acknowledge properly.

Attached test case might help to reproduce when run against externally running stand alone ActiveMQ broker. Another way to see the problem is that try to load test using JMETER by sending messages to a queue with a camel route that moves messages from this queue to another and you will notice that it stops moving after while or copied duplicates in case of 5.2 RC3.

Sorry about such a huge description but it is a real problem! A different team at our company are having this issue in production with 5.1. They are using it as an embedded broker with derby for persistence.