Details
Major Description
I am seeing a problem in using ActiveMQ5.0.0 (AMQ Message Store) with Spring's DefaultMessageListenerContainer. Whenever we set the DefaultMessageListenerContainer currentConsumer value to more than 1 (i.e. multiple concurrent consumers), it causes activeMQ broker to produce error shown below:
ERROR RecoveryListenerAdapter - Message id xxxxxx could not be recovered from the data store!
After this point, no further messages will be dispatched from the ActiveMQ JMS queue, and new messages send to the JMS queue seems to disappear and are not logged into the persistence store. The error seem to indicate problem with the persistence store. We are making use of the new activemq5.0.0 AMQ Message Store.
I have included a test program (packaged in TestCase.zip) that will re-produce the issue. The testcase contains the following content:
- activemq.xml : contains the config to start the ActiveMQ broker used by the test program. It contains a basic stripped down configuration (using default config settings on AMQ Message Store).
- JmsQueueSender.java : util class making use of Spring's JmsTemplate to send JMS messages.
- ActiveMQTestClass.java : main class that needs to be started. It loads the sender class via spring, and also starts up the Spring's DefaultMessageListenerContainer to dispatch messages. It will use the sender class JmsQueueSender to send 100 messages every 10sec.
- DefaultMessageListener.java : JMS Pojo class that will be called to handle the messages dispatched from the configured Spring's DefaultMessageListenerContainer class. It currently just prints the message content and sleeps for 200 millisec. The sleep is intentional, as it simulate the time it takes in our real application to process the message content. It also somehow (along with the printing of the message content) seem to make the error/program to occur more quickly.
- testSpring.xml : Spring configuration used by the test program. One main thing to note is the DefaultMessageListenerContainer class is configured with 10 concurrentConsumers. When setting this value to 1, the problem doesn't seem to happen, but as soon as this value is set beyond 1, it causes problem in the ActiveMQ broker, relating to problem in unable to recover from datastore.
I have included the ActiveMQ broker logfile (activemq.log) which has debug turned on (to provide more info). As soon as the error shown below is logged, the consumer is unable to dispatch any more messages from the ActiveMQ queue. The errors reported are:
2008-01-10 16:37:24,474 [/127.0.0.1:3432] ERROR RecoveryListenerAdapter - Message id ID:B00118579B7DA-3430-1199983032693-0:1:1:1:99 could not be recovered from the data store!
2008-01-10 16:37:24,474 [/127.0.0.1:3432] ERROR RecoveryListenerAdapter - Message id ID:B00118579B7DA-3430-1199983032693-0:1:1:1:99 could not be recovered from the data store!
2008-01-10 16:37:24,474 [/127.0.0.1:3432] ERROR RecoveryListenerAdapter - Message id ID:B00118579B7DA-3430-1199983032693-0:1:1:1:99 could not be recovered from the data store!
The error doesn't seem to happen at predictable fixed points, which is why the test program will send 100 message every 10sec, and over a number of iterations, it will eventually show the problem in the broker.
Let me know if further information is required. I would really like to get the concurrent consumer to dispatch the messages, in order to reduce the time it takes to process multiple burst of messages.