ActiveMQ
  1. ActiveMQ
  2. AMQ-1445 RecoveryListenerAdapter - Message id ID... could not be recovered from the data store!
  3. AMQ-1544

ERROR RecoveryListenerAdapter - Message id xxxxxx could not be recovered from the data store! (when using Spring's DefaultMessageListenerContainer with more than 1 concurrentConsumers to dispatch JMS messages)

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.0.0
    • Fix Version/s: 5.1.0
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      Windows XP and Sun Solaris on x86

      Description

      I am seeing a problem in using ActiveMQ5.0.0 (AMQ Message Store) with Spring's DefaultMessageListenerContainer. Whenever we set the DefaultMessageListenerContainer currentConsumer value to more than 1 (i.e. multiple concurrent consumers), it causes activeMQ broker to produce error shown below:

      ERROR RecoveryListenerAdapter - Message id xxxxxx could not be recovered from the data store!

      After this point, no further messages will be dispatched from the ActiveMQ JMS queue, and new messages send to the JMS queue seems to disappear and are not logged into the persistence store. The error seem to indicate problem with the persistence store. We are making use of the new activemq5.0.0 AMQ Message Store.

      I have included a test program (packaged in TestCase.zip) that will re-produce the issue. The testcase contains the following content:

      • activemq.xml : contains the config to start the ActiveMQ broker used by the test program. It contains a basic stripped down configuration (using default config settings on AMQ Message Store).
      • JmsQueueSender.java : util class making use of Spring's JmsTemplate to send JMS messages.
      • ActiveMQTestClass.java : main class that needs to be started. It loads the sender class via spring, and also starts up the Spring's DefaultMessageListenerContainer to dispatch messages. It will use the sender class JmsQueueSender to send 100 messages every 10sec.
      • DefaultMessageListener.java : JMS Pojo class that will be called to handle the messages dispatched from the configured Spring's DefaultMessageListenerContainer class. It currently just prints the message content and sleeps for 200 millisec. The sleep is intentional, as it simulate the time it takes in our real application to process the message content. It also somehow (along with the printing of the message content) seem to make the error/program to occur more quickly.
      • testSpring.xml : Spring configuration used by the test program. One main thing to note is the DefaultMessageListenerContainer class is configured with 10 concurrentConsumers. When setting this value to 1, the problem doesn't seem to happen, but as soon as this value is set beyond 1, it causes problem in the ActiveMQ broker, relating to problem in unable to recover from datastore.

      I have included the ActiveMQ broker logfile (activemq.log) which has debug turned on (to provide more info). As soon as the error shown below is logged, the consumer is unable to dispatch any more messages from the ActiveMQ queue. The errors reported are:

      2008-01-10 16:37:24,474 [/127.0.0.1:3432] ERROR RecoveryListenerAdapter - Message id ID:B00118579B7DA-3430-1199983032693-0:1:1:1:99 could not be recovered from the data store!

      2008-01-10 16:37:24,474 [/127.0.0.1:3432] ERROR RecoveryListenerAdapter - Message id ID:B00118579B7DA-3430-1199983032693-0:1:1:1:99 could not be recovered from the data store!

      2008-01-10 16:37:24,474 [/127.0.0.1:3432] ERROR RecoveryListenerAdapter - Message id ID:B00118579B7DA-3430-1199983032693-0:1:1:1:99 could not be recovered from the data store!

      The error doesn't seem to happen at predictable fixed points, which is why the test program will send 100 message every 10sec, and over a number of iterations, it will eventually show the problem in the broker.

      Let me know if further information is required. I would really like to get the concurrent consumer to dispatch the messages, in order to reduce the time it takes to process multiple burst of messages.

        Activity

        Hide
        Rob Davies added a comment -

        This issue could not be replicated from trunk - there has been a lot of fixes in this area recently

        Show
        Rob Davies added a comment - This issue could not be replicated from trunk - there has been a lot of fixes in this area recently
        Hide
        Jeremy Haile added a comment - - edited

        We are seeing this issue in 5.2.0. We are using Spring 2.5.5 and using DefaultMessageListenerContainer. Our logs are full of this when starting up with multiple consumers:

        2009-02-17 12:13:57,679 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:13:57,756 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:13:58,839 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:13:59,125 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:13:59,195 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:14:00,076 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:14:00,305 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:14:00,538 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:14:00,895 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:14:00,963 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:14:01,416 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        2009-02-17 12:14:01,567 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched

        Show
        Jeremy Haile added a comment - - edited We are seeing this issue in 5.2.0. We are using Spring 2.5.5 and using DefaultMessageListenerContainer. Our logs are full of this when starting up with multiple consumers: 2009-02-17 12:13:57,679 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:13:57,756 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:13:58,839 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:13:59,125 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:13:59,195 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:14:00,076 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:14:00,305 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:14:00,538 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:14:00,895 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:14:00,963 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:14:01,416 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched 2009-02-17 12:14:01,567 [actionPerformed] ERROR RecoveryListenerAdapter - Message id ID:smithwicks-34271-1234883750803-0:21:40:1:1 could not be recovered from the data store - already dispatched
        Hide
        Jeremy Haile added a comment -

        Or maybe this is related to AMQ-1976? We're seeing this issue using the latest 5.2.0

        Show
        Jeremy Haile added a comment - Or maybe this is related to AMQ-1976 ? We're seeing this issue using the latest 5.2.0
        Hide
        Gary Tully added a comment -

        when you say the latest 5.2.0, do you mean a current 5.3-SNAPSHOT?

        Show
        Gary Tully added a comment - when you say the latest 5.2.0, do you mean a current 5.3-SNAPSHOT?
        Hide
        freetwix added a comment -

        I just had the same failure with an activemq 5.2 as shown as logfile entries in the first comment here.
        we are using a stomp connector and after restarting a bunch of durable consumers, the message appears repeatingly in the logfile (seems to be just one message causing the problem).
        after this, the whole message processing for all durable consumers stops and the pending queue size is increasing. any consumers of queues are not affected.
        is there a way to recovery the message processing from this state of the broker?

        greets

        Show
        freetwix added a comment - I just had the same failure with an activemq 5.2 as shown as logfile entries in the first comment here. we are using a stomp connector and after restarting a bunch of durable consumers, the message appears repeatingly in the logfile (seems to be just one message causing the problem). after this, the whole message processing for all durable consumers stops and the pending queue size is increasing. any consumers of queues are not affected. is there a way to recovery the message processing from this state of the broker? greets
        Hide
        Tommi Laukkanen added a comment -

        I'm having the same problem with ActiveMQ 5.2. Only way I can "recover" from this error is to add deleteAllMessagesOnStartup="true" attribute to broker config and restart the broker. This will delete all my messages which is not that nice...

        Are there any other solutions to this yet?

        Show
        Tommi Laukkanen added a comment - I'm having the same problem with ActiveMQ 5.2. Only way I can "recover" from this error is to add deleteAllMessagesOnStartup="true" attribute to broker config and restart the broker. This will delete all my messages which is not that nice... Are there any other solutions to this yet?
        Hide
        Marc DeXeT added a comment -

        I have the same issue with ActiveMQ 5.2.
        If I activate only a TopicSubscriber, I got the issue.
        If I activate only the Producer, I don't get the issue.
        The only way to recover I've found is to delete the store directory. Rather bully it isn't ?

        Show
        Marc DeXeT added a comment - I have the same issue with ActiveMQ 5.2. If I activate only a TopicSubscriber, I got the issue. If I activate only the Producer, I don't get the issue. The only way to recover I've found is to delete the store directory. Rather bully it isn't ?

          People

          • Assignee:
            Rob Davies
            Reporter:
            Wilson Luong
          • Votes:
            1 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development