ActiveMQ
  1. ActiveMQ
  2. AMQ-2336

Redeliveried messages stops consumers from going on consuming the rest of messages in the queue

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 5.2.0, 5.3.0
    • Fix Version/s: 5.5.0
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      Windows XP, Java 1.5.0_11
      Spring 2.5.6 DefaultMessageListenerContainer

      Description

      Here are the steps to reproduce de problem:

      • redeliveryPolicy configured in ActiveMQConnectionFactory as initialRedeliveryDelay 10000, maximumRedeliveries 6, useExponentialBackOff true, backOffMultiplier 3. With these settings, messages, in case of error, stays in the queue several minutes.
      • DefaultMessageListenerContainer with concurrentConsumers, maxConcurrentConsumers set to 1, sessionTransacted=true, and cacheLevel CACHE_CONSUMER
      • We introduce a "bad" message in a queue (configured to cause a RuntimeException in the MDP associated with the queue). The message waits in the queue to be redelivered again (seen in jconsole)
      • We introduce a "good" message in a queue. The MDP associated with the queue should process this message inmediatly with no exception, but instead, the message is not been processed until last redelivery of the "bad" message, and it happens several minutes later. When this happens, message is processed Ok.

      This problem can cause queues to grow unnecessarily if few messages are waiting to be redelivered. What I expected to happen is messages being processed as they arrive if there is a consumer available, but it seems consumers with a rollback in a transaction are not available until the rollbacked message is processed again and finally sent to DLQ.

      In addition, in Activemq version 5.2.0, when I remove consumers while waiting for next redelivery of the message, broker throws this exception (this not happens with 5.3-SNAPSHOT):

      30 jul 2009 14:17:45,140 ERROR [ActiveMQ Transport: tcp:///127.0.0.1:3656] org.apache.activemq.broker.TransportConnection.Service - sync error occurred:
      javax.jms.JMSException: Transaction 'TX:ID:mymachine-4579-1248953104078-0:5207:52' has not been started.
      javax.jms.JMSException: Transaction 'TX:ID:mymachine-4579-1248953104078-0:5207:52' has not been started.
      at org.apache.activemq.broker.TransactionBroker.getTransaction(TransactionBroker.java:270)
      at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:190)
      at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
      at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
      at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
      at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
      at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
      at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:143)
      at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:206)
      at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
      at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
      at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
      at java.lang.Thread.run(Thread.java:595

        Activity

        Hide
        Timothy Bish added a comment -

        This is working as designed for a transacted consumer.

        Show
        Timothy Bish added a comment - This is working as designed for a transacted consumer.
        Hide
        Diego Rodriguez added a comment -

        Ok in the case of just 1 consumer. My intention was to simplify the reproduction of the problem

        It still happens with 2 consumers and this sequence (the test pattern is the same, send a message to the queue and watch in jconsole and log if it is processed and queue size):

        • 1 bad message: message stays in the queue waiting for redelivery. Queue size : 1
        • 1 good message: it is processed ok. Queue size : 1
        • 1 good message: it is NOT processed. Queue size : 2
        • 1 good message: it is processed ok. Queue size : 2
        • 1 good message: it is NOT processed. Queue size : 3
        • 1 good message: it is processed ok. Queue size : 3
        • 1 good message: it is processed ok. Queue size : 3
        • 1 good message: it is processed ok. Queue size : 3
          ......................

        With 3 consumers

        • 1 bad message: message stays in the queue waiting for redelivery. Queue size : 1
        • 1 good message: it is processed ok. Queue size : 1
        • 1 good message: it is processed ok. Queue size : 1
        • 1 good message: it is NOT processed. Queue size : 2
        • 1 good message: it is processed ok. Queue size : 2
        • 1 good message: it is processed ok. Queue size : 2
        • 1 good message: it is NOT processed. Queue size : 3
        • 1 good message: it is processed ok. Queue size : 3
        • 1 good message: it is processed ok. Queue size : 3
        • 1 good message: it is processed ok. Queue size : 3

        This seems odd to me, first because not all messages got stucked in the queue and every new message after queue size reaches 3 messages is processed inmediatly.

        I'm going to install ActiveMQ in a production system in a few days and I am worried about the fact of messages being stucked in the queue for minutes because of redeliveries.

        Show
        Diego Rodriguez added a comment - Ok in the case of just 1 consumer. My intention was to simplify the reproduction of the problem It still happens with 2 consumers and this sequence (the test pattern is the same, send a message to the queue and watch in jconsole and log if it is processed and queue size): 1 bad message: message stays in the queue waiting for redelivery. Queue size : 1 1 good message: it is processed ok. Queue size : 1 1 good message: it is NOT processed. Queue size : 2 1 good message: it is processed ok. Queue size : 2 1 good message: it is NOT processed. Queue size : 3 1 good message: it is processed ok. Queue size : 3 1 good message: it is processed ok. Queue size : 3 1 good message: it is processed ok. Queue size : 3 ...................... With 3 consumers 1 bad message: message stays in the queue waiting for redelivery. Queue size : 1 1 good message: it is processed ok. Queue size : 1 1 good message: it is processed ok. Queue size : 1 1 good message: it is NOT processed. Queue size : 2 1 good message: it is processed ok. Queue size : 2 1 good message: it is processed ok. Queue size : 2 1 good message: it is NOT processed. Queue size : 3 1 good message: it is processed ok. Queue size : 3 1 good message: it is processed ok. Queue size : 3 1 good message: it is processed ok. Queue size : 3 This seems odd to me, first because not all messages got stucked in the queue and every new message after queue size reaches 3 messages is processed inmediatly. I'm going to install ActiveMQ in a production system in a few days and I am worried about the fact of messages being stucked in the queue for minutes because of redeliveries.
        Hide
        Gary Tully added a comment -

        this is message ordering in action, it is expected behavior. The good message cannot be received till the bad message is discarded. A second consumer will see the new messages, but a good message cannot jump a bad message in the queue for a particular consumer.

        Show
        Gary Tully added a comment - this is message ordering in action, it is expected behavior. The good message cannot be received till the bad message is discarded. A second consumer will see the new messages, but a good message cannot jump a bad message in the queue for a particular consumer.

          People

          • Assignee:
            Unassigned
            Reporter:
            Diego Rodriguez
          • Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development