Details

    • Type: Wish Wish
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.1.0
    • Fix Version/s: 5.6.0
    • Component/s: Broker
    • Labels:
      None

      Description

      When a message is redelivered the consumer blocks for the amount of time specified by the redelivery delay. For a high load scenario where message order is irrelevant this is just reducing performance and will result in a complete halt if the delay is long and several bad messages are consumed in a short time.

      I think what I basically wish for is how it worked in versions 3.x, prior to fix for AMQ-268. So I would very much like to have configurable option to NOT block consumers when redelivering messages.

      If no-one feels up to it, I'd still appreciate some hints and I could try to fix it myself. Looking at ActiveMQMessageConsumer.rollback(), I was thinking something in the lines of just scheduling a task to put the message back on queue after a delay - if configured to, instead of stopping delivery and a schedule a task to resume delivery again. But I do not possess an understanding of AMQ thorough enough to predict potential side effects of this, so any analysis would be helpful.

        Issue Links

        There are no Sub-Tasks for this issue.

          Activity

          Hide
          Gary Tully added a comment -

          the url query, b/c it is a connection factory attribute is:

          jms.nonBlockingRedelivery=true
          Show
          Gary Tully added a comment - the url query, b/c it is a connection factory attribute is: jms.nonBlockingRedelivery= true
          Hide
          Chris Pratt added a comment -

          Is the URL parameter for this nonBlockingRedelivery=true, jms.nonBlockingRedelivery=true, jms.redeliveryPolicy.nonBlockingRedelivery=true, or something else entirely?
          (*Chris*)

          Show
          Chris Pratt added a comment - Is the URL parameter for this nonBlockingRedelivery=true, jms.nonBlockingRedelivery=true, jms.redeliveryPolicy.nonBlockingRedelivery=true, or something else entirely? (*Chris*)
          Hide
          Gary Tully added a comment -

          https://issues.apache.org/jira/browse/AMQ-3894 implements broker based redelivery

          Show
          Gary Tully added a comment - https://issues.apache.org/jira/browse/AMQ-3894 implements broker based redelivery
          Hide
          Timothy Bish added a comment -

          By submitting patches and tests that make it better.

          Show
          Timothy Bish added a comment - By submitting patches and tests that make it better.
          Hide
          Michael Piotrowski added a comment -

          I see a problem. I downloaded AMQ 5.6-SNAPSHOT, configured it and my ConnectionFactory (wrapped) bean looks like this:

          <bean id="jmsConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory"
          destroy-method="stop">
          <property name="connectionFactory">
          <bean class="org.apache.activemq.ActiveMQConnectionFactory">
          <property name="brokerURL">
          <value>$

          {jms.broker.urls}

          </value>
          </property>

          <property name="redeliveryPolicy">
          <bean class="org.apache.activemq.RedeliveryPolicy">
          <property name="maximumRedeliveries" value="4"/>
          <property name="initialRedeliveryDelay" value="3000"/>
          <property name="useExponentialBackOff" value="true"/>
          <property name="backOffMultiplier" value="2"/>
          </bean>
          </property>

          <property name="nonBlockingRedelivery" value="true" />
          </bean>
          </property>
          </bean>

          The problem is that if 'nonBlockingRedelivery' is set to TRUE AFTER RedeliveryPolicy property, redelivering takes much more time than specified in RedeliveryPolicy object properties. On the other hand - while 'nonBlockingRedelivery' is put BEFORE declaration od RedeliveryPolicy - it takes under consideration only 'maximumRedeliveries' property - but with no delay and multiplier. That makes whole redelivery thing totally wrong - i got defult 6 requests one by one.

          Is this normal and expected behaviour? How can I achieve taking redeliveryPolicy properties to run?

          Show
          Michael Piotrowski added a comment - I see a problem. I downloaded AMQ 5.6-SNAPSHOT, configured it and my ConnectionFactory (wrapped) bean looks like this: <bean id="jmsConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" destroy-method="stop"> <property name="connectionFactory"> <bean class="org.apache.activemq.ActiveMQConnectionFactory"> <property name="brokerURL"> <value>$ {jms.broker.urls} </value> </property> <property name="redeliveryPolicy"> <bean class="org.apache.activemq.RedeliveryPolicy"> <property name="maximumRedeliveries" value="4"/> <property name="initialRedeliveryDelay" value="3000"/> <property name="useExponentialBackOff" value="true"/> <property name="backOffMultiplier" value="2"/> </bean> </property> <property name="nonBlockingRedelivery" value="true" /> </bean> </property> </bean> The problem is that if 'nonBlockingRedelivery' is set to TRUE AFTER RedeliveryPolicy property, redelivering takes much more time than specified in RedeliveryPolicy object properties. On the other hand - while 'nonBlockingRedelivery' is put BEFORE declaration od RedeliveryPolicy - it takes under consideration only 'maximumRedeliveries' property - but with no delay and multiplier. That makes whole redelivery thing totally wrong - i got defult 6 requests one by one. Is this normal and expected behaviour? How can I achieve taking redeliveryPolicy properties to run?
          Hide
          Per Steffensen added a comment -

          It is nice that there will finally be done something about this problem, eventhough I am not sure exactly that it is the same problem as I have been struggling with. This issue talks about the problem occuring if just on message is to be redelivered. The problem I have, only completely blocks delivery of new fresh messages, when there is more than maxMessagesPerSessions * maxSessions messages waiting for redelivery on the same consumer at the same time. I am using MDBs in Glassfish v2.1 as the consumer - maybe that makes the difference. For those with access - see more on https://fusesource.com/issues/browse/DEV-2279.

          Will this fix also solve my problem?

          Regards, Per Steffensen

          Show
          Per Steffensen added a comment - It is nice that there will finally be done something about this problem, eventhough I am not sure exactly that it is the same problem as I have been struggling with. This issue talks about the problem occuring if just on message is to be redelivered. The problem I have, only completely blocks delivery of new fresh messages, when there is more than maxMessagesPerSessions * maxSessions messages waiting for redelivery on the same consumer at the same time. I am using MDBs in Glassfish v2.1 as the consumer - maybe that makes the difference. For those with access - see more on https://fusesource.com/issues/browse/DEV-2279 . Will this fix also solve my problem? Regards, Per Steffensen
          Hide
          Timothy Bish added a comment -

          Fix added to trunk. New option nonBlockingRedelivery controls this. Setting nonBlockingRedelivery also implicitly enables transactedIndividualAck on the MessageConsumer.

          Show
          Timothy Bish added a comment - Fix added to trunk. New option nonBlockingRedelivery controls this. Setting nonBlockingRedelivery also implicitly enables transactedIndividualAck on the MessageConsumer.
          Hide
          Susan Macey added a comment -

          Updated unit test

          Show
          Susan Macey added a comment - Updated unit test
          Hide
          Timothy Bish added a comment -

          @Susan. If the new test differs from the one in the original patch you need to upload again and tick the grant license to apache. Note that I tested against the original patch tests with the addition of the option jms.nonBlockingRedelivery for the nonblocking test case and it works fine.

          Show
          Timothy Bish added a comment - @Susan. If the new test differs from the one in the original patch you need to upload again and tick the grant license to apache. Note that I tested against the original patch tests with the addition of the option jms.nonBlockingRedelivery for the nonblocking test case and it works fine.
          Hide
          Timothy Bish added a comment -

          I have this working somewhat. If anyone wants to contribute some unit tests with use cases now is the time, will help to ensure its solid in the 5.6 release.

          Show
          Timothy Bish added a comment - I have this working somewhat. If anyone wants to contribute some unit tests with use cases now is the time, will help to ensure its solid in the 5.6 release.
          Hide
          Susan Macey added a comment -

          Unit tests:: 1 test is for current behavior and passes; another fails, should pass once the ticket is fixed.

          Show
          Susan Macey added a comment - Unit tests:: 1 test is for current behavior and passes; another fails, should pass once the ticket is fixed.
          Hide
          Susan Macey added a comment -

          Unit test that validates current behavior, consumer blocks through all redeliveries of a message.

          Show
          Susan Macey added a comment - Unit test that validates current behavior, consumer blocks through all redeliveries of a message.
          Hide
          Fengming Lou added a comment -

          Since a message goes to dead letter queue once maximumRedeliveries is exceeded. One "work around" is to take advantage of individualDeadLetterStrategy. By shortening the maximuRedelieveries with the following configuration, the message will go back to the original queue and not to block new messages.
          <deadLetterStrategy>
          <individualDeadLetterStrategy queuePrefix="BACK.TO.WHERE.IT.FROM" useQueueForQueueMessages="true" />
          </deadLetterStrategy>

          Show
          Fengming Lou added a comment - Since a message goes to dead letter queue once maximumRedeliveries is exceeded. One "work around" is to take advantage of individualDeadLetterStrategy. By shortening the maximuRedelieveries with the following configuration, the message will go back to the original queue and not to block new messages. <deadLetterStrategy> <individualDeadLetterStrategy queuePrefix="BACK.TO.WHERE.IT.FROM" useQueueForQueueMessages="true" /> </deadLetterStrategy>
          Hide
          Kaustubh Khasnis added a comment -

          oh ok.. Which means one should get a mix of redelivered messages and undelivered messages, with redelivered messages not confirming to the order.

          Show
          Kaustubh Khasnis added a comment - oh ok.. Which means one should get a mix of redelivered messages and undelivered messages, with redelivered messages not confirming to the order.
          Hide
          Markus Schmidt added a comment -

          Hi Kaustubh,

          if you read on for a few sentences, you will find this:

          "Redelivered messages do not have to be delivered in exactly their original delivery order."

          So, as I understand it, the fix we all want here would conform to the documentation.

          Show
          Markus Schmidt added a comment - Hi Kaustubh, if you read on for a few sentences, you will find this: "Redelivered messages do not have to be delivered in exactly their original delivery order." So, as I understand it, the fix we all want here would conform to the documentation.
          Hide
          Kaustubh Khasnis added a comment -

          Hi all,
          but doesn't JMS spec says that the message delivery should stop and start from the oldest acknowledged message?? Just a thought
          (reference: http://download.oracle.com/javaee/6/api/javax/jms/Session.html#recover%28%29)

          Show
          Kaustubh Khasnis added a comment - Hi all, but doesn't JMS spec says that the message delivery should stop and start from the oldest acknowledged message?? Just a thought (reference: http://download.oracle.com/javaee/6/api/javax/jms/Session.html#recover%28%29 )
          Hide
          Mark Bakker added a comment -

          Thanks Claus, I just wanted to point out that it is not a work around for a production evironment. I spend alot of time to fix this issue. Just want to make the road to succes a more direct line for someone else.

          Show
          Mark Bakker added a comment - Thanks Claus, I just wanted to point out that it is not a work around for a production evironment. I spend alot of time to fix this issue. Just want to make the road to succes a more direct line for someone else.
          Hide
          Claus Ibsen added a comment -

          I said it was a side note, to tell that async redeliver is possible using the technique with a scheduled task. Not that it will resolve your issue. The problem is the AMQ broker, and Camel cannot magically resolve that issue.

          And beside Camel supports a threads in the DSL so you can hand over the message to another thread to continue routing that message. By which the consumer thread can pickup the next message.

          Show
          Claus Ibsen added a comment - I said it was a side note, to tell that async redeliver is possible using the technique with a scheduled task. Not that it will resolve your issue. The problem is the AMQ broker, and Camel cannot magically resolve that issue. And beside Camel supports a threads in the DSL so you can hand over the message to another thread to continue routing that message. By which the consumer thread can pickup the next message.
          Hide
          Mark Bakker added a comment -

          Well I totaly agree with Farhad Dehghani

          Show
          Mark Bakker added a comment - Well I totaly agree with Farhad Dehghani
          Hide
          Mark Bakker added a comment -

          Camel redelivery doesn't survive a server crash while redelivering. besides that the Camel docs state to use the underlying message broker in production situations, because of loosing messages when the Camel JVM gets stoped, killed, or crashes.

          Besides this issue Camel redelivery will hold a thread while redelivering (not returning the working thread to the consumer thread pool) and reducing the true put of consuming messages of this queue in general. When the message in redelivery are equal to number of consumer threads no new messages will get processed, and this time keep your finger crossed the server doesn't need to turn of or crashes. A semi work around is to set the redelivery to a short time to keep the system running, this will not fly for all usecases, and as stated it should not be used as a production configuration.

          Show
          Mark Bakker added a comment - Camel redelivery doesn't survive a server crash while redelivering. besides that the Camel docs state to use the underlying message broker in production situations, because of loosing messages when the Camel JVM gets stoped, killed, or crashes. Besides this issue Camel redelivery will hold a thread while redelivering (not returning the working thread to the consumer thread pool) and reducing the true put of consuming messages of this queue in general. When the message in redelivery are equal to number of consumer threads no new messages will get processed, and this time keep your finger crossed the server doesn't need to turn of or crashes. A semi work around is to set the redelivery to a short time to keep the system running, this will not fly for all usecases, and as stated it should not be used as a production configuration.
          Hide
          Farhad Dehghani added a comment -

          As long as camel uses activemq as the underlying messaging system, then it may necessarily have the same issue. You may be able to spawn a new thread to handle consumption or redelivery of a message, but my guess is that the original consumer thread would somehow remain blocked until activemq get exhausted to retry according to the redelivery policy. Please correct me if I'm wrong.
          I have tried this with spring-integration, by setting the "concurrentConsumers" on the jms-listener container up to a higher value than 1, and I succeed to continue picking messages from the queue, but the original consumer remains blocked.

          Show
          Farhad Dehghani added a comment - As long as camel uses activemq as the underlying messaging system, then it may necessarily have the same issue. You may be able to spawn a new thread to handle consumption or redelivery of a message, but my guess is that the original consumer thread would somehow remain blocked until activemq get exhausted to retry according to the redelivery policy. Please correct me if I'm wrong. I have tried this with spring-integration, by setting the "concurrentConsumers" on the jms-listener container up to a higher value than 1, and I succeed to continue picking messages from the queue, but the original consumer remains blocked.
          Hide
          Claus Ibsen added a comment -

          Just a note

          Apache Camel have support for using async redelivery, by as suggested, using a scheduled task to trigger the redelivery. In Camel there is a option you can configure if the redelivery should be sync/async.

          Show
          Claus Ibsen added a comment - Just a note Apache Camel have support for using async redelivery, by as suggested, using a scheduled task to trigger the redelivery. In Camel there is a option you can configure if the redelivery should be sync/async.
          Hide
          Gary Tully added a comment -

          that is pretty strong feedback, lets try and get to this for 5.6.
          Contributions (in the form of a patch) are most welcome

          Show
          Gary Tully added a comment - that is pretty strong feedback, lets try and get to this for 5.6. Contributions (in the form of a patch) are most welcome
          Hide
          Markus Schmidt added a comment -

          Couldn't agree more with Mark and Farhad, This is a real pain and makes ActiveMQ look like the most error-prone part of our system In our case it would also be hard to drop the use of the redelivery functionality at all, so I still hope for a fix. I can't go back to older releases, they have other issues.

          Show
          Markus Schmidt added a comment - Couldn't agree more with Mark and Farhad, This is a real pain and makes ActiveMQ look like the most error-prone part of our system In our case it would also be hard to drop the use of the redelivery functionality at all, so I still hope for a fix. I can't go back to older releases, they have other issues.
          Hide
          Mark Bakker added a comment -

          I completely agree with Farhad Dehghani, it is silly to freese the complete message delivery when some messages shoot in redelivery.

          In a few weeks we are again smashing our head to the wall because of this problem. 10M+ messages over the queue, hoping we won't get any rolbacks... Try explaning this to a customer.

          Show
          Mark Bakker added a comment - I completely agree with Farhad Dehghani, it is silly to freese the complete message delivery when some messages shoot in redelivery. In a few weeks we are again smashing our head to the wall because of this problem. 10M+ messages over the queue, hoping we won't get any rolbacks... Try explaning this to a customer.
          Hide
          Farhad Dehghani added a comment -

          I can't believe that you refuse to fix this issue after being around for such a long time. The fix represented in the https://issues.apache.org/activemq/browse/AMQ-2710 is too simplistic and cannot be applied in complex systems. There are workarounds for sure, but shouldn't be necessary. On the other hand this blocking behavior of redelivery mechanism makes using the RedeliveryPolicy somewhat irrelevant.
          The blocking/non-blocking behavior should definitely be made optional. Not fixing it would be admission to failure.

          Show
          Farhad Dehghani added a comment - I can't believe that you refuse to fix this issue after being around for such a long time. The fix represented in the https://issues.apache.org/activemq/browse/AMQ-2710 is too simplistic and cannot be applied in complex systems. There are workarounds for sure, but shouldn't be necessary. On the other hand this blocking behavior of redelivery mechanism makes using the RedeliveryPolicy somewhat irrelevant. The blocking/non-blocking behavior should definitely be made optional. Not fixing it would be admission to failure.
          Hide
          Timothy Bish added a comment -

          Working as designed, workaround available as specified in the comments.

          Show
          Timothy Bish added a comment - Working as designed, workaround available as specified in the comments.
          Hide
          Gary Tully added a comment -

          Check out a workaround using camel as described in https://issues.apache.org/activemq/browse/AMQ-2710

          Show
          Gary Tully added a comment - Check out a workaround using camel as described in https://issues.apache.org/activemq/browse/AMQ-2710
          Hide
          Paul Khodchenkov added a comment -

          This is still reproducable in 5.4 .

          Show
          Paul Khodchenkov added a comment - This is still reproducable in 5.4 .
          Hide
          John Miller added a comment -

          This bug is present int ActiveMQ 5.1 and 5.2

          Show
          John Miller added a comment - This bug is present int ActiveMQ 5.1 and 5.2
          Hide
          John Miller added a comment -

          I have the same problem. When a message listener throws a RuntimeException, ActiveMQ tries to re-deliver the same message according to redelivery policy, and the rest of the messages are blocked until the message which caused the exception is redelivered with success or put into the dead letters queue. I want my messages to be redelivered with initial redelivery delay 10 seconds with back-off multiplier 3, maximum 2redelivery attempts, exponential back-off is on. It means that the messages in the queue are blocked for 10+30+90 = 130 seconds !

          Show
          John Miller added a comment - I have the same problem. When a message listener throws a RuntimeException, ActiveMQ tries to re-deliver the same message according to redelivery policy, and the rest of the messages are blocked until the message which caused the exception is redelivered with success or put into the dead letters queue. I want my messages to be redelivered with initial redelivery delay 10 seconds with back-off multiplier 3, maximum 2redelivery attempts, exponential back-off is on. It means that the messages in the queue are blocked for 10+30+90 = 130 seconds !
          Hide
          Dima added a comment -

          We have the same problem. We solve it in the next way: I wrap MessageListener by another, and if main MessageListener throws Exception wrapper redirect this mesage to sibling queue. This sibling queue listen the same MessageListener and good messages is not blocked

          Show
          Dima added a comment - We have the same problem. We solve it in the next way: I wrap MessageListener by another, and if main MessageListener throws Exception wrapper redirect this mesage to sibling queue. This sibling queue listen the same MessageListener and good messages is not blocked

            People

            • Assignee:
              Timothy Bish
              Reporter:
              Demian Mrakovich
            • Votes:
              10 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development