Qpid
  1. Qpid
  2. QPID-350

Broker infinite loop on restart with immediate messages

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: M1
    • Fix Version/s: None
    • Component/s: Java Broker
    • Labels:
      None

      Description

      Messages being sent as immediate from .NET client to Java client, across Java broker.

      Saw this problem on broker restart (with persistent messages):

      2007-02-07 16:03:08,058 ERROR [pool-1-thread-3] queue.ConcurrentSelectorDeliveryManager (ConcurrentSelectorDeliveryManager.java:317) - Unable to deliver message as dequeue failed: org.apache.qpid.server.queue.FailedDequeueException: Failed to dequeue message from rad_u286664_fid-mngodev4
      org.apache.qpid.server.queue.FailedDequeueException: Failed to dequeue message from rad_u286664_fid-mngodev4
      at org.apache.qpid.server.queue.AMQQueue.dequeue(AMQQueue.java:548)
      at org.apache.qpid.server.queue.SubscriptionImpl.sendToConsumer(SubscriptionImpl.java:260)
      at org.apache.qpid.server.queue.SubscriptionImpl.send(SubscriptionImpl.java:212)
      at org.apache.qpid.server.queue.ConcurrentSelectorDeliveryManager.sendNextMessage(ConcurrentSelectorDeliveryManager.java:308)
      at org.apache.qpid.server.queue.ConcurrentSelectorDeliveryManager.sendNextMessage(ConcurrentSelectorDeliveryManager.java:361)
      at org.apache.qpid.server.queue.ConcurrentSelectorDeliveryManager.processQueue(ConcurrentSelectorDeliveryManager.java:338)
      at org.apache.qpid.server.queue.ConcurrentSelectorDeliveryManager.access$100(ConcurrentSelectorDeliveryManager.java:47)
      at org.apache.qpid.server.queue.ConcurrentSelectorDeliveryManager$Runner.run(ConcurrentSelectorDeliveryManager.java:471)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:595)
      Caused by: org.apache.qpid.AMQException: Unable to find message with id 7 on queue rad_u286664_fid-mngodev4
      at org.apache.qpid.server.store.berkeleydb.BDBMessageStore.dequeueMessage(BDBMessageStore.java:370)
      at org.apache.qpid.server.queue.WeakReferenceMessageHandle.dequeue(WeakReferenceMessageHandle.java:201)
      at org.apache.qpid.server.queue.AMQMessage.dequeue(AMQMessage.java:457)
      at org.apache.qpid.server.queue.AMQQueue.dequeue(AMQQueue.java:535)
      ... 10 more

        Activity

        Hide
        Aidan Skinner added a comment -

        This doesn't happen anymore

        Show
        Aidan Skinner added a comment - This doesn't happen anymore
        Hide
        Martin Ritchie added a comment -

        Rupert to test and reproduce or close.

        Show
        Martin Ritchie added a comment - Rupert to test and reproduce or close.
        Hide
        Martin Ritchie added a comment -

        Problem looks to be that the message is missing from the given MessageStore as a result the FailedDequeueException occurs. But as the message is not removed from the Queue the DeliveryManger continues to try and deliver it as it is next in the list.

        In this case the message should be moved to an DeadLetterQueue.. which we don't have yet.

        Wrapping the sub.send(message, _queue);

        line in the ConcurrentSelectorDeliveryManager should prevent this recursive restart problem .. but only currently at the expense of throwing the message away. The reason behind why message (7 in this case) cannot be found is perhaps more important.

        try

        { sub.send(message, _queue); //remove sent message from our queue. removed = messageQueue.poll(); //If we don't remove the message from _messages // Otherwise the Async send will never end }

        catch (FailedDequeueException fde)

        { // Message should be moved to dead letter queue _log.error("MESSAGE LOSS: unable to Dequeue Message so removing from queue:" + fde); removed = messageQueue.poll(); // sendToDeadletter(removed); }
        Show
        Martin Ritchie added a comment - Problem looks to be that the message is missing from the given MessageStore as a result the FailedDequeueException occurs. But as the message is not removed from the Queue the DeliveryManger continues to try and deliver it as it is next in the list. In this case the message should be moved to an DeadLetterQueue.. which we don't have yet. Wrapping the sub.send(message, _queue); line in the ConcurrentSelectorDeliveryManager should prevent this recursive restart problem .. but only currently at the expense of throwing the message away. The reason behind why message (7 in this case) cannot be found is perhaps more important. try { sub.send(message, _queue); //remove sent message from our queue. removed = messageQueue.poll(); //If we don't remove the message from _messages // Otherwise the Async send will never end } catch (FailedDequeueException fde) { // Message should be moved to dead letter queue _log.error("MESSAGE LOSS: unable to Dequeue Message so removing from queue:" + fde); removed = messageQueue.poll(); // sendToDeadletter(removed); }

          People

          • Assignee:
            Rupert Smith
            Reporter:
            Marnie McCormack
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development