ActiveMQ
  1. ActiveMQ
  2. AMQ-674

Composite Destination persisted messages never get cleaned up, halt message producers

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.2.1
    • Fix Version/s: 4.0 RC2
    • Component/s: Broker, Message Store
    • Labels:
      None

      Description

      well, we've just got bit by something huge.

      Previously we have been shipping messages from producer to consumer via a named Queue 'Index2.EntityIncrementalIndexer', where the machine index2 has a consumer called "EntityIncrementalIndexer" using persisted messages.

      The design used Composite destinations so that the destination could be:

      "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndexer"

      and have the 2 hosts get sent the same message simply, and if one goes down, it'll catch up later. This gave us the ability to have a mirrored configuration, and go tertiary later if we want simply by adding a new name in the composite destation.

      With a single name, this works great, been working fine for months. Yesterday we activated the true composite destination, and both machines started getting their messages fine.

      no problems so far, BUT, 12 hours later we have noticed that the broker has now stopped accepting messages, and a look at the activemq_msgs table shows 33228 messages

        Activity

        Jeff Turner made changes -
        Project Import Fri Nov 26 22:32:02 EST 2010 [ 1290828722158 ]
        Hide
        james strachan added a comment -

        BTW in 4.x the queue dispatch mechanism has been completely changed so that we can deal with massive queues with millions of messages on them; we don't have to keep around the entire queue contents in RAM in 4.x so you should not see the "WARN Queue is full, waiting for it to be dequeued. " messages in 4.x

        Show
        james strachan added a comment - BTW in 4.x the queue dispatch mechanism has been completely changed so that we can deal with massive queues with millions of messages on them; we don't have to keep around the entire queue contents in RAM in 4.x so you should not see the "WARN Queue is full, waiting for it to be dequeued. " messages in 4.x
        Hide
        james strachan added a comment -

        I'm with you now. Composite destinations should work fine for you - its a way of using multiple queues to simulate durable topics - which is a good thing in general. (I'd like to provide queue-topic unification at some point so you can see durable topics as being like multiple virtual queues which multiple consumers can consume on the same queue to get load balancing - while avoiding the multiple-persistence of the message to each queue which happens now with composite destinations - but thats another topic).

        My mention of RAM is that this error messsage...

        14:45:03 WARN Queue is full, waiting for it to be dequeued.

        is created when the queue dispatcher uses up its available RAM because it keeps all unconsumed queue messages in RAM until they are consumed. So its sounding like some messages on some queue are not getting consumed. Unfortunately in 3.x there's no easy way to look and see for sure - if you used 4.x you could just use JConsole to see all the queues and what messages are on each etc.

        Show
        james strachan added a comment - I'm with you now. Composite destinations should work fine for you - its a way of using multiple queues to simulate durable topics - which is a good thing in general. (I'd like to provide queue-topic unification at some point so you can see durable topics as being like multiple virtual queues which multiple consumers can consume on the same queue to get load balancing - while avoiding the multiple-persistence of the message to each queue which happens now with composite destinations - but thats another topic). My mention of RAM is that this error messsage... 14:45:03 WARN Queue is full, waiting for it to be dequeued. is created when the queue dispatcher uses up its available RAM because it keeps all unconsumed queue messages in RAM until they are consumed. So its sounding like some messages on some queue are not getting consumed. Unfortunately in 3.x there's no easy way to look and see for sure - if you used 4.x you could just use JConsole to see all the queues and what messages are on each etc.
        Hide
        Paul Smith added a comment -

        There is a logical queue within our system called "EntityIncrementalIndexer' (don't think JMS here, just thing in-out pipeline), which updates to mail & docs in our app are sent as incremental update messages to index nodes.

        Each index node, (Index1 & Index2) need to have their own queue, and we use the Composite Destination feature to clone a message to both destinations:

        producer -> JMS -> Index1
        -> Index2

        The producer is configured to know how many nodes in the cluster, and sends queue messages via composite destinations, at startup it sets aa variable about the index node list, and creates a Desitaniotn of the form "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndex". I thought each Index node would then read from it's own queue, with ActiveMQ pushing the 1 sent message into BOTH queues.

        If your immediate thought is "Shouldn't this be done via Durable Subscription topics?", you are probably right, but I've been bitten by vendors Durable Subscription implementation before, and thought the Composite Destination approach might be simpler, and more elegant. (wrong! )

        Now, I'm confused by your comment about RAM. You see I don't believe any of teh messages sent since we started using 2 nodes in the composite destination ever got 'missed'. That is from our logs I believe that all messages arrived and were consumed by the destination. I believe the messages are being persisted, but not marked as consumed.

        Show
        Paul Smith added a comment - There is a logical queue within our system called "EntityIncrementalIndexer' (don't think JMS here, just thing in-out pipeline), which updates to mail & docs in our app are sent as incremental update messages to index nodes. Each index node, (Index1 & Index2) need to have their own queue, and we use the Composite Destination feature to clone a message to both destinations: producer -> JMS -> Index1 -> Index2 The producer is configured to know how many nodes in the cluster, and sends queue messages via composite destinations, at startup it sets aa variable about the index node list, and creates a Desitaniotn of the form "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndex". I thought each Index node would then read from it's own queue, with ActiveMQ pushing the 1 sent message into BOTH queues. If your immediate thought is "Shouldn't this be done via Durable Subscription topics?", you are probably right, but I've been bitten by vendors Durable Subscription implementation before, and thought the Composite Destination approach might be simpler, and more elegant. (wrong! ) Now, I'm confused by your comment about RAM. You see I don't believe any of teh messages sent since we started using 2 nodes in the composite destination ever got 'missed'. That is from our logs I believe that all messages arrived and were consumed by the destination. I believe the messages are being persisted, but not marked as consumed.
        Hide
        james strachan added a comment -

        The issue isn't with Composite Destinations per se - they are pretty trivial things, just a 1-many relationship of destinations. The issue is when you run out of RAM due to too many messages stuck on some queue, things kinda lock up until messages get consumed.

        So I definitely recommend you upgrade to 4.0-RC2...
        http://people.apache.org/~chirino/incubator-activemq-4.0-RC2/

        which has all these issues fixed.

        Note I'm a little confused when you talk about Index1.Foo and Index2.Foo and machines index1/index2. Index1.Foo is a queue which will exist on the broker your producer talks to right? Are you networking 2 brokers together in a store/forward network?

        BTW one other benefit of 4.x is you can use JMX to view all the queues & their queue depths to easily grok whats going on...
        http://activemq.org/JMX

        Show
        james strachan added a comment - The issue isn't with Composite Destinations per se - they are pretty trivial things, just a 1-many relationship of destinations. The issue is when you run out of RAM due to too many messages stuck on some queue, things kinda lock up until messages get consumed. So I definitely recommend you upgrade to 4.0-RC2... http://people.apache.org/~chirino/incubator-activemq-4.0-RC2/ which has all these issues fixed. Note I'm a little confused when you talk about Index1.Foo and Index2.Foo and machines index1/index2. Index1.Foo is a queue which will exist on the broker your producer talks to right? Are you networking 2 brokers together in a store/forward network? BTW one other benefit of 4.x is you can use JMX to view all the queues & their queue depths to easily grok whats going on... http://activemq.org/JMX
        Hide
        Paul Smith added a comment -

        Just to be clear, so persistent Composite Destination support is basically useless in the 3.x series?

        You see, the consumers are definitely keeping up with the producers, we definitely do not have slow consumers here. THere's no way that there is 32,000 unconsumed messages. (In fact the consumer nodes are sitting there doing nothing, not being delivered anything)

        I don't mind upgrading to 4.x (they don't call me the upgrade-a-tron for nothing), just want to clarify that there is no point pursuing a 3.2x fix?

        cheers,

        Paul

        Show
        Paul Smith added a comment - Just to be clear, so persistent Composite Destination support is basically useless in the 3.x series? You see, the consumers are definitely keeping up with the producers, we definitely do not have slow consumers here. THere's no way that there is 32,000 unconsumed messages. (In fact the consumer nodes are sitting there doing nothing, not being delivered anything) I don't mind upgrading to 4.x (they don't call me the upgrade-a-tron for nothing), just want to clarify that there is no point pursuing a 3.2x fix? cheers, Paul
        james strachan made changes -
        Field Original Value New Value
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Fix Version/s 4.0 RC1 [ 11727 ]
        Hide
        james strachan added a comment -

        This is a known issue with the architecture of 3.x; once you hit queue full; the system generally pauses until you consume more messages. You're only option is to increase the RAM limit - but it is generally delaying the inevitable.

        We recommend upgrading to 4.x which has this issue resolved

        Show
        james strachan added a comment - This is a known issue with the architecture of 3.x; once you hit queue full; the system generally pauses until you consume more messages. You're only option is to increase the RAM limit - but it is generally delaying the inevitable. We recommend upgrading to 4.x which has this issue resolved
        Hide
        Paul Smith added a comment -

        We're using SQL 2000 as the JDBC store btw.

        Show
        Paul Smith added a comment - We're using SQL 2000 as the JDBC store btw.
        Hide
        Paul Smith added a comment -

        hmm, it snipped my SQL

        select container, count from activemq_msgs group by container

        Index2.EntityIncrementalIndexer 6
        Index2.EntityIncrementalIndexer,Index1.EntityIncrementalIndexer 33228

        activemq.log:

        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN memory limit low - forced to remove expired messages: PROCESSED_REQUESTS
        14:45:03 WARN memory limit low - forced to remove expired messages: INCOMING_REQUESTS
        14:45:03 WARN memory limit low - forced to remove expired messages: Index2.EntityFullIndexer
        14:45:03 WARN memory limit low - forced to remove expired messages: Index1.EntityFullIndexer
        14:45:03 WARN memory limit low - forced to remove expired messages: Index2.EntityIncrementalIndexer,Index1.EntityIncrementalIndexer
        14:45:03 WARN memory limit low - forced to remove expired messages: Index1.EntityIncrementalIndexer
        14:45:03 WARN memory limit low - forced to remove expired messages: Index2.EntityIncrementalIndexer
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.
        14:45:03 WARN Queue is full, waiting for it to be dequeued.

        Show
        Paul Smith added a comment - hmm, it snipped my SQL select container, count from activemq_msgs group by container Index2.EntityIncrementalIndexer 6 Index2.EntityIncrementalIndexer,Index1.EntityIncrementalIndexer 33228 activemq.log: 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN memory limit low - forced to remove expired messages: PROCESSED_REQUESTS 14:45:03 WARN memory limit low - forced to remove expired messages: INCOMING_REQUESTS 14:45:03 WARN memory limit low - forced to remove expired messages: Index2.EntityFullIndexer 14:45:03 WARN memory limit low - forced to remove expired messages: Index1.EntityFullIndexer 14:45:03 WARN memory limit low - forced to remove expired messages: Index2.EntityIncrementalIndexer,Index1.EntityIncrementalIndexer 14:45:03 WARN memory limit low - forced to remove expired messages: Index1.EntityIncrementalIndexer 14:45:03 WARN memory limit low - forced to remove expired messages: Index2.EntityIncrementalIndexer 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued. 14:45:03 WARN Queue is full, waiting for it to be dequeued.
        Paul Smith created issue -

          People

          • Assignee:
            Unassigned
            Reporter:
            Paul Smith
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development