ActiveMQ
  1. ActiveMQ
  2. AMQ-2102

Master/slave out of sync with multiple consumers

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 5.2.0
    • Fix Version/s: 5.3.0
    • Component/s: Broker
    • Labels:
      None

      Description

      I'm seeing exceptions like this in a simple master/slave setup:

      ERROR Service - Async error occurred: javax.jms.JMSException: Slave broker out of sync with master: Dispatched message (ID:DUL1SJAMES-L2-1231-1233929569359-0:4:1:1:207) was not in the pending list for MasterSlaveBug
      javax.jms.JMSException: Slave broker out of sync with master: Dispatched message (ID:DUL1SJAMES-L2-1231-1233929569359-0:4:1:1:207) was not in the pending list for MasterSlaveBug

      The problem only happens when there are multiple consumers listening to the queue, and is more likely to occur as there are more consumers listening. I've written a test program that demonstrates the problem.

      I start the master and slave with an empty data directory and let them both startup and settle. Then start the test program. The test program creates a specified number of consumers, and then starts queuing 256 messages. The consumers process the message by sending a reply. The producer counts the replies. Both consumers and the producer see all the messages, but with multiple consumers it is very likely that the error above will occur and several of the messages will still be queued on the slave.

      While debugging through the activemq code, I noticed that both the master and the slave dispatch the message to a consumer's pending list independently. In other words, it is possible that the master will add the message to consumer A's pending list and the slave will add the message to consumer B's pending list. Once the message has been processed by consumer A, the master sends a message to the slaving which specifies consumer A so that the slave can remove the message. The slave looks on its copy of consumer A's pending list and cannot find the message. As a result, it throws this exception and the message stays stuck on consumer B's pending list on the slave.

      Master and slave configurations along with MasterSlaveBug.java are attached to this issue.

      Start master and slave brokers:
      activemq xbean:master.xml
      activemq xbean:slave.xml

      Run with (only one consumer, the bug does not appear):
      java -classpath .:activemq-all-5.2.0.jar MasterSlaveBug 1
      Run with (sixteen consumers, the bug does appear):
      java -classpath .:activemq-all-5.2.0.jar MasterSlaveBug 16

      1. slaveDispatchOnNotification.patch
        16 kB
        Gary Tully
      2. slave.xml
        2 kB
        Dan James
      3. MasterSlavePatch.patch
        1 kB
        ying
      4. MasterSlaveBug.java
        13 kB
        Dan James
      5. master.xml
        2 kB
        Dan James
      6. AMQ-2102-03102009.patch
        23 kB
        ying
      7. AMQ2102.12-03.patch
        28 kB
        Gary Tully

        Activity

        Hide
        ying added a comment -

        thank you too. just finished a testing of 2 million messages with this new patch test. it works fine with no mismatch. the consumer stop consuming is due to systemUsage config, also heap needs to bump up otherwise it will get JVM java.lang.OutOfMemoryError: GC overhead limit exceeded.

        I am currently observing heap. I have 4 pair of master/slave, looks like the one pair which the consumer is connecting to, after 2 million msgs, has higher used heap, about >10mb higher than the rest of the broker, even after killing all producers and consumers. is there a possible leak? i will continue watch out and have more tests.

        Show
        ying added a comment - thank you too. just finished a testing of 2 million messages with this new patch test. it works fine with no mismatch. the consumer stop consuming is due to systemUsage config, also heap needs to bump up otherwise it will get JVM java.lang.OutOfMemoryError: GC overhead limit exceeded. I am currently observing heap. I have 4 pair of master/slave, looks like the one pair which the consumer is connecting to, after 2 million msgs, has higher used heap, about >10mb higher than the rest of the broker, even after killing all producers and consumers. is there a possible leak? i will continue watch out and have more tests.
        Hide
        Gary Tully added a comment -

        fix committed, Ying, thanks for your input. r753214

        slave dispatch now occurs on processDispatchNotification when master has chosen the subscriber. With large volumes and prefetch sizes and short lived consumers messages on the slave may not be in memory and need to be pulled directly from the store. Normally this is not the case though.

        Having the master and slave do separate dispatch does not work due to re deliveries and consumer add/remove order and load effects on thread scheduling. with the stricter enforcement of ack delivery to subscriptions this behavior unfolded.

        Show
        Gary Tully added a comment - fix committed, Ying, thanks for your input. r753214 slave dispatch now occurs on processDispatchNotification when master has chosen the subscriber. With large volumes and prefetch sizes and short lived consumers messages on the slave may not be in memory and need to be pulled directly from the store. Normally this is not the case though. Having the master and slave do separate dispatch does not work due to re deliveries and consumer add/remove order and load effects on thread scheduling. with the stricter enforcement of ack delivery to subscriptions this behavior unfolded.
        Hide
        Gary Tully added a comment -

        Yea Ying, I came to the same conclusion. The message needs to be pulled directly from the store when the pageSize is full and all subscriptions are full also.
        This patch is a tidier version of the original. Can you validate on your side. It does not need as many changes to MasterBroker async sends are they will impede performance.
        Also, there is a suppression of dispatchNotifications when a subscription is removed.

        Show
        Gary Tully added a comment - Yea Ying, I came to the same conclusion. The message needs to be pulled directly from the store when the pageSize is full and all subscriptions are full also. This patch is a tidier version of the original. Can you validate on your side. It does not need as many changes to MasterBroker async sends are they will impede performance. Also, there is a suppression of dispatchNotifications when a subscription is removed.
        Hide
        ying added a comment -

        Attached AMQ-2102-03102009.patch seems fix all the slave broker out of synch issue. This is patch is created on activemq 5.2.0 official release tagged source. sorry that I run a tight schedule and don't have time to patch on trunk.

        One thing missed in this patch is:
        {{{
        public MessageReference getMessage(MessageId id) throws IOException

        { Message msg = this.store.getMessage(id); return msg; }

        }}}

        in org.apache.activemq.broker.region.cursors.QueueStorePrefetch.java

        Here is some explaination:

        1. MasterBroker.java changes make sure that each command will arrive on the slave side in the same order. This is necessary because for each specific message, message itself, dispatch notification and message ack has to come in the exact order, otherwise, slave out of sync

        2. Queue.java if dispatch notification comes and the messages still not in pending list or pagedIn, it goes to the store to grab the message and add to the pending then do processMessageDispatchNotification. This is necessary because that multiple consumers are involved (eg. dispatch notification for message 201 could come first while only 200 messages on slave side is pagedin or in the pending list)

        Please review it. I urgently need a review of this to make sure the changes are fine.

        A remaining issue which might or might related to this fix:

        I see "consumer stop consuming message" when large number of messages is produced and consumed and I have to restart the broker pair. otherwise, the brokers seems hanging. do you have any insight what might go wrong?

        Show
        ying added a comment - Attached AMQ-2102 -03102009.patch seems fix all the slave broker out of synch issue. This is patch is created on activemq 5.2.0 official release tagged source. sorry that I run a tight schedule and don't have time to patch on trunk. One thing missed in this patch is: {{{ public MessageReference getMessage(MessageId id) throws IOException { Message msg = this.store.getMessage(id); return msg; } }}} in org.apache.activemq.broker.region.cursors.QueueStorePrefetch.java Here is some explaination: 1. MasterBroker.java changes make sure that each command will arrive on the slave side in the same order. This is necessary because for each specific message, message itself, dispatch notification and message ack has to come in the exact order, otherwise, slave out of sync 2. Queue.java if dispatch notification comes and the messages still not in pending list or pagedIn, it goes to the store to grab the message and add to the pending then do processMessageDispatchNotification. This is necessary because that multiple consumers are involved (eg. dispatch notification for message 201 could come first while only 200 messages on slave side is pagedin or in the pending list) Please review it. I urgently need a review of this to make sure the changes are fine. A remaining issue which might or might related to this fix: I see "consumer stop consuming message" when large number of messages is produced and consumed and I have to restart the broker pair. otherwise, the brokers seems hanging. do you have any insight what might go wrong?
        Hide
        ying added a comment -

        defer the message dispatch on slave is right I think. but the issue is when multiple consumer is involved, message dispatch notification and message ack are not coming in order in teams of command id.

        is there a way on the slave side to say, when I get a messagedispatchNotification, page in that message and dispatch it?

        Show
        ying added a comment - defer the message dispatch on slave is right I think. but the issue is when multiple consumer is involved, message dispatch notification and message ack are not coming in order in teams of command id. is there a way on the slave side to say, when I get a messagedispatchNotification, page in that message and dispatch it?
        Hide
        ying added a comment -

        Gary, i am using the same MasterSlaveBug test to see whether it is still having slave broker out of sync.

        As to duplicates, i just simply have a consumer and in it I have a
        {{{
        private static HashSet<String> dupCheck = new HashSet<String>();
        if(!dupCheck.add (textMessage.getText()))

        { System.out.println("### Message already received [" + textMessage.getText() + "]"); }

        }}}

        Show
        ying added a comment - Gary, i am using the same MasterSlaveBug test to see whether it is still having slave broker out of sync. As to duplicates, i just simply have a consumer and in it I have a {{{ private static HashSet<String> dupCheck = new HashSet<String>(); if(!dupCheck.add (textMessage.getText())) { System.out.println("### Message already received [" + textMessage.getText() + "]"); } }}}
        Hide
        Gary Tully added a comment -

        ying, could you possibly codify your test in Junit and attach it. The broker setup from this test may be useful: http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/bugs/AMQ2102Test.java?view=markup
        then I can run your test case in tandem against my changes. thanks.

        Show
        Gary Tully added a comment - ying, could you possibly codify your test in Junit and attach it. The broker setup from this test may be useful: http://svn.apache.org/viewvc/activemq/trunk/activemq-core/src/test/java/org/apache/activemq/bugs/AMQ2102Test.java?view=markup then I can run your test case in tandem against my changes. thanks.
        Hide
        ying added a comment -

        this update helps. did another test and it still has the slave out of sync with broker and another bad thing is consumer now can see duplicate messages. more work needed.

        Show
        ying added a comment - this update helps. did another test and it still has the slave out of sync with broker and another bad thing is consumer now can see duplicate messages. more work needed.
        Hide
        Gary Tully added a comment -

        sorry, my patch is still a bit of a work in progress. I was using this jira as a shared holder. For the cce,

         private QueueMessageReference getMatchingMessage(MessageDispatchNotification messageDispatchNotification) throws Exception 

        needs to return a QueueMessageReference.
        Add the following to get it to work better. I still need to validate that the dispatch that that happens here is correct w.r.t a normal dispatch. I have run it by applying the patch to trunk, so only with your changes to MasterBroker, ignoring the MasterBroker changes in the patch. With 30000 messages I still get errors, but of a different kind. More work needed.

                    if (message == null) {            
                        synchronized (messages) {
                            try {
                                messages.reset();
                                while (messages.hasNext()) {
                                    MessageReference node = messages.next();
        +                            node.incrementReferenceCount();
        +                            messages.remove();
                                    if (messageId.equals(node.getMessageId())) {
        +                                message = this.createMessageReference(node.getMessage());
        
        Show
        Gary Tully added a comment - sorry, my patch is still a bit of a work in progress. I was using this jira as a shared holder. For the cce, private QueueMessageReference getMatchingMessage(MessageDispatchNotification messageDispatchNotification) throws Exception needs to return a QueueMessageReference. Add the following to get it to work better. I still need to validate that the dispatch that that happens here is correct w.r.t a normal dispatch. I have run it by applying the patch to trunk, so only with your changes to MasterBroker, ignoring the MasterBroker changes in the patch. With 30000 messages I still get errors, but of a different kind. More work needed. if (message == null ) { synchronized (messages) { try { messages.reset(); while (messages.hasNext()) { MessageReference node = messages.next(); + node.incrementReferenceCount(); + messages.remove(); if (messageId.equals(node.getMessageId())) { + message = this .createMessageReference(node.getMessage());
        Hide
        ying added a comment -

        hi, Gary
        unfortunately the patch I did yesterday does not fix this issue completely. i did more load test and it failed for the same exception.

        i then used your patch and get the following with 5000 msg 16 consumers. I will try to troubleshoot this but i will appreciate your help. Thanks

        ERROR MasterBroker - Slave Failed
        java.lang.ClassCastException: org.apache.activemq.command.ActiveMQTextMessage cannot be cast to org.apache.activemq.broker.region.QueueMessageReference
        at org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:49)
        at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:242)
        at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
        at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
        at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
        at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
        at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
        at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
        at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
        at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
        at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
        at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
        ERROR MasterBroker - Slave Failed
        javax.jms.JMSException: Slave broker out of sync with master - Message: ID:yhe-1337-1236273055924-0:16:1:1:738 on queue://MasterSlaveBug does not exist among pending(200) for subscription: ID:yhe-3603-1236271682179-0:13:1:1
        at org.apache.activemq.broker.region.Queue.getMatchingMessage(Queue.java:1411)
        at org.apache.activemq.broker.region.Queue.processDispatchNotification(Queue.java:1357)
        at org.apache.activemq.broker.region.AbstractRegion.processDispatchNotificationViaDestination(AbstractRegion.java:433)
        at org.apache.activemq.broker.region.QueueRegion.processDispatchNotification(QueueRegion.java:77)
        at org.apache.activemq.broker.region.RegionBroker.processDispatchNotification(RegionBroker.java:585)
        at org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202)
        at org.apache.activemq.broker.MutableBrokerFilter.processDispatchNotification(MutableBrokerFilter.java:209)
        at org.apache.activemq.broker.TransportConnection.processMessageDispatchNotification(TransportConnection.java:466)
        at org.apache.activemq.command.MessageDispatchNotification.visit(MessageDispatchNotification.java:77)
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
        at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
        at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

        Show
        ying added a comment - hi, Gary unfortunately the patch I did yesterday does not fix this issue completely. i did more load test and it failed for the same exception. i then used your patch and get the following with 5000 msg 16 consumers. I will try to troubleshoot this but i will appreciate your help. Thanks ERROR MasterBroker - Slave Failed java.lang.ClassCastException: org.apache.activemq.command.ActiveMQTextMessage cannot be cast to org.apache.activemq.broker.region.QueueMessageReference at org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:49) at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:242) at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373) at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462) at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194) at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74) at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74) at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85) at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456) at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205) at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305) at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179) at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104) at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68) at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36) ERROR MasterBroker - Slave Failed javax.jms.JMSException: Slave broker out of sync with master - Message: ID:yhe-1337-1236273055924-0:16:1:1:738 on queue://MasterSlaveBug does not exist among pending(200) for subscription: ID:yhe-3603-1236271682179-0:13:1:1 at org.apache.activemq.broker.region.Queue.getMatchingMessage(Queue.java:1411) at org.apache.activemq.broker.region.Queue.processDispatchNotification(Queue.java:1357) at org.apache.activemq.broker.region.AbstractRegion.processDispatchNotificationViaDestination(AbstractRegion.java:433) at org.apache.activemq.broker.region.QueueRegion.processDispatchNotification(QueueRegion.java:77) at org.apache.activemq.broker.region.RegionBroker.processDispatchNotification(RegionBroker.java:585) at org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202) at org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202) at org.apache.activemq.broker.BrokerFilter.processDispatchNotification(BrokerFilter.java:202) at org.apache.activemq.broker.MutableBrokerFilter.processDispatchNotification(MutableBrokerFilter.java:209) at org.apache.activemq.broker.TransportConnection.processMessageDispatchNotification(TransportConnection.java:466) at org.apache.activemq.command.MessageDispatchNotification.visit(MessageDispatchNotification.java:77) at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305) at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179) at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104) at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68) at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
        Hide
        Gary Tully added a comment -

        This patch defers dispatch on a slave til the dispatch notification so that the slave can honor the dispatch decision made on the master. However, if the master and slave are kept in sync wrt. consumer additions it is not needed. Adding it here as a place holder in case we still need to refactor in this manner.

        Show
        Gary Tully added a comment - This patch defers dispatch on a slave til the dispatch notification so that the slave can honor the dispatch decision made on the master. However, if the master and slave are kept in sync wrt. consumer additions it is not needed. Adding it here as a place holder in case we still need to refactor in this manner.
        Hide
        Gary Tully added a comment -

        Ying, I have applied your patch. It is simple and it works. Thanks. I guess remove consumers need a similar lock, just need another test to verify. For the moment, I will shelve the refactor to defer slave dispatch till the notification as it does not seem necessary.
        Dan, thanks for the test case, that is included also, although it does not always fail, it always produced the 'slave out of sync' errors on the console.

        Show
        Gary Tully added a comment - Ying, I have applied your patch. It is simple and it works. Thanks. I guess remove consumers need a similar lock, just need another test to verify. For the moment, I will shelve the refactor to defer slave dispatch till the notification as it does not seem necessary. Dan, thanks for the test case, that is included also, although it does not always fail, it always produced the 'slave out of sync' errors on the console.
        Hide
        ying added a comment -

        thanks for the comment.

        from my approach, the attached patch seems work and I haven't seen a failure using 5000 msg and 16 consumers and i will do more load test maybe. and my master/slave are on the same machine.

        The thing that seems strange to me is the block in preProcessDispatch to sendAsyncToSlave if temporary queue and sendSyncToSlave if persistent queue makes a difference. do you see an explanation? because if no code change for this, i will only see the same error for temporary queue messages and persistent queue with the addConsumer fix is fine.

        your approach seems better, if you can have more control on the slave side. please share what you have and i may dig in that route.

        ying

        Show
        ying added a comment - thanks for the comment. from my approach, the attached patch seems work and I haven't seen a failure using 5000 msg and 16 consumers and i will do more load test maybe. and my master/slave are on the same machine. The thing that seems strange to me is the block in preProcessDispatch to sendAsyncToSlave if temporary queue and sendSyncToSlave if persistent queue makes a difference. do you see an explanation? because if no code change for this, i will only see the same error for temporary queue messages and persistent queue with the addConsumer fix is fine. your approach seems better, if you can have more control on the slave side. please share what you have and i may dig in that route. ying
        Hide
        Gary Tully added a comment -

        The master cannot know when the slave is done, as the slave does less work, it should be able to keep up, but it the slave machine or JVM is going slow, then this is a problem.

        I wonder if ensuring the order of consmer creation in the same on the master and slave is generally sufficient? I am working on having the slave dispatch when it gets a notification of dispatch from the master, the messageDispatchNotification, at which point it can dispatch to the subscription chosen by the master. This seems to work but it won't deal with a slow master.

        Show
        Gary Tully added a comment - The master cannot know when the slave is done, as the slave does less work, it should be able to keep up, but it the slave machine or JVM is going slow, then this is a problem. I wonder if ensuring the order of consmer creation in the same on the master and slave is generally sufficient? I am working on having the slave dispatch when it gets a notification of dispatch from the master, the messageDispatchNotification, at which point it can dispatch to the subscription chosen by the master. This seems to work but it won't deal with a slow master.
        Hide
        ying added a comment -

        let slave and master dispatch message is really a problem. When master sends MessageAck, how does it know the slave have finished the dispatch? If slave haven't finished, it will fail for "Unmatched acknowledege".

        Show
        ying added a comment - let slave and master dispatch message is really a problem. When master sends MessageAck, how does it know the slave have finished the dispatch? If slave haven't finished, it will fail for "Unmatched acknowledege".
        Hide
        ying added a comment -

        public Subscription addConsumer(ConnectionContext context, ConsumerInfo info) throws Exception {
        synchronized (addConsumerObj)

        { sendSyncToSlave(info); Subscription answer = super.addConsumer(context, info); return answer; }

        }

        in MasterBroker.java

        this will help but does not completely fix it. it makes sure all consumers are added in the same order to the master and slave.

        Show
        ying added a comment - public Subscription addConsumer(ConnectionContext context, ConsumerInfo info) throws Exception { synchronized (addConsumerObj) { sendSyncToSlave(info); Subscription answer = super.addConsumer(context, info); return answer; } } in MasterBroker.java this will help but does not completely fix it. it makes sure all consumers are added in the same order to the master and slave.
        Hide
        ying added a comment -

        Definitely a major issue. I just run into this pre-production and this needs to be fixed.

        Show
        ying added a comment - Definitely a major issue. I just run into this pre-production and this needs to be fixed.
        Hide
        Doug Newton added a comment -

        I just ran into this exact issue as well in pre-production testing where we are running just two consumers on a queue. This is a major issue for our production deploy.

        Show
        Doug Newton added a comment - I just ran into this exact issue as well in pre-production testing where we are running just two consumers on a queue. This is a major issue for our production deploy.

          People

          • Assignee:
            Gary Tully
            Reporter:
            Dan James
          • Votes:
            2 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development