Details
Major Description
Please see this arch diagram: http://activemq.2283324.n4.nabble.com/file/n3762364/ActiveMQ_-_Composite_Queues.png
There are two brokers: broker1 & broker2. Messages arriving at Q.Index.Transit.DC2 on broker1 is forwarded to broker2. Configuration files for both brokers are below.
Scenario:
1. Start both brokers in separate consoles
2. Send 3000 messages in AcknowledgementMode.Transactional to broker1 (Q.Index.Replication).
3. While messages are flowing kill (shutdown console) broker2 and then broker1.
4. Start broker1 and then broker2.
5. Repeat 3&4 while messages are still flowing.
6. When all messages are published from producer to broker1, observe message counts in broker1 Transit and broker2 Indexing queues.
You will see that both admin console and JConsole reporting more messages than there actually are in Q.A.Indexing and Q.B.Indexing queues.
Take a look at this screenshot: http://activemq.2283324.n4.nabble.com/file/n3762364/ActiveMQ_-_Admin_Console.png
All queues had 3003 messages (instead of 3000). When I consumed all messages in Q.A.Indexing with my consumer, it successfully consumed 3000 messages (as expected) but admin console still reports there are additional 3 messages pending in the queue. When I click on "Browse" to inspect the messages in admin console it reports that there are no messages. When I restart this broker, pending message count corrects itself and reports 0.
This is a problem for the monitoring and operations: There is no way of knowing whether there are really 3 messages left and consumer is experiencing problems or consumer is alive but there are no messages to consume.
Here are the configuration files for both brokers:
Broker1 configuration: http://activemq.2283324.n4.nabble.com/file/n3762374/activemq_-_broker1.xml
Broker2 configuration: http://activemq.2283324.n4.nabble.com/file/n3762374/activemq_-_broker2.xml