With large volumes of messages, some messages can arrive out of order on occasion. Scenario is hard to reproduce but a single producer and single consumer can cause the problem if the volume is sufficiently high. The Producer needs to exceed the memory limit such that the cache is filled and when the broker moves over to the store from the cache, the consumer need to ack a message that will be redelivered from the store but would normally be suppressed as a duplicate.
One way to avoid the problem is to disable the cache for a Queue:
<amq:policyEntry queue=">" useCache="false" >
Description
With large volumes of messages, some messages can arrive out of order on occasion. Scenario is hard to reproduce but a single producer and single consumer can cause the problem if the volume is sufficiently high. The Producer needs to exceed the memory limit such that the cache is filled and when the broker moves over to the store from the cache, the consumer need to ack a message that will be redelivered from the store but would normally be suppressed as a duplicate.
One way to avoid the problem is to disable the cache for a Queue:
two problems, the test case with the commit shows both.
there was an inadvertent audit rollback on every ack which meant the audit did not suppress all duplicates.
Under high load, way too many duplicates were being produced by the reference store. A queue caches its received messages up to a point and dispatches from this cache. However once the cache is exhausts, all remaining messages in the store are re dispatched. The resolution is to set the batch start point once the cache is full such that resuming after the cache is exhausted begins with the first non cached message rather than with the current start of the store.
The additional setBatch method that is present in the KahaStore probably needs to be pulled up into the MessageStore interface. currently there is a cast down in the cursor. If this solution proves generally applicable we can do this.
Gary Tully added a comment - 03/Dec/08 10:04 AM two problems, the test case with the commit shows both.
there was an inadvertent audit rollback on every ack which meant the audit did not suppress all duplicates.
Under high load, way too many duplicates were being produced by the reference store. A queue caches its received messages up to a point and dispatches from this cache. However once the cache is exhausts, all remaining messages in the store are re dispatched. The resolution is to set the batch start point once the cache is full such that resuming after the cache is exhausted begins with the first non cached message rather than with the current start of the store.
The additional setBatch method that is present in the KahaStore probably needs to be pulled up into the MessageStore interface. currently there is a cast down in the cursor. If this solution proves generally applicable we can do this.
Bug
Resolved
Major
Messages are arriving out of sequence with large message volumes
there was an inadvertent audit rollback on every ack which meant the audit did not suppress all duplicates.
Under high load, way too many duplicates were being produced by the reference store. A queue caches its received messages up to a point and dispatches from this cache. However once the cache is exhausts, all remaining messages in the store are re dispatched. The resolution is to set the batch start point once the cache is full such that resuming after the cache is exhausted begins with the first non cached message rather than with the current start of the store.
The additional setBatch method that is present in the KahaStore probably needs to be pulled up into the MessageStore interface. currently there is a cast down in the cursor. If this solution proves generally applicable we can do this.