Affects Version/s: 5.8.0, 5.9.0, 5.9.1, 5.10.0
Fix Version/s: 5.11.0
The test case is a simple producer/consumer:
Producer: 20 messages are injected, with a timeout of 10s.
Consumer: The redelivery policy is set to 0 retries (the issue exists with other values). The consumer uses transactions and throws a runtime exception on each message received.
queue stats show 20 enqueue, 19 dequeue, 1 pending
DLQ stat show 20 enqueue: all 20 messages go to DLQ, IDs ending in 1-10 for failing, 11-20 for expiry (approx)
the pending item (ID ending in 10) is a "ghost message" , and remains stuck indefinitely in queue.test
if you browse, the message is not shown
A) if you restart the broker, after a short while the message is cleaned:
jvm 1 | WARN | Duplicate message add attempt rejected. Destination: QUEUE://ActiveMQ.DLQ, Message id: ID:REDACTED-52872-1405079629779-1:1:1:1:10
jvm 1 | WARN | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@5b427f3c:ActiveMQ.DLQ,batchResetNeeded=false,storeHasMessages=true,size=0,cacheEnabled=true,maxBatchSize:20,hasSpace:tru
e - cursor got duplicate: ID:REDACTED--52872-1405079629779-1:1:1:1:10, 4
jvm 1 | WARN | duplicate message from store ID:REDACTED--52872-1405079629779-1:1:1:1:10, redirecting for dlq processing
B) if you purge, ActiveMQ logs a warning: "WARN | queue://queue.test after purge complete, message count stats report: 1"
the queue is marked as being empty.
however if you restart the broker, the message re-appears shorty, before being cleaned as above
SUPPLEMENTARY: with activeMQ 5.9.0 and above , if you run the injection several times, the CPU usage of ActiveMQ climbs drastically until the queue is purged.