Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Abandoned
-
5.11.4
-
None
-
None
-
None
Description
We have experienced a problem during somewhat high load (>500 000 messages over 30 minutes to multiple queues), where 2 messages was both delivered and DLQ’ed, 8 messages was delivered twice and 7 messages disappeared (but then upon inspection 6 of them is present in the AMQ database, somehow without AMQ noticing).
We are running an ActiveMQ 5.11.4 broker in JDBC Master Slave setup with MSSQL server as persistent store, with 2 slaves (i.e. hot standbys).
There was no master-switching (failover) during the incident.
We have no indication that there were problems on the MS SQL server at the time.
There are only two log lines in the ActiveMQ log at the time of the incident:
2016-08-29 06:03:54,857 [ActiveMQ BrokerService[svg-amq03_61616] Task-351933] WARN o.a.a.b.r.c.AbstractStoreCursor - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@24b740dc:stow:AgreementService.private.createOrder,batchResetNeeded=false,size=1,cacheEnabled=false,maxBatchSize:1,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=stow:AgreementService.private.createOrder,pendingSize:1 - cursor got duplicate from store ID:svg-agreement01-49217-1471461423933-1:15:1:6980:1 seq: 1233709
2016-08-29 06:03:54,857 [ActiveMQ BrokerService[svg-amq03_61616] Task-351933] WARN o.a.activemq.broker.region.Queue - duplicate message from store ID:svg-agreement01-49217-1471461423933-1:15:1:6980:1, redirecting for dlq processing
2016-08-29 06:03:54,920 [ActiveMQ BrokerService[svg-amq03_61616] Task-351926] WARN o.a.a.b.r.c.AbstractStoreCursor - org.apache.activemq.broker.region.cursors.QueueStorePrefetch@24b740dc:stow:AgreementService.private.createOrder,batchResetNeeded=false,size=2,cacheEnabled=false,maxBatchSize:2,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=stow:AgreementService.private.createOrder,pendingSize:1 - cursor got duplicate from store ID:svg-agreement01-49217-1471461423933-1:15:1:6981:1 seq: 1233746
2016-08-29 06:03:54,920 [ActiveMQ BrokerService[svg-amq03_61616] Task-351926] WARN o.a.activemq.broker.region.Queue - duplicate message from store ID:svg-agreement01-49217-1471461423933-1:15:1:6981:1, redirecting for dlq processing
The two DLQ’ed messages from the log lines were both delivered correctly, and then also DLQ’ed.
At the same time we got 8 other messages delivered twice, and 7 messages looked like they were gone. When querying the AMQ database, 6 of the 7 lost messages are present in the database, but not present when querying MBeans for the queues – leaving 1 message without a trace. (We know it was sent, due to log lines from the application).
All of this happened at the same time (during same second), and all of the problematic messages in question were on the same queue.
Any idea why this happenend, and how to avoid it the future?