Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Won't Fix
-
5.13.2
-
None
-
None
-
a 3 node cluster running on RHEL 6.7
Description
In this 3 node cluster running ActiveMQ 5.13.2 and levelDB, we tested failover with a producer that generates 200 AccountIds ( from 0 to 199 ) , and expected that the consumer would receive the same number of messages after a triggered failover ( kill -9 master-process-id).
The producer used syncSend, the consumer used transactional session, and the destination is a queue.
We printed out the message sequence and its corresponding accountID on both the producer and the consumer consoles after a message was successful send and receive respectively.
The result showed that the producer sent 199 messages ( one failed due to fail over), but the consumer received much less messages than 199.
For example, on the consumer console, before the failover we had
......
received = 13 accountid=12
received = 14 accountid=13
then, right after the failover the consumer console had
received = 15 accountid=44
received = 16 accountid=45
...
Notice that accounted from 14 to 43 are ever lost.
We scan the broker logs of both the new and old master and noticed that the pagedInPendingDispatch.size in the "old" master had value = 30 which matches the number of lost messages. We repeated this test few times and each time this size matched the number of lost messages.
The configuration we used is listed below:
Broker configuration:
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="broker" dataDirectory="${activemq.data}" useJmx="true" useShutdownHook="false" shutdownOnMasterFailure="true" systemExitOnShutdown="true" systemExitOnShutdownExitCode="200">
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" >
<pendingMessageLimitStrategy>
<constantPendingMessageLimitStrategy limit="1000"/>
</pendingMessageLimitStrategy>
</policyEntry>
<policyEntry queue=">" useCache="false" expireMessagesPeriod="0" />
</policyEntries>
</policyMap>
</destinationPolicy> <persistenceAdapter>
<replicatedLevelDB
directory="${activemq.base}/leveldb-data"
replicas="3"
bind="tcp://0.0.0.0:0"
zkAddress="test1:2181,test2:2181,test3:2181"
zkPath="/activemq/leveldb"
sync="quorum_disk"
paranoidChecks="true"
logCompression="snappy"
/>
</persistenceAdapter>