[QPID-5975] HA extra/missing messages when running qpid-txtest2 in a loop with failover.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.28
• Fix Version/s: 0.31
• Component/s: C++ Clustering
• Assignee: Alan Conway
• Reporter: Alan Conway
• Created: 08/Aug/14 07:33
• Updated: 19/Mar/15 12:22
• Resolved: 28/Aug/14 21:50

Description

Running qpid-txtest2 in a loop against a 3 node cluster while killing primary brokers to force failover eventually results in the test failing due to extra or missing messages.

To reproduce:

In separate windows:
$ while ha stat; qpid-txtest2 -b 20.0.20.200 --total-messages 100 --connection-options '

{reconnect:true}

' --messages-per-tx 10 --tx-count 10; do true; done
$ while ha wait -a 2; do sleep .5; ha kill; done # ha script attached.

Results:

Eventually (up to 10 minutes) the test fails with a message like:

The following ids were missing:
'msg-100'
'msg-51'
'msg-52'
'msg-53'

OR

The following extra ids were encountered:
'msg-1'
'msg-1'
'msg-10'

Expected result:

Test runs forever, no failures.