[QPID-4138] HA sporadic failure in ha_tests

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.17
• Fix Version/s: 0.18
• Component/s: C++ Clustering
• Assignee: Alan Conway
• Reporter: Alan Conway
• Created: 13/Jul/12 19:12
• Updated: 29/Jul/13 18:53
• Resolved: 19/Jul/12 19:21

Description

Sporadic failure in ha_tests. 2 types of failure observed:

==== failure 1
ha_tests.LongTests.test_failover_send_receive ....................................... start
Traceback (most recent call last):
File "/home/aconway/qpidha/qpid/cpp/src/tests/ha_tests.py", line 884, in test_failover_send_receive
brokers[(i+1)%3].wait_status("ready")
File "/home/aconway/qpidha/qpid/cpp/src/tests/ha_tests.py", line 117, in wait_status
assert retry(try_get_status, timeout=20), "%s status != %r"%(self, status)
AssertionError: Broker<003:cluster0-0.log qpidd-23 :58163> status != 'ready'
ha_tests.LongTests.test_failover_send_receive ....................................... fail
Error during test: Traceback (most recent call last):
File "/home/aconway/qpidha/opt/src/tests/python/commands/qpid-python-test", line 340, in run
phase()
File "/home/aconway/qpidha/qpid/cpp/src/tests/ha_tests.py", line 884, in test_failover_send_receive
brokers[(i+1)%3].wait_status("ready")
File "/home/aconway/qpidha/qpid/cpp/src/tests/ha_tests.py", line 117, in wait_status
assert retry(try_get_status, timeout=20), "%s status != %r"%(self, status)
AssertionError: Broker<003:cluster0-0.log qpidd-23 :58163> status != 'ready'

==== failure 2

ha_tests.LongTests.test_failover_send_receive ........................................ start
Traceback (most recent call last):
File "/home/aconway/qpidha/qpid/cpp/src/tests/ha_tests.py", line 884, in test_failover_send_receive
brokers[(i+1)%3].wait_status("ready")
File "/home/aconway/qpidha/qpid/cpp/src/tests/ha_tests.py", line 117, in wait_status
assert retry(try_get_status, timeout=20), "%s status != %r"%(self, status)
AssertionError: Broker<003:cluster0-0.log qpidd-23 :47682> status != 'ready'
ha_tests.LongTests.test_failover_send_receive ........................................ fail
Error during test: Traceback (most recent call last):
File "/home/aconway/qpidha/opt/src/tests/python/commands/qpid-python-test", line 340, in run
phase()
File "/home/aconway/qpidha/qpid/cpp/src/tests/ha_tests.py", line 884, in test_failover_send_receive
brokers[(i+1)%3].wait_status("ready")
File "/home/aconway/qpidha/qpid/cpp/src/tests/ha_tests.py", line 117, in wait_status
assert retry(try_get_status, timeout=20), "%s status != %r"%(self, status)
AssertionError: Broker<003:cluster0-0.log qpidd-23 :47682> status != 'ready'

==== failure 3

ha_tests.ReplicationTests.test_send_receive ......................................... fail
Error during test: Traceback (most recent call last):
File "/home/aconway/qpidha/opt/src/tests/python/commands/qpid-python-test", line 340, in run
phase()
File "/home/aconway/qpidha/qpid/cpp/src/tests/ha_tests.py", line 382, in test_send_receive
brokers[2].assert_browse_backup("q", expect, transform=sn)
File "/home/aconway/qpidha/qpid/cpp/src/tests/ha_tests.py", line 158, in assert_browse_backup
assert_browse_retry(bs, queue, expected, **kwargs)
File "/home/aconway/qpidha/qpid/cpp/src/tests/brokertest.py", line 466, in assert_browse_retry
assert expect_contents == actual_contents, msg
AssertionError: browse failed: [991L, 992L, 993L, 994L, 995L, 996L, 997L, 998L, 999L, 1000L] != [990, 991, 992, 993, 994, 995, 996, 997, 998, 999, 1000]