[IMPALA-12550] test_statestored_auto_failover_with_disabling_network flaky

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.4.0
• Fix Version/s: Impala 4.4.0
• Component/s: Backend
• Epic Color: ghx-label-8
• Assignee: Wenzhe Zhou
• Reporter: Wenzhe Zhou
• Created: 08/Nov/23 16:45
• Updated: 13/Nov/23 21:13
• Resolved: 10/Nov/23 03:24

Description

TestStatestoredHA.test_statestored_auto_failover_with_disabling_network failed with following stack trace when repeatedly run this test.
tests/custom_cluster/test_statestored_ha.py:645: in test_statestored_auto_failover_with_disabling_network
"statestore.in-ha-recovery-mode", expected_value=False, timeout=120)
tests/common/impala_service.py:144: in wait_for_metric_value
self.__metric_timeout_assert(metric_name, expected_value, timeout)
tests/common/impala_service.py:213: in __metric_timeout_assert
assert 0, assert_string
E AssertionError: Metric statestore.in-ha-recovery-mode did not reach value False in 120s.

From log messages, the issue was caused by the delay of HA Handshake RPC between two statestore instances. Sometimes the active statestore took a few minutes to response the handshake requests from standby statestore.

This issue is different from IMPALA-12525, which was caused locking issue on subscribers side.

Issue Links

• is caused by: IMPALA-12156 Support Impala Statestore HA