[IMPALA-10927] TestFetchAndSpooling.test_rows_sent_counters is flaky in core-s3 based test

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: Impala 4.2.0
• Labels: broken-build flaky-test
• Target Version: Impala 4.2.0
• Epic Color: ghx-label-10
• Assignee: Qifan Chen
• Reporter: Qifan Chen
• Created: 22/Sep/21 14:44
• Updated: 02/Jul/22 03:27
• Resolved: 27/Jun/22 15:08

Description

RowsSentRate counter is seen with a value of 0 and the following is the error report. A fix in the similar area was described in IMPALA-8957, where a delay via DEBUG_ACTION BPRS_BEFORE_ADD_BATCH was introduced.

query_test.test_fetch.TestFetchAndSpooling.test_rows_sent_counters[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] (from pytest)

Failing for the past 1 build (Since Failed#206 )
Took 0.47 sec.
Error Message
query_test/test_fetch.py:101: in test_rows_sent_counters     assert float(rows_sent_rate.group(1)) > 0 E   assert 0.0 > 0 E    +  where 0.0 = float('0') E    +    where '0' = <built-in method group of _sre.SRE_Match object at 0x7f9d22693030>(1) E    +      where <built-in method group of _sre.SRE_Match object at 0x7f9d22693030> = <_sre.SRE_Match object at 0x7f9d22693030>.group
Stacktrace
query_test/test_fetch.py:101: in test_rows_sent_counters
    assert float(rows_sent_rate.group(1)) > 0
E   assert 0.0 > 0
E    +  where 0.0 = float('0')
E    +    where '0' = <built-in method group of _sre.SRE_Match object at 0x7f9d22693030>(1)
E    +      where <built-in method group of _sre.SRE_Match object at 0x7f9d22693030> = <_sre.SRE_Match object at 0x7f9d22693030>.group
Standard Error
SET client_identifier=query_test/test_fetch.py::TestFetchAndSpooling::()::test_rows_sent_counters[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|table;
-- connecting to: localhost:21000
-- connecting to localhost:21050 with impyla
-- 2021-09-10 04:41:56,852 INFO     MainThread: Closing active operation
-- connecting to localhost:28000 with impyla
-- 2021-09-10 04:41:56,925 INFO     MainThread: Closing active operation
-- connecting to localhost:11050 with impyla
-- 2021-09-10 04:41:56,939 INFO     MainThread: Could not connect to ('::1', 11050, 0, 0)

-- 2021-09-10 04:41:56,939 ERROR    MainThread: Could not connect to any of [('::1', 11050, 0, 0), ('127.0.0.1', 11050)]
-- 2021-09-10 04:41:56,939 INFO     MainThread: HS2 FENG connection setup failed, continuing...: Could not connect to any of [('::1', 11050, 0, 0), ('127.0.0.1', 11050)]
SET client_identifier=query_test/test_fetch.py::TestFetchAndSpooling::()::test_rows_sent_counters[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':0;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshold':0}|table;
SET batch_size=0;
SET num_nodes=0;
SET disable_codegen_rows_threshold=0;
SET disable_codegen=False;
SET abort_on_error=1;
SET debug_action=BPRS_BEFORE_ADD_ROWS:SLEEP@1000;
SET exec_single_node_rows_threshold=0;
-- executing against localhost:21000

select id from functional.alltypes limit 10;

-- 2021-09-10 04:41:56,988 INFO     MainThread: Started query b04941a7ffff5e31:1da6c8eb00000000
Options