[IMPALA-10038] TestScannersFuzzing::()::test_fuzz_alltypes timed out after 2 hours (may be flaky)

• Type: Bug
• Status: Reopened
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: broken-build
• Environment: Centos 7.4, 16 vCPU, 64 GB RAM, data cache enabled
• Epic Color: ghx-label-8
• Assignee: Quanlong Huang
• Reporter: Laszlo Gaal
• Created: 01/Aug/20 22:23
• Updated: 13/Dec/22 00:02

Description

This was seen on Centos 7., with the data cache enabled during and exhaustive run

Test step:

query_test.test_scanners_fuzz.TestScannersFuzzing.test_fuzz_alltypes[protocol: beeswax | exec_option: {'debug_action': '-1:OPEN:SET_DENY_RESERVATION_PROBABILITY@1.0', 'abort_on_error': False, 'mem_limit': '512m', 'num_nodes': 0} | table_format: avro/none]

Test backtrace:

query_test/test_scanners_fuzz.py:82: in test_fuzz_alltypes
    self.run_fuzz_test(vector, src_db, table_name, unique_database, table_name)
query_test/test_scanners_fuzz.py:238: in run_fuzz_test
    result = self.execute_query(query, query_options = query_options)
common/impala_test_suite.py:811: in wrapper
    return function(*args, **kwargs)
common/impala_test_suite.py:843: in execute_query
    return self.__execute_query(self.client, query, query_options)
common/impala_test_suite.py:909: in __execute_query
    return impalad_client.execute(query, user=user)
common/impala_connection.py:205: in execute
    return self.__beeswax_client.execute(sql_stmt, user=user)
beeswax/impala_beeswax.py:187: in execute
    handle = self.__execute_query(query_string.strip(), user=user)
beeswax/impala_beeswax.py:365: in __execute_query
    self.wait_for_finished(handle)
beeswax/impala_beeswax.py:389: in wait_for_finished
    time.sleep(0.05)
E   Failed: Timeout >7200s

Captured stderr:

~~~~~~~~~~~~~~~~~~~~~ Stack of <unknown> (139967888086784) ~~~~~~~~~~~~~~~~~~~~~
  File "/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/execnet/gateway_base.py", line 277, in _perform_spawn
    reply.run()
  File "/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/execnet/gateway_base.py", line 213, in run
    self._result = func(*args, **kwargs)
  File "/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/execnet/gateway_base.py", line 954, in _thread_receiver
    msg = Message.from_io(io)
  File "/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/execnet/gateway_base.py", line 418, in from_io
    header = io.read(9)  # type 1, channel 4, payload 4
  File "/data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc7.5.0/lib/python2.7/site-packages/execnet/gateway_base.py", line 386, in read
    data = self._read(numbytes-len(buf))
ERROR:impala_test_suite:Should not throw error when abort_on_error=0: 'Timeout >7200s'