[IMPALA-10363] test_mixed_catalog_ddls_with_invalidate_metadata failed after reaching timeout (120 seconds)

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Catalog
• Labels: broken-build flaky-test
• Epic Color: ghx-label-3
• Assignee: Quanlong Huang
• Reporter: Fang-Yu Rao
• Created: 28/Nov/20 01:13
• Updated: 12/Jul/21 14:10

Description

We found in an s3 run that the E2E test test_mixed_catalog_ddls_with_invalidate_metadata at https://github.com/apache/impala/blob/6539c50f6611d9e2ba9c3051b8e4c8bd6e79d1e1/tests/custom_cluster/test_concurrent_ddls.py#L58-L59 could fail after reaching timeout of 120 seconds with the following error message.

Error Message
Failed: Timeout >120.0s
Stacktrace
custom_cluster/test_concurrent_ddls.py:59: in test_mixed_catalog_ddls_with_invalidate_metadata
    self._run_ddls_with_invalidation(unique_database, sync_ddl=False)
custom_cluster/test_concurrent_ddls.py:146: in _run_ddls_with_invalidation
    for i in pool.imap_unordered(run_ddls, xrange(1, NUM_ITERS + 1)):
/usr/lib64/python2.7/multiprocessing/pool.py:642: in next
    self._cond.wait(timeout)
/usr/lib64/python2.7/threading.py:339: in wait
    waiter.acquire()
E   Failed: Timeout >120.0s

Error Message
Failed: Timeout >120.0s
Stacktrace
custom_cluster/test_concurrent_ddls.py:59: in test_mixed_catalog_ddls_with_invalidate_metadata
    self._run_ddls_with_invalidation(unique_database, sync_ddl=False)
custom_cluster/test_concurrent_ddls.py:146: in _run_ddls_with_invalidation
    for i in pool.imap_unordered(run_ddls, xrange(1, NUM_ITERS + 1)):
/usr/lib64/python2.7/multiprocessing/pool.py:642: in next
    self._cond.wait(timeout)
/usr/lib64/python2.7/threading.py:339: in wait
    waiter.acquire()
E   Failed: Timeout >120.0s

Based on the following two lines in the console output of the Jenkins job, it looks like the query id of the failed query is

08:20:05 custom_cluster/test_concurrent_ddls.py::TestConcurrentDdls::test_mixed_catalog_ddls_with_invalidate_metadata -- 2020-11-27 08:20:05,777 INFO     Thread-85: Started query 4b4d32c23dedf060:21e898dd00000000
08:21:14 FAILED

It looks like there were multiple queries being executed during this test. In this regard, I provide the corresponding log files generated when the test was executed. Let me know if other log files are needed as well. Thanks!

This test was recently added in IMPALA-7506. Maybe stigahuang could provide some insight into it.

This JIRA may also be related to IMPALA-10301.

Issue Links

• is related to: IMPALA-10301 Insert query hangs in test_local_catalog_ddls_with_invalidate_metadata_sync_ddl
• is related to: IMPALA-10369 Dump server stacktraces when test_concurrent_ddls.py timeout