[IMPALA-9168] TestConcurrentDdls flaky on s3 (Could not resolve table reference)

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 3.4.0
• Component/s: Frontend
• Labels: broken-build flaky
• Epic Color: ghx-label-10
• Assignee: Quanlong Huang
• Reporter: Joe McDonnell
• Created: 18/Nov/19 23:12
• Updated: 07/Feb/20 23:59
• Resolved: 07/Feb/20 23:59

Description

There have been a couple failures for TestConcurrentDdls on s3. They are on different tests but have a similar symptom.

TestConcurrentDdls.test_mixed_catalog_ddls_with_invalidate_metadata sees:

custom_cluster/test_concurrent_ddls.py:59: in test_mixed_catalog_ddls_with_invalidate_metadata
    self._run_ddls_with_invalidation(unique_database, sync_ddl=False)
custom_cluster/test_concurrent_ddls.py:136: in _run_ddls_with_invalidation
    for i in pool.imap_unordered(run_ddls, xrange(1, NUM_ITERS + 1)):
/usr/lib64/python2.7/multiprocessing/pool.py:655: in next
    raise value
E   AssertionError: ImpalaBeeswaxException:
E      INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
E      MESSAGE: AnalysisException: Could not resolve table reference: 'test_mixed_catalog_ddls_with_invalidate_metadata_8d71c4db.test_10_part'
E     
E   assert <bound method type.is_acceptable_error of <class 'test_concurrent_ddls.TestConcurrentDdls'>>("ImpalaBeeswaxException:\n INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: AnalysisException: Could not resolve table reference: 'test_mixed_catalog_ddls_with_invalidate_metadata_8d71c4db.test_10_part'\n", False)
E    +  where <bound method type.is_acceptable_error of <class 'test_concurrent_ddls.TestConcurrentDdls'>> = TestConcurrentDdls.is_acceptable_error
E    +  and   "ImpalaBeeswaxException:\n INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: AnalysisException: Could not resolve table reference: 'test_mixed_catalog_ddls_with_invalidate_metadata_8d71c4db.test_10_part'\n" = str(ImpalaBeeswaxException())

TestConcurrentDdls.test_local_catalog_ddls_with_invalidate_metadata_sync_ddl sees:

custom_cluster/test_concurrent_ddls.py:83: in test_local_catalog_ddls_with_invalidate_metadata_sync_ddl
    self._run_ddls_with_invalidation(unique_database, sync_ddl=True)
custom_cluster/test_concurrent_ddls.py:136: in _run_ddls_with_invalidation
    for i in pool.imap_unordered(run_ddls, xrange(1, NUM_ITERS + 1)):
/usr/lib64/python2.7/multiprocessing/pool.py:655: in next
    raise value
E   AssertionError: ImpalaBeeswaxException:
E      INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
E      MESSAGE: AnalysisException: Table does not exist: test_local_catalog_ddls_with_invalidate_metadata_sync_ddl_b87f02d6.test_15
E     
E   assert <bound method type.is_acceptable_error of <class 'test_concurrent_ddls.TestConcurrentDdls'>>("ImpalaBeeswaxException:\n INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: AnalysisException: Table does not exist: test_local_catalog_ddls_with_invalidate_metadata_sync_ddl_b87f02d6.test_15\n", True)
E    +  where <bound method type.is_acceptable_error of <class 'test_concurrent_ddls.TestConcurrentDdls'>> = TestConcurrentDdls.is_acceptable_error
E    +  and   "ImpalaBeeswaxException:\n INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: AnalysisException: Table does not exist: test_local_catalog_ddls_with_invalidate_metadata_sync_ddl_b87f02d6.test_15\n" = str(ImpalaBeeswaxException())

These are the only two occurrences so far, and both are on s3. This test came in with "IMPALA-7506: support global INVALIDATE METADATA in local catalog mode"