[IMPALA-12670] CatalogdMetaProvider.getIfPresent() not throwing the underlying InconsistentMetadataFetchException

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0, Impala 3.3.0, Impala 3.4.0, Impala 3.4.1, Impala 4.1.0, Impala 4.2.0, Impala 4.1.1, Impala 4.1.2, Impala 4.3.0
• Fix Version/s: Impala 4.4.0
• Component/s: Frontend
• Epic Color: ghx-label-9
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 03/Jan/24 03:24
• Updated: 05/Jan/24 14:02
• Resolved: 05/Jan/24 14:02

Description

TestConcurrentDdls.test_local_catalog_ddls_with_invalidate_metadata could fail due to InconsistentMetadataFetchException:

tests/custom_cluster/test_concurrent_ddls.py:72: in test_local_catalog_ddls_with_invalidate_metadata
    self._run_ddls_with_invalidation(unique_database, sync_ddl=False)
tests/custom_cluster/test_concurrent_ddls.py:148: in _run_ddls_with_invalidation
    worker[i].get(timeout=100)
toolchain/toolchain-packages-gcc10.4.0/python-2.7.16/lib/python2.7/multiprocessing/pool.py:572: in get
    raise self._value
E   AssertionError: ImpalaBeeswaxException:
E      INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
E      MESSAGE: RuntimeException: java.util.concurrent.ExecutionException: org.apache.impala.catalog.local.InconsistentMetadataFetchException: Fetching TABLE failed. Could not find TCatalogObject(type:TABLE, catalog_version:0, table:TTable(db_name:functional, tbl_name:alltypestiny))
E     CAUSED BY: ExecutionException: org.apache.impala.catalog.local.InconsistentMetadataFetchException: Fetching TABLE failed. Could not find TCatalogObject(type:TABLE, catalog_version:0, table:TTable(db_name:functional, tbl_name:alltypestiny))
E     CAUSED BY: InconsistentMetadataFetchException: Fetching TABLE failed. Could not find TCatalogObject(type:TABLE, catalog_version:0, table:TTable(db_name:functional, tbl_name:alltypestiny))
E     
E   assert <bound method type.is_acceptable_error of <class 'test_concurrent_ddls.TestConcurrentDdls'>>("ImpalaBeeswaxException:\n INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: RuntimeException: ja...ould not find TCatalogObject(type:TABLE, catalog_version:0, table:TTable(db_name:functional, tbl_name:alltypestiny))\n", False)
E    +  where <bound method type.is_acceptable_error of <class 'test_concurrent_ddls.TestConcurrentDdls'>> = TestConcurrentDdls.is_acceptable_error
E    +  and   "ImpalaBeeswaxException:\n INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>\n MESSAGE: RuntimeException: ja...ould not find TCatalogObject(type:TABLE, catalog_version:0, table:TTable(db_name:functional, tbl_name:alltypestiny))\n" = str(ImpalaBeeswaxException())

The exception in impalad.INFO:

I0103 11:19:32.221755  8786 jni-util.cc:302] 3b476356afa44e74:6f637cfa00000000] java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.impala.catalog.local.InconsistentMetadataFetchException: Fetching TABLE failed. Could not find TCatalogObject(type:TABLE, catalog_version:0, table:TTable(db_name:functional, tbl_name:alltypestiny))
        at org.apache.impala.catalog.local.CatalogdMetaProvider.getIfPresent(CatalogdMetaProvider.java:901)
        at org.apache.impala.catalog.local.CatalogdMetaProvider.getTableIfPresent(CatalogdMetaProvider.java:750)
        at org.apache.impala.catalog.local.LocalDb.getTableIfCached(LocalDb.java:128)
        at org.apache.impala.catalog.local.LocalDb.getTable(LocalDb.java:143)
        at org.apache.impala.analysis.StmtMetadataLoader.getMissingTables(StmtMetadataLoader.java:314)
        at org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:169)
        at org.apache.impala.analysis.StmtMetadataLoader.loadTables(StmtMetadataLoader.java:145)
        at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2348)
        at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2110)
        at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1883)
        at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:169)
Caused by: java.util.concurrent.ExecutionException: org.apache.impala.catalog.local.InconsistentMetadataFetchException: Fetching TABLE failed. Could not find TCatalogObject(type:TABLE, catalog_version:0, table:TTable(db_name:functional, tbl_name:alltypestiny))
        at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
        at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
        at org.apache.impala.catalog.local.CatalogdMetaProvider.getIfPresent(CatalogdMetaProvider.java:898)
        ... 10 more
Caused by: org.apache.impala.catalog.local.InconsistentMetadataFetchException: Fetching TABLE failed. Could not find TCatalogObject(type:TABLE, catalog_version:0, table:TTable(db_name:functional, tbl_name:alltypestiny))
        at org.apache.impala.catalog.local.CatalogdMetaProvider.sendRequest(CatalogdMetaProvider.java:465)
        at org.apache.impala.catalog.local.CatalogdMetaProvider.access$100(CatalogdMetaProvider.java:199)
        at org.apache.impala.catalog.local.CatalogdMetaProvider$4.call(CatalogdMetaProvider.java:776)
        at org.apache.impala.catalog.local.CatalogdMetaProvider$4.call(CatalogdMetaProvider.java:768)
        at org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:562)
        at org.apache.impala.catalog.local.CatalogdMetaProvider.loadTable(CatalogdMetaProvider.java:764)
        at org.apache.impala.catalog.local.LocalTable.loadTableMetadata(LocalTable.java:152)
        at org.apache.impala.catalog.local.LocalTable.load(LocalTable.java:104)
        at org.apache.impala.catalog.local.LocalDb.getTable(LocalDb.java:147)
        ... 7 more

Other tests in TestConcurrentDdls that use local catalog mode could also hit the same issue, e.g. test_mixed_catalog_ddls_with_invalidate_metadata.

Issue Links

• is related to: IMPALA-12682 Consider other cache implementation in CatalogdMetaProvider
• relates to: IMPALA-7534 Handle invalidation races in CatalogdMetaProvider cache
• relates to: IMPALA-11501 Add flag to allow metadata-cache operations on masked tables