[IMPALA-13170] InconsistentMetadataFetchException due to database dropped when showing databases

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 3.4.0
• Fix Version/s: Impala 4.5.0, Impala 4.4.1
• Component/s: Catalog
• Epic Color: ghx-label-14
• Assignee: Quanlong Huang
• Reporter: Yida Wu
• Created: 20/Jun/24 04:41
• Updated: 14/Aug/24 00:53
• Resolved: 01/Jul/24 09:22

Description

Using impalad 3.4.0, an InconsistentMetadataFetchException occurs when running "show databases" in Impala while simultaneously executing "drop database" to drop the newly created database in Hive.
Step is:
1, Creates database (Hive)
2, Creates tables (Hive)
3, Drops tables (Hive)
4, Run show databases (Impala) Drop database (Hive)
Logs in Impalad:

I0610 02:18:32.435815 278475 CatalogdMetaProvider.java:1354] 11111:22222] Invalidated objects in cache: [list of database names, HMS_METADATA for DB test_hive]
I0610 02:18:32.436224 278475 jni-util.cc:288] 11111:22222] org.apache.impala.catalog.local.InconsistentMetadataFetchException: Fetching DATABASE failed. Could not find TCatalogObject(type:DATABASE, catalog_version:0, db:TDatabase(db_name:test_hive))                                                                                                                                                                                                                                   
  at org.apache.impala.catalog.local.CatalogdMetaProvider.sendRequest(CatalogdMetaProvider.java:424)
  at org.apache.impala.catalog.local.CatalogdMetaProvider.access$100(CatalogdMetaProvider.java:185)
  at org.apache.impala.catalog.local.CatalogdMetaProvider$2.call(CatalogdMetaProvider.java:643)
  at org.apache.impala.catalog.local.CatalogdMetaProvider$2.call(CatalogdMetaProvider.java:638)
  at org.apache.impala.catalog.local.CatalogdMetaProvider.loadWithCaching(CatalogdMetaProvider.java:521)
  at org.apache.impala.catalog.local.CatalogdMetaProvider.loadDb(CatalogdMetaProvider.java:635)
  at org.apache.impala.catalog.local.LocalDb.getMetaStoreDb(LocalDb.java:91) 
  at org.apache.impala.catalog.local.LocalDb.getOwnerUser(LocalDb.java:294)
  at org.apache.impala.service.Frontend.getDbs(Frontend.java:1066)
  at org.apache.impala.service.JniFrontend.getDbs(JniFrontend.java:301)
I0610 02:18:32.436257 278475 status.cc:129] 11111:22222] InconsistentMetadataFetchException: Fetching DATABASE failed. Could not find TCatalogObject(type:DATABASE, catalog_version:0, 

Logs in Catalog:

I0610 02:18:16.190133 222885 MetastoreEvents.java:505] EventId: 141467532 EventType: CREATE_DATABASE Successfully added database test_hive 
...
I0610 02:18:32.276082 222885 MetastoreEvents.java:516] EventId: 141467562 EventType: DROP_DATABASE Creating event 141467562 of type DROP_DATABASE on database test_hive
I0610 02:18:32.277876 222885 MetastoreEvents.java:254] Total number of events received: 6 Total number of events filtered out: 0
I0610 02:18:32.277910 222885 MetastoreEvents.java:258] Incremented skipped metric to 2564
I0610 02:18:32.279537 222885 MetastoreEvents.java:505] EventId: 141467562 EventType: DROP_DATABASE Removed Database test_hive

The case is similar to IMPALA-9441. We may want to handle the error in a better way in Frontend.getDbs().

Issue Links

• relates to: IMPALA-9441 TestHS2.test_get_schemas is flaky in local catalog mode