[IMPALA-12084] INVALIDATE METADATA under unloaded db hit DCHECK in LocalCatalog mode

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0, Impala 4.1.0, Impala 4.2.0, Impala 4.1.1, Impala 4.1.2
• Fix Version/s: Impala 4.3.0
• Component/s: Catalog
• Target Version: Impala 4.1.3
• Epic Color: ghx-label-3
• Assignee: Quanlong Huang
• Reporter: Quanlong Huang
• Created: 23/Apr/23 09:19
• Updated: 08/May/23 23:45
• Resolved: 08/May/23 23:45

Description

When running in LocalCatalog mode, INVALIDATE METADATA under unloaded db will hit the following DCHECK:

F0423 17:12:50.566385  7565 catalog-op-executor.cc:88] 9c4c9d5f5d7974bc:b97e6dae00000000] Check failed: !obj.db.__isset.metastore_db Minimal database TCatalogObject should have empty metastore_db
*** Check failure stack trace: *** 
    @          0x37ce3ed  google::LogMessage::Fail()
    @          0x37d0324  google::LogMessage::SendToLog()
    @          0x37cddcc  google::LogMessage::Flush()
    @          0x37d0849  google::LogMessageFatal::~LogMessageFatal()
    @          0x195389d  VerifyMinimalResponse()
    @          0x195809c  impala::CatalogOpExecutor::Exec()
    @          0x16ce391  impala::ClientRequestState::ExecDdlRequestImpl()
    @          0x16d713c  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @          0x18cdb52  impala::Thread::SuperviseThread()
    @          0x18ce95b  boost::detail::thread_data<>::run()
    @          0x23cd927  thread_proxy
    @     0x7f196963f6db  start_thread
    @     0x7f19663b761f  clone

This is a bug introduced by IMPALA-9936 in which we added the DCHECK. This use case is still sending back the full DB object to the coordinator.

For reproducing the issue:

# Start Impala cluster with --hms_event_polling_interval_s=0
bin/start-impala-cluster.py --catalogd_args="--catalog_topic_mode=minimal --hms_event_polling_interval_s=0" --impalad_args=--use_local_catalog

# Create new db and new table in Hive
beeline -u "jdbc:hive2://localhost:11050"
> create database mydb;
> create table mydb.mytbl (i int);

# Run IM in Impala
impala-shell.sh
> invalidate metadata mydb.mytbl;
# The coordinator will crash at hitting the DCHECK

Issue Links

• is caused by: IMPALA-9936 Only send invalidations in DDL responses to LocalCatalog coordinators