[IMPALA-9135] DDLs with sync_ddl may fail with concurrent INVALIDATE METADATA

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Catalog
• Labels: concurrency
• Epic Color: ghx-label-7
• Assignee: Unassigned
• Reporter: Quanlong Huang
• Created: 08/Nov/19 01:43
• Updated: 22/May/24 18:50

Description

This can be revealed by tests/custom_cluster/test_concurrent_ddls.py added in https://gerrit.cloudera.org/c/14307

If running with INVALIDATE METADATA concurrently, the DDLs may run out of attemps in CatalogServiceCatalog.waitForSyncDdlVersion() to wait for the target update being sent, no matter how large we increase the maxNumAttempts.

The error logs:

E1107 17:34:25.092439  7353 CatalogServiceCatalog.java:2626] Couldn't retrieve the covering topic version for catalog objects. Updated objects: [TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_9_part version: 349], deleted objects: []
I1107 17:34:25.093451  7353 jni-util.cc:288] org.apache.impala.catalog.CatalogException: Couldn't retrieve the catalog topic version for the SYNC_DDL operation after 5 attempts.The operation has been successfully executed but its effects may have not been broadcast to all the coordinators.
        at org.apache.impala.catalog.CatalogServiceCatalog.waitForSyncDdlVersion(CatalogServiceCatalog.java:2630)
        at org.apache.impala.service.CatalogOpExecutor.execDdlRequest(CatalogOpExecutor.java:414)
        at org.apache.impala.service.JniCatalog.execDdl(JniCatalog.java:167)
I1107 17:34:25.142006  6389 catalog-server.cc:337] A catalog update with 2 entries is assembled. Catalog version: 356 Last sent catalog version: 355
I1107 17:34:25.142168  6381 catalog-server.cc:641] Collected update: 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_15_part, version=357, original size=101, compressed size=98
I1107 17:34:25.142215  6381 catalog-server.cc:641] Collected update: 1:CATALOG_SERVICE_ID, version=357, original size=49, compressed size=52
I1107 17:34:25.142287  7356 CatalogServiceCatalog.java:2642] Operation using SYNC_DDL is waiting for catalog topic version: 357. Time to identify topic version (msec): 19
I1107 17:34:25.192239  6389 catalog-server.cc:337] A catalog update with 2 entries is assembled. Catalog version: 357 Last sent catalog version: 356
I1107 17:34:25.192428  6381 catalog-server.cc:641] Collected update: 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_16_part, version=358, original size=101, compressed size=98
I1107 17:34:25.192462  6381 catalog-server.cc:641] Collected update: 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_11_part, version=359, original size=101, compressed size=98
I1107 17:34:25.192484  6381 catalog-server.cc:641] Collected update: 1:TABLE:test_ddls_with_invalidate_metadata_sync_ddl_f41e97e6.test_12_part, version=360, original size=101, compressed size=98
I1107 17:34:25.192535  6381 catalog-server.cc:641] Collected update: 1:CATALOG_SERVICE_ID, version=360, original size=49, compressed size=52
I1107 17:34:25.192613  7355 CatalogServiceCatalog.java:2642] Operation using SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic version (msec): 13
I1107 17:34:25.192695  7351 CatalogServiceCatalog.java:2642] Operation using SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic version (msec): 45
I1107 17:34:25.192734  7350 CatalogServiceCatalog.java:2642] Operation using SYNC_DDL is waiting for catalog topic version: 360. Time to identify topic version (msec): 29
I1107 17:34:25.222911  7353 status.cc:126] CatalogException: Couldn't retrieve the catalog topic version for the SYNC_DDL operation after 5 attempts.The operation has been successfully executed but its effects may have not been broadcast to all the coordinators.
    @          0x1c5ae50  impala::Status::Status()
    @          0x24f7ad2  impala::JniUtil::GetJniExceptionMsg()
    @          0x1c41987  impala::JniCall::Call<>()
    @          0x1c3fec9  impala::JniUtil::CallJniMethod<>()
    @          0x1c3e0e6  impala::Catalog::ExecDdl()
    @          0x1c1ed17  CatalogServiceThriftIf::ExecDdl()
    @          0x1cb3047  impala::CatalogServiceProcessor::process_ExecDdl()
    @          0x1cb2d95  impala::CatalogServiceProcessor::dispatchCall()
    @          0x1c08d65  apache::thrift::TDispatchProcessor::process()
    @          0x20e8a0d  apache::thrift::server::TAcceptQueueServer::Task::run()
    @          0x20de040  impala::ThriftThread::RunRunnable()
    @          0x20df766  boost::_mfi::mf2<>::operator()()
    @          0x20df5fc  boost::_bi::list3<>::operator()<>()
    @          0x20df348  boost::_bi::bind_t<>::operator()()
    @          0x20df25b  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @          0x1ffb6e9  boost::function0<>::operator()()
    @          0x2573dea  impala::Thread::SuperviseThread()
    @          0x257c16e  boost::_bi::list5<>::operator()<>()
    @          0x257c092  boost::_bi::bind_t<>::operator()()
    @          0x257c055  boost::detail::thread_data<>::run()
    @          0x3d61599  thread_proxy
    @     0x7f1ce6ca46b9  start_thread
    @     0x7f1ce343f41c  clone
E1107 17:34:25.222932  7353 catalog-server.cc:112] CatalogException: Couldn't retrieve the catalog topic version for the SYNC_DDL operation after 5 attempts.The operation has been successfully executed but its effects may have not been broadcast to all the coordinators.

Issue Links

• is related to: IMPALA-8877 CatalogException during stress test: Table <foo> modified while operation was in progress
• requires: IMPALA-9214 REFRESH with sync_ddl may fail with concurrent INVALIDATE METADATA
• requires: IMPALA-9211 CreateTable with sync_ddl may fail with concurrent INVALIDATE METADATA