[IMPALA-13003] Server exits early failing to create impala_query_log with AlreadyExistsException

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 4.4.0
• Fix Version/s: Impala 4.4.0
• Component/s: Backend
• Labels: iceberg workload-management
• Epic Link: Workload Management
• Epic Color: ghx-label-6
• Assignee: Michael Smith
• Reporter: Andrew Sherman
• Created: 15/Apr/24 20:30
• Updated: 16/Jul/24 23:50
• Resolved: 17/Apr/24 17:29

Description

At startup workload management tries to create the query log table here:

  // The initialization code only works when run in a separate thread for reasons unknown.
  ABORT_IF_ERROR(SetupDbTable(internal_server_.get(), table_name));

This code is exiting:

I0413 23:40:05.183876 21006 client-request-state.cc:1348] 1d4878dbc9214c81:6dc8cc2e00000000] ImpalaRuntimeException: Error making 'createTable' RPC to Hive Metastore:
CAUSED BY: AlreadyExistsException: Table was created concurrently: sys.impala_query_log
I0413 23:40:05.184055 20955 impala-server.cc:2582] Connection 27432606d99dcdae:218860164eb206bb from client in-memory.localhost:0 to server internal-server closed. The connection had 1 associated session(s).
I0413 23:40:05.184067 20955 impala-server.cc:1780] Closing session: 27432606d99dcdae:218860164eb206bb
I0413 23:40:05.184083 20955 impala-server.cc:1836] Closed session: 27432606d99dcdae:218860164eb206bb, client address: <in-memory.localhost:0>.
F0413 23:40:05.184111 20955 workload-management.cc:304] query timed out waiting for results
. Impalad exiting.
I0413 23:40:05.184728 20883 impala-server.cc:1564] Query successfully unregistered: query_id=1d4878dbc9214c81:6dc8cc2e00000000
Minidump in thread [20955]completed-queries running query 0000000000000000:0000000000000000, fragment instance 0000000000000000:0000000000000000
Wrote minidump to /data/jenkins/workspace/impala-cdw-master-core-ubsan/repos/Impala/logs/custom_cluster_tests/minidumps/impalad/402f37cc-4663-4c78-086ca295-a9e5943c.dmp

with stack

F0413 23:40:05.184111 20955 workload-management.cc:304] query timed out waiting for results
. Impalad exiting.
*** Check failure stack trace: ***
    @          0x8e96a4d  google::LogMessage::Fail()
    @          0x8e98984  google::LogMessage::SendToLog()
    @          0x8e9642c  google::LogMessage::Flush()
    @          0x8e98ea9  google::LogMessageFatal::~LogMessageFatal()
    @          0x3da3a9a  impala::ImpalaServer::CompletedQueriesThread()
    @          0x3a8df93  boost::_mfi::mf0<>::operator()()
    @          0x3a8de97  boost::_bi::list1<>::operator()<>()
    @          0x3a8dd77  boost::_bi::bind_t<>::operator()()
    @          0x3a8d672  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @          0x301e7d0  boost::function0<>::operator()()
    @          0x43ce415  impala::Thread::SuperviseThread()
    @          0x43e2dc7  boost::_bi::list5<>::operator()<>()
    @          0x43e29e7  boost::_bi::bind_t<>::operator()()
    @          0x43e21c5  boost::detail::thread_data<>::run()
    @          0x7984c37  thread_proxy
    @     0x7f75b6982ea5  start_thread
    @     0x7f75b36a7b0d  __clone
Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,address=30000,server=y,suspend=n   -Dsun.java.command=impalad
Minidump in thread [20955]completed-queries running query 0000000000000000:0000000000000000, fragment instance 0000000000000000:0000000000000000

I think the key error is

CAUSED BY: AlreadyExistsException: Table was created concurrently: sys.impala_query_log

which suggests that creating the table with "if not exists" is not sufficient to protect against concurrent creations.