Details
-
Bug
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
Impala 2.3.0
-
None
Description
The behavior around CTAS has changed a little in 2.3 and I don't think this was intended. In both hs2/beeswax, I've found that the client most issue a close_operation() call before closing the connection otherwise the creation will fail when the connection is closed. In 2.2 the CTAS would succeed.
Workaround is to add an explicit call to close_operation, as below.
import impala.dbapi
conn = impala.dbapi.connect()
cursor = conn.cursor()
cursor.execute("DROP TABLE IF EXISTS foo")
cursor.execute("CREATE TABLE foo AS SELECT * FROM tpch_100_parquet.customer")
cursor.close_operation() # Now required, otherwise creation will fail when the conn is closed.
cursor.close() # I think Impyla should be changed to call close_operation() here
conn.close()
logs
I1016 02:15:12.613498 24189 query-exec-state.cc:855] Executing FinalizeDml() using CatalogService
I1016 02:15:12.708907 24189 query-exec-state.cc:948] Inserted 6001215 row(s)
I1016 02:15:12.735350 21565 impala-hs2-server.cc:656] CloseSession(): request=TCloseSessionReq {
01: sessionHandle (struct) = TSessionHandle {
01: sessionId (struct) = THandleIdentifier {
01: guid (string) = "\"M\xf5:\xc9\x80I\x91\xa8I\x95U:\x92t0",
02: secret (string) = "\xdf\xd8^_d\x0fF\xd2\xac\x1a\xdf\xddC4\xbc)",
},
},
}
I1016 02:15:12.770092 21565 status.cc:112] Session closed
@ 0xfa5d97 impala::Status::Status()
@ 0x12d618b impala::ImpalaServer::CloseSessionInternal()
@ 0x132b815 impala::ImpalaServer::CloseSession()
@ 0x14eb7d4 apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseSession()
@ 0x14ead74 apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
@ 0x14c51d1 impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
@ 0x12dfed1 apache::thrift::TDispatchProcessor::process()
@ 0x20a7ffa apache::thrift::server::TThreadPoolServer::Task::run()
@ 0x2094bff apache::thrift::concurrency::ThreadManager::Task::run()
@ 0x2097856 apache::thrift::concurrency::ThreadManager::Worker::run()
@ 0x120dcc5 impala::ThriftThread::RunRunnable()
@ 0x120f371 boost::_mfi::mf2<>::operator()()
@ 0x120f1cc boost::_bi::list3<>::operator()<>()
@ 0x120ef5d boost::_bi::bind_t<>::operator()()
@ 0x120ee7e boost::detail::function::void_function_obj_invoker0<>::invoke()
@ 0x124374e boost::function0<>::operator()()
@ 0x144aef6 impala::Thread::SuperviseThread()
@ 0x1452d93 boost::_bi::list4<>::operator()<>()
@ 0x1452cdd boost::_bi::bind_t<>::operator()()
@ 0x1452c70 boost::detail::thread_data<>::run()
@ 0x7f9929ecc70a thread_proxy
@ 0x7f9927cefdf5 start_thread
@ 0x7f9926baa1ad __clone
I1016 02:15:12.770107 21565 impala-server.cc:862] UnregisterQuery(): query_id=c34dd8a16a2b9995:f95f497cb3500eac
I1016 02:15:12.770117 21565 impala-server.cc:943] Cancel(): query_id=c34dd8a16a2b9995:f95f497cb3500eac
I1016 02:15:12.770133 21565 coordinator.cc:1156] Cancel() query_id=c34dd8a16a2b9995:f95f497cb3500eac
I1016 02:15:12.770527 21565 query-exec-state.cc:530] Delete newly created table due to error in CTAS query: Session closed
If a sleep() is added before the close(), the web ui shows the query state is "finished".
Maybe if the session is closing and the CTAS has already executed successfully then the table can be kept.
Fyi, I tried this in hue and things seemed to work. I ran a CTAS and went back to CM. The table was still visible in impala-shell. Not sure if that's a good test though.