Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2562

CTAS now requires client to explicitly close operation

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 2.3.0
    • Fix Version/s: Impala 2.3.0
    • Component/s: None

      Description

      The behavior around CTAS has changed a little in 2.3 and I don't think this was intended. In both hs2/beeswax, I've found that the client most issue a close_operation() call before closing the connection otherwise the creation will fail when the connection is closed. In 2.2 the CTAS would succeed.

      Workaround is to add an explicit call to close_operation, as below.

      import impala.dbapi
      conn = impala.dbapi.connect()
      cursor = conn.cursor()
      cursor.execute("DROP TABLE IF EXISTS foo")
      cursor.execute("CREATE TABLE foo AS SELECT * FROM tpch_100_parquet.customer")
      cursor.close_operation()   # Now required, otherwise creation will fail when the conn is closed.
      cursor.close()   # I think Impyla should be changed to call close_operation() here
      conn.close()
      

      logs

      I1016 02:15:12.613498 24189 query-exec-state.cc:855] Executing FinalizeDml() using CatalogService
      I1016 02:15:12.708907 24189 query-exec-state.cc:948] Inserted 6001215 row(s)
      I1016 02:15:12.735350 21565 impala-hs2-server.cc:656] CloseSession(): request=TCloseSessionReq {
        01: sessionHandle (struct) = TSessionHandle {
          01: sessionId (struct) = THandleIdentifier {
            01: guid (string) = "\"M\xf5:\xc9\x80I\x91\xa8I\x95U:\x92t0",
            02: secret (string) = "\xdf\xd8^_d\x0fF\xd2\xac\x1a\xdf\xddC4\xbc)",
          },
        },
      }
      I1016 02:15:12.770092 21565 status.cc:112] Session closed
          @           0xfa5d97  impala::Status::Status()
          @          0x12d618b  impala::ImpalaServer::CloseSessionInternal()
          @          0x132b815  impala::ImpalaServer::CloseSession()
          @          0x14eb7d4  apache::hive::service::cli::thrift::TCLIServiceProcessor::process_CloseSession()
          @          0x14ead74  apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
          @          0x14c51d1  impala::ImpalaHiveServer2ServiceProcessor::dispatchCall()
          @          0x12dfed1  apache::thrift::TDispatchProcessor::process()
          @          0x20a7ffa  apache::thrift::server::TThreadPoolServer::Task::run()
          @          0x2094bff  apache::thrift::concurrency::ThreadManager::Task::run()
          @          0x2097856  apache::thrift::concurrency::ThreadManager::Worker::run()
          @          0x120dcc5  impala::ThriftThread::RunRunnable()
          @          0x120f371  boost::_mfi::mf2<>::operator()()
          @          0x120f1cc  boost::_bi::list3<>::operator()<>()
          @          0x120ef5d  boost::_bi::bind_t<>::operator()()
          @          0x120ee7e  boost::detail::function::void_function_obj_invoker0<>::invoke()
          @          0x124374e  boost::function0<>::operator()()
          @          0x144aef6  impala::Thread::SuperviseThread()
          @          0x1452d93  boost::_bi::list4<>::operator()<>()
          @          0x1452cdd  boost::_bi::bind_t<>::operator()()
          @          0x1452c70  boost::detail::thread_data<>::run()
          @     0x7f9929ecc70a  thread_proxy
          @     0x7f9927cefdf5  start_thread
          @     0x7f9926baa1ad  __clone
      I1016 02:15:12.770107 21565 impala-server.cc:862] UnregisterQuery(): query_id=c34dd8a16a2b9995:f95f497cb3500eac
      I1016 02:15:12.770117 21565 impala-server.cc:943] Cancel(): query_id=c34dd8a16a2b9995:f95f497cb3500eac
      I1016 02:15:12.770133 21565 coordinator.cc:1156] Cancel() query_id=c34dd8a16a2b9995:f95f497cb3500eac
      I1016 02:15:12.770527 21565 query-exec-state.cc:530] Delete newly created table due to error in CTAS query: Session closed
      

      If a sleep() is added before the close(), the web ui shows the query state is "finished".

      Maybe if the session is closing and the CTAS has already executed successfully then the table can be kept.

      Fyi, I tried this in hue and things seemed to work. I ran a CTAS and went back to CM. The table was still visible in impala-shell. Not sure if that's a good test though.

        Attachments

          Activity

            People

            • Assignee:
              jyu@cloudera.com Juan Yu
              Reporter:
              caseyc casey
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: