[IMPALA-6728] The combination use_kudu_kinit=false and use_krpc=true crashes Impalad

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 3.0, Impala 2.12.0
• Fix Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Security
• Labels: crash
• Target Version: Impala 2.12.0
• Epic Color: ghx-label-1
• Assignee: Michael Ho
• Reporter: Michael Ho
• Created: 23/Mar/18 02:06
• Updated: 27/Mar/18 19:56
• Resolved: 27/Mar/18 19:56

Description

Although this is a weird combination, it's possible to start Impala with the configuration combination use_kudu_kinit=false and use_krpc=true. In which case, Impala will crash with the following backtrace:

 

(gdb) bt
#0  0x00007f8b2f484885 in raise () from /lib64/libc.so.6
#1  0x00007f8b2f485e61 in abort () from /lib64/libc.so.6
#2  0x00007f8b31a6bf35 in os::abort(bool) () from /usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
#3  0x00007f8b31c0ff33 in VMError::report_and_die() () from /usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
#4  0x00007f8b31a7222f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
#5  0x00007f8b31a68253 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
#6  <signal handler called>
#7  0x00007f8b2f7d8c3a in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
#8  0x00000000011e9886 in kudu::rpc::WrapSaslCall(sasl_conn*, std::function<int ()> const&) ()
#9  0x00000000011fbde8 in kudu::rpc::ServerNegotiation::InitSaslServer() ()
#10 0x0000000001200ee0 in kudu::rpc::ServerNegotiation::AuthenticateBySasl(kudu::faststring*) ()
#11 0x0000000001202ea2 in kudu::rpc::ServerNegotiation::Negotiate() ()
#12 0x00000000011f5e9d in kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime) ()
#13 0x00000000011dab45 in kudu::internal::Invoker<4, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void (scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime), void (scoped_refptr<kudu::rpc::Connection>, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void (scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(kudu::internal::BindStateBase*) ()
#14 0x000000000127f3f8 in kudu::ThreadPool::DispatchThread(bool) ()
#15 0x000000000117fed5 in kudu::Thread::SuperviseThread(void*) ()
#16 0x00007f8b2f7d5806 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f8b2f53064d in clone () from /lib64/libc.so.6
#18 0x0000000000000000 in ?? ()

Apparently, it crashed at the following line here in WrapSaslCall(). has_kerberos_keytab is true because we passed a non-empty keytab to the messenger builder.

Status WrapSaslCall(sasl_conn_t* conn, const std::function<int()>& call) {
  // In many cases, the GSSAPI SASL plugin will generate a nice error
  // message as a message logged at SASL_LOG_FAIL logging level, but then
  // return a useless one in sasl_errstring(). So, we set a global thread-local
  // variable to capture any auth failure log message while we make the
  // call into the library.
  //
  // The thread-local thing is a bit of a hack, but the logging callback
  // is set globally rather than on a per-connection basis.
  string err;
  g_auth_failure_capture = &err;

  // Take the 'kerberos_reinit_lock' here to avoid a possible race with ticket renewal.
  if (has_kerberos_keytab) kudu::security::KerberosReinitLock()->ReadLock(); <<----
  int rc = call();
  if (has_kerberos_keytab) kudu::security::KerberosReinitLock()->ReadUnlock();
  g_auth_failure_capture = nullptr;

cc'ing sailesh