Uploaded image for project: 'IMPALA'
  2. IMPALA-2567 KRPC milestone 1
  3. IMPALA-6728

The combination use_kudu_kinit=false and use_krpc=true crashes Impalad



    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 3.0, Impala 2.12.0
    • Impala 3.0, Impala 2.12.0
    • Security


      Although this is a weird combination, it's possible to start Impala with the configuration combination use_kudu_kinit=false and use_krpc=true. In which case, Impala will crash with the following backtrace:


      (gdb) bt
      #0  0x00007f8b2f484885 in raise () from /lib64/libc.so.6
      #1  0x00007f8b2f485e61 in abort () from /lib64/libc.so.6
      #2  0x00007f8b31a6bf35 in os::abort(bool) () from /usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
      #3  0x00007f8b31c0ff33 in VMError::report_and_die() () from /usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
      #4  0x00007f8b31a7222f in JVM_handle_linux_signal () from /usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
      #5  0x00007f8b31a68253 in signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_162/jre/lib/amd64/server/libjvm.so
      #6  <signal handler called>
      #7  0x00007f8b2f7d8c3a in pthread_rwlock_rdlock () from /lib64/libpthread.so.0
      #8  0x00000000011e9886 in kudu::rpc::WrapSaslCall(sasl_conn*, std::function<int ()> const&) ()
      #9  0x00000000011fbde8 in kudu::rpc::ServerNegotiation::InitSaslServer() ()
      #10 0x0000000001200ee0 in kudu::rpc::ServerNegotiation::AuthenticateBySasl(kudu::faststring*) ()
      #11 0x0000000001202ea2 in kudu::rpc::ServerNegotiation::Negotiate() ()
      #12 0x00000000011f5e9d in kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime) ()
      #13 0x00000000011dab45 in kudu::internal::Invoker<4, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void (scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime), void (scoped_refptr<kudu::rpc::Connection>, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>, void (scoped_refptr<kudu::rpc::Connection> const&, kudu::TriStateFlag, kudu::TriStateFlag, kudu::MonoTime)>::Run(kudu::internal::BindStateBase*) ()
      #14 0x000000000127f3f8 in kudu::ThreadPool::DispatchThread(bool) ()
      #15 0x000000000117fed5 in kudu::Thread::SuperviseThread(void*) ()
      #16 0x00007f8b2f7d5806 in start_thread () from /lib64/libpthread.so.0
      #17 0x00007f8b2f53064d in clone () from /lib64/libc.so.6
      #18 0x0000000000000000 in ?? ()

      Apparently, it crashed at the following line here in WrapSaslCall(). has_kerberos_keytab is true because we passed a non-empty keytab to the messenger builder.

      Status WrapSaslCall(sasl_conn_t* conn, const std::function<int()>& call) {
        // In many cases, the GSSAPI SASL plugin will generate a nice error
        // message as a message logged at SASL_LOG_FAIL logging level, but then
        // return a useless one in sasl_errstring(). So, we set a global thread-local
        // variable to capture any auth failure log message while we make the
        // call into the library.
        // The thread-local thing is a bit of a hack, but the logging callback
        // is set globally rather than on a per-connection basis.
        string err;
        g_auth_failure_capture = &err;
        // Take the 'kerberos_reinit_lock' here to avoid a possible race with ticket renewal.
        if (has_kerberos_keytab) kudu::security::KerberosReinitLock()->ReadLock(); <<----
        int rc = call();
        if (has_kerberos_keytab) kudu::security::KerberosReinitLock()->ReadUnlock();
        g_auth_failure_capture = nullptr;

      cc'ing sailesh




            kwho Michael Ho
            kwho Michael Ho
            0 Vote for this issue
            5 Start watching this issue