Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4497

Kudu client usage on secure clusters results in crash

    Details

      Description

      In the process of writing an integration test (systest), I discovered that the latest Impala build crashes on kerberized clusters when using the C++ Kudu client.

      This is likely an issue that will need to be addressed in the toolchain, but it's too soon to say.

      #0  0x00007efc558775e5 in raise () from /lib64/libc.so.6
      #1  0x00007efc55878dc5 in abort () from /lib64/libc.so.6
      #2  0x00007efc57a54a55 in os::abort(bool) () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
      #3  0x00007efc57bd4f87 in VMError::report_and_die() () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
      #4  0x00007efc57a5996f in JVM_handle_linux_signal () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
      #5  <signal handler called>
      #6  0x00007efc55be2635 in pthread_mutex_trylock () from /lib64/libpthread.so.0
      #7  0x00007efc56c1ad03 in kudu::Mutex::TryAcquire() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/mutex.cc:60
      #8  0x00007efc56c1af1f in kudu::Mutex::Acquire() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/mutex.cc:79
      #9  0x00007efc56add16e in kudu::Mutex::lock() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/mutex.h:46
      #10 0x00007efc56b28530 in kudu::rpc::SaslMutexLock(void*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_common.cc:151
      #11 0x00007efc587045e8 in sasl_dispose () from /usr/lib64/libsasl2.so.2
      #12 0x00007efc56b30028 in kudu::rpc::SaslDeleter::operator()(sasl_conn*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_common.h:87
      #13 0x00007efc56b30b3d in kudu::internal::gscoped_ptr_impl<sasl_conn, kudu::rpc::SaslDeleter>::reset(sasl_conn*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/gscoped_ptr.h:254
      #14 0x00007efc56b30393 in gscoped_ptr<sasl_conn, kudu::rpc::SaslDeleter>::reset(sasl_conn*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/gscoped_ptr.h:375
      #15 0x00007efc56b54c00 in kudu::rpc::SaslClient::Negotiate()::{lambda()#1}::operator()() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_client.cc:188
      #16 0x00007efc56b57c12 in kudu::ScopedCleanup<kudu::rpc::SaslClient::Negotiate()::{lambda()#1}>::~ScopedCleanup() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/scoped_cleanup.h:35
      #17 0x00007efc56b5540d in kudu::rpc::SaslClient::Negotiate() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_client.cc:246
      #18 0x00007efc56b525d6 in kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::MonoTime const&) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/negotiation.cc:189
      #19 0x00007efc56b52d6a in kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/negotiation.cc:222
      #20 0x00007efc56b1a327 in kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>::Run(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&) ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/bind_internal.h:228
      #21 0x00007efc56b193ec in kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>, void ()(kudu::rpc::Connection*, kudu::MonoTime const&)>::MakeItSo(kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>, kudu::rpc::Connection*, kudu::MonoTime const&) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/bind_internal.h:887
      #22 0x00007efc56b17c56 in kudu::internal::Invoker<2, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>, void ()(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&), void ()(scoped_refptr<kudu::rpc::Connection>, kudu::MonoTime)>, void ()(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>::Run(kudu::internal::BindStateBase*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/bind_internal.h:1150
      #23 0x00007efc56be09e2 in kudu::Callback<void ()()>::Run() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/callback.h:396
      #24 0x00007efc56c47e21 in boost::_mfi::cmf0<void, kudu::Callback<void ()()> >::operator()(kudu::Callback<void ()()> const&) const ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:120
      #25 0x00007efc56c478e8 in void boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()()> > >::operator()<boost::_mfi::cmf0<void, kudu::Callback<void ()()> >, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::cmf0<void, kudu::Callback<void ()()> >&, boost::_bi::list0&, int) ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:259
      #26 0x00007efc56c47071 in boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()()> > > >::operator()() ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:1222
      #27 0x00007efc56c468b4 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()()> > > >, void>::invoke(boost::detail::function::function_buffer&) ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:159
      #28 0x00007efc56b3dd3a in boost::function0<void>::operator()() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:771
      #29 0x00007efc56c44dac in kudu::FunctionRunnable::Run() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/threadpool.cc:48
      #30 0x00007efc56c4424c in kudu::ThreadPool::DispatchThread(bool) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/threadpool.cc:342
      #31 0x00007efc56c48452 in boost::_mfi::mf1<void, kudu::ThreadPool, bool>::operator()(kudu::ThreadPool*, bool) const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:165
      #32 0x00007efc56c4805b in void boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> >::operator()<boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, kudu::ThreadPool, bool>&, boost::_bi::list0&, int) ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:319
      #33 0x00007efc56c47aff in boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> > >::operator()() ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:1222
      #34 0x00007efc56c47432 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> > >, void>::invoke(boost::detail::function::function_buffer&) ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:159
      #35 0x00007efc56b3dd3a in boost::function0<void>::operator()() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:771
      #36 0x00007efc56c3b0a4 in kudu::Thread::SuperviseThread(void*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/thread.cc:588
      #37 0x00007efc55be0aa1 in start_thread () from /lib64/libpthread.so.0
      #38 0x00007efc5592daad in clone () from /lib64/libc.so.6
      
      1. hs_err.log
        101 kB
        Matthew Jacobs

        Issue Links

          Activity

          Hide
          tlipcon Todd Lipcon added a comment -

          Filed https://issues.apache.org/jira/browse/KUDU-1749. The fix should be pretty trivial. MJ's going to try commenting out the line of code that's causing this.

          Show
          tlipcon Todd Lipcon added a comment - Filed https://issues.apache.org/jira/browse/KUDU-1749 . The fix should be pretty trivial. MJ's going to try commenting out the line of code that's causing this.
          Hide
          mjacobs Matthew Jacobs added a comment -

          When the Kudu fix is pushed, we will build a new toolchain and bump the Impala bits to fix this.

          Show
          mjacobs Matthew Jacobs added a comment - When the Kudu fix is pushed, we will build a new toolchain and bump the Impala bits to fix this.
          Hide
          mjacobs Matthew Jacobs added a comment -

          commit de88f0c4af3a07ae6bd6b8c94edcb8748468f522
          Author: Matthew Jacobs <mj@cloudera.com>
          Date: Fri Nov 18 07:15:58 2016 -0800

          IMPALA-4497: Fix Kudu client crash w/ SASL initialization

          Kudu commit cd40d1c5126d0fd541d95370cdb2998b60f2f82c added a
          call to sasl_set_mutex, which ends up in the client process
          (Impala). That conflicts with Impala's SASL initialization code
          resulting in a crash when the Kudu client is used to
          open a connection to Kudu.

          Kudu commit f2aeba6c059ea61f9cf8984d3e84d6c27b64d463
          provided a method to disable the Kudu client's SASL
          initialization to avoid this. This commit calls this method
          on process startup.

          Testing on kerberized cluster which was previously crashing.

          Change-Id: Ic465c31406a6df2753f6cc7481b6b391ed5e671b
          Reviewed-on: http://gerrit.cloudera.org:8080/5139
          Reviewed-by: Matthew Jacobs <mj@cloudera.com>
          Tested-by: Internal Jenkins

          Show
          mjacobs Matthew Jacobs added a comment - commit de88f0c4af3a07ae6bd6b8c94edcb8748468f522 Author: Matthew Jacobs <mj@cloudera.com> Date: Fri Nov 18 07:15:58 2016 -0800 IMPALA-4497 : Fix Kudu client crash w/ SASL initialization Kudu commit cd40d1c5126d0fd541d95370cdb2998b60f2f82c added a call to sasl_set_mutex, which ends up in the client process (Impala). That conflicts with Impala's SASL initialization code resulting in a crash when the Kudu client is used to open a connection to Kudu. Kudu commit f2aeba6c059ea61f9cf8984d3e84d6c27b64d463 provided a method to disable the Kudu client's SASL initialization to avoid this. This commit calls this method on process startup. Testing on kerberized cluster which was previously crashing. Change-Id: Ic465c31406a6df2753f6cc7481b6b391ed5e671b Reviewed-on: http://gerrit.cloudera.org:8080/5139 Reviewed-by: Matthew Jacobs <mj@cloudera.com> Tested-by: Internal Jenkins

            People

            • Assignee:
              mjacobs Matthew Jacobs
              Reporter:
              mjacobs Matthew Jacobs
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development