Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4497

Kudu client usage on secure clusters results in crash

    Details

      Description

      In the process of writing an integration test (systest), I discovered that the latest Impala build crashes on kerberized clusters when using the C++ Kudu client.

      This is likely an issue that will need to be addressed in the toolchain, but it's too soon to say.

      #0  0x00007efc558775e5 in raise () from /lib64/libc.so.6
      #1  0x00007efc55878dc5 in abort () from /lib64/libc.so.6
      #2  0x00007efc57a54a55 in os::abort(bool) () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
      #3  0x00007efc57bd4f87 in VMError::report_and_die() () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
      #4  0x00007efc57a5996f in JVM_handle_linux_signal () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
      #5  <signal handler called>
      #6  0x00007efc55be2635 in pthread_mutex_trylock () from /lib64/libpthread.so.0
      #7  0x00007efc56c1ad03 in kudu::Mutex::TryAcquire() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/mutex.cc:60
      #8  0x00007efc56c1af1f in kudu::Mutex::Acquire() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/mutex.cc:79
      #9  0x00007efc56add16e in kudu::Mutex::lock() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/mutex.h:46
      #10 0x00007efc56b28530 in kudu::rpc::SaslMutexLock(void*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_common.cc:151
      #11 0x00007efc587045e8 in sasl_dispose () from /usr/lib64/libsasl2.so.2
      #12 0x00007efc56b30028 in kudu::rpc::SaslDeleter::operator()(sasl_conn*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_common.h:87
      #13 0x00007efc56b30b3d in kudu::internal::gscoped_ptr_impl<sasl_conn, kudu::rpc::SaslDeleter>::reset(sasl_conn*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/gscoped_ptr.h:254
      #14 0x00007efc56b30393 in gscoped_ptr<sasl_conn, kudu::rpc::SaslDeleter>::reset(sasl_conn*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/gscoped_ptr.h:375
      #15 0x00007efc56b54c00 in kudu::rpc::SaslClient::Negotiate()::{lambda()#1}::operator()() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_client.cc:188
      #16 0x00007efc56b57c12 in kudu::ScopedCleanup<kudu::rpc::SaslClient::Negotiate()::{lambda()#1}>::~ScopedCleanup() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/scoped_cleanup.h:35
      #17 0x00007efc56b5540d in kudu::rpc::SaslClient::Negotiate() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_client.cc:246
      #18 0x00007efc56b525d6 in kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::MonoTime const&) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/negotiation.cc:189
      #19 0x00007efc56b52d6a in kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/negotiation.cc:222
      #20 0x00007efc56b1a327 in kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>::Run(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&) ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/bind_internal.h:228
      #21 0x00007efc56b193ec in kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>, void ()(kudu::rpc::Connection*, kudu::MonoTime const&)>::MakeItSo(kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>, kudu::rpc::Connection*, kudu::MonoTime const&) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/bind_internal.h:887
      #22 0x00007efc56b17c56 in kudu::internal::Invoker<2, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>, void ()(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&), void ()(scoped_refptr<kudu::rpc::Connection>, kudu::MonoTime)>, void ()(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>::Run(kudu::internal::BindStateBase*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/bind_internal.h:1150
      #23 0x00007efc56be09e2 in kudu::Callback<void ()()>::Run() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/callback.h:396
      #24 0x00007efc56c47e21 in boost::_mfi::cmf0<void, kudu::Callback<void ()()> >::operator()(kudu::Callback<void ()()> const&) const ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:120
      #25 0x00007efc56c478e8 in void boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()()> > >::operator()<boost::_mfi::cmf0<void, kudu::Callback<void ()()> >, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::cmf0<void, kudu::Callback<void ()()> >&, boost::_bi::list0&, int) ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:259
      #26 0x00007efc56c47071 in boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()()> > > >::operator()() ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:1222
      #27 0x00007efc56c468b4 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()()> > > >, void>::invoke(boost::detail::function::function_buffer&) ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:159
      #28 0x00007efc56b3dd3a in boost::function0<void>::operator()() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:771
      #29 0x00007efc56c44dac in kudu::FunctionRunnable::Run() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/threadpool.cc:48
      #30 0x00007efc56c4424c in kudu::ThreadPool::DispatchThread(bool) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/threadpool.cc:342
      #31 0x00007efc56c48452 in boost::_mfi::mf1<void, kudu::ThreadPool, bool>::operator()(kudu::ThreadPool*, bool) const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:165
      #32 0x00007efc56c4805b in void boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> >::operator()<boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, kudu::ThreadPool, bool>&, boost::_bi::list0&, int) ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:319
      #33 0x00007efc56c47aff in boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> > >::operator()() ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:1222
      #34 0x00007efc56c47432 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> > >, void>::invoke(boost::detail::function::function_buffer&) ()
          at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:159
      #35 0x00007efc56b3dd3a in boost::function0<void>::operator()() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:771
      #36 0x00007efc56c3b0a4 in kudu::Thread::SuperviseThread(void*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/thread.cc:588
      #37 0x00007efc55be0aa1 in start_thread () from /lib64/libpthread.so.0
      #38 0x00007efc5592daad in clone () from /lib64/libc.so.6
      

        Attachments

        1. hs_err.log
          101 kB
          Matthew Jacobs

          Issue Links

            Activity

              People

              • Assignee:
                mjacobs Matthew Jacobs
                Reporter:
                mjacobs Matthew Jacobs
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: