[IMPALA-4497] Kudu client usage on secure clusters results in crash

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.8.0
• Component/s: Backend
• Labels: crash kerberos kudu security
• Target Version: Impala 2.8.0
• Assignee: Matthew Jacobs
• Reporter: Matthew Jacobs
• Created: 16/Nov/16 21:00
• Updated: 19/Nov/16 18:19
• Resolved: 19/Nov/16 18:19

Description

In the process of writing an integration test (systest), I discovered that the latest Impala build crashes on kerberized clusters when using the C++ Kudu client.

This is likely an issue that will need to be addressed in the toolchain, but it's too soon to say.

#0  0x00007efc558775e5 in raise () from /lib64/libc.so.6
#1  0x00007efc55878dc5 in abort () from /lib64/libc.so.6
#2  0x00007efc57a54a55 in os::abort(bool) () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#3  0x00007efc57bd4f87 in VMError::report_and_die() () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#4  0x00007efc57a5996f in JVM_handle_linux_signal () from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x00007efc55be2635 in pthread_mutex_trylock () from /lib64/libpthread.so.0
#7  0x00007efc56c1ad03 in kudu::Mutex::TryAcquire() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/mutex.cc:60
#8  0x00007efc56c1af1f in kudu::Mutex::Acquire() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/mutex.cc:79
#9  0x00007efc56add16e in kudu::Mutex::lock() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/mutex.h:46
#10 0x00007efc56b28530 in kudu::rpc::SaslMutexLock(void*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_common.cc:151
#11 0x00007efc587045e8 in sasl_dispose () from /usr/lib64/libsasl2.so.2
#12 0x00007efc56b30028 in kudu::rpc::SaslDeleter::operator()(sasl_conn*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_common.h:87
#13 0x00007efc56b30b3d in kudu::internal::gscoped_ptr_impl<sasl_conn, kudu::rpc::SaslDeleter>::reset(sasl_conn*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/gscoped_ptr.h:254
#14 0x00007efc56b30393 in gscoped_ptr<sasl_conn, kudu::rpc::SaslDeleter>::reset(sasl_conn*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/gscoped_ptr.h:375
#15 0x00007efc56b54c00 in kudu::rpc::SaslClient::Negotiate()::{lambda()#1}::operator()() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_client.cc:188
#16 0x00007efc56b57c12 in kudu::ScopedCleanup<kudu::rpc::SaslClient::Negotiate()::{lambda()#1}>::~ScopedCleanup() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/scoped_cleanup.h:35
#17 0x00007efc56b5540d in kudu::rpc::SaslClient::Negotiate() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/sasl_client.cc:246
#18 0x00007efc56b525d6 in kudu::rpc::DoClientNegotiation(kudu::rpc::Connection*, kudu::MonoTime const&) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/negotiation.cc:189
#19 0x00007efc56b52d6a in kudu::rpc::Negotiation::RunNegotiation(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/rpc/negotiation.cc:222
#20 0x00007efc56b1a327 in kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>::Run(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&) ()
    at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/bind_internal.h:228
#21 0x00007efc56b193ec in kudu::internal::InvokeHelper<false, void, kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>, void ()(kudu::rpc::Connection*, kudu::MonoTime const&)>::MakeItSo(kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>, kudu::rpc::Connection*, kudu::MonoTime const&) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/bind_internal.h:887
#22 0x00007efc56b17c56 in kudu::internal::Invoker<2, kudu::internal::BindState<kudu::internal::RunnableAdapter<void (*)(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>, void ()(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&), void ()(scoped_refptr<kudu::rpc::Connection>, kudu::MonoTime)>, void ()(scoped_refptr<kudu::rpc::Connection> const&, kudu::MonoTime const&)>::Run(kudu::internal::BindStateBase*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/bind_internal.h:1150
#23 0x00007efc56be09e2 in kudu::Callback<void ()()>::Run() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/gutil/callback.h:396
#24 0x00007efc56c47e21 in boost::_mfi::cmf0<void, kudu::Callback<void ()()> >::operator()(kudu::Callback<void ()()> const&) const ()
    at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:120
#25 0x00007efc56c478e8 in void boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()()> > >::operator()<boost::_mfi::cmf0<void, kudu::Callback<void ()()> >, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::cmf0<void, kudu::Callback<void ()()> >&, boost::_bi::list0&, int) ()
    at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:259
#26 0x00007efc56c47071 in boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()()> > > >::operator()() ()
    at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:1222
#27 0x00007efc56c468b4 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::cmf0<void, kudu::Callback<void ()()> >, boost::_bi::list1<boost::_bi::value<kudu::Callback<void ()()> > > >, void>::invoke(boost::detail::function::function_buffer&) ()
    at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:159
#28 0x00007efc56b3dd3a in boost::function0<void>::operator()() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:771
#29 0x00007efc56c44dac in kudu::FunctionRunnable::Run() () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/threadpool.cc:48
#30 0x00007efc56c4424c in kudu::ThreadPool::DispatchThread(bool) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/threadpool.cc:342
#31 0x00007efc56c48452 in boost::_mfi::mf1<void, kudu::ThreadPool, bool>::operator()(kudu::ThreadPool*, bool) const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/mem_fn_template.hpp:165
#32 0x00007efc56c4805b in void boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> >::operator()<boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, kudu::ThreadPool, bool>&, boost::_bi::list0&, int) ()
    at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:319
#33 0x00007efc56c47aff in boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> > >::operator()() ()
    at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/bind/bind.hpp:1222
#34 0x00007efc56c47432 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, kudu::ThreadPool, bool>, boost::_bi::list2<boost::_bi::value<kudu::ThreadPool*>, boost::_bi::value<bool> > >, void>::invoke(boost::detail::function::function_buffer&) ()
    at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:159
#35 0x00007efc56b3dd3a in boost::function0<void>::operator()() const () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/thirdparty/installed/common/include/boost/function/function_template.hpp:771
#36 0x00007efc56c3b0a4 in kudu::Thread::SuperviseThread(void*) () at /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-88b02349d88d335caac18bf8b930eac6d327ed40/src/kudu/util/thread.cc:588
#37 0x00007efc55be0aa1 in start_thread () from /lib64/libpthread.so.0
#38 0x00007efc5592daad in clone () from /lib64/libc.so.6

Issue Links

• relates to: IMPALA-4509 Impala should provide SASL a mutex