[IMPALA-550] Cancelling INSERT query causes segfault in HdfsTableSink::FinalizePartitionFile() @ be/src/exec/hdfs-table-sink.cc:431 when session is closed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 1.1.1
• Fix Version/s: Impala 1.1.1
• Assignee: Unassigned
• Reporter: Lenni Kuff
• Created: 21/Aug/13 19:26
• Updated: 23/Aug/13 22:28
• Resolved: 23/Aug/13 22:28

Description

I think what might be happening here is that cancel is called one when the user initiated cancel is sent, afterwards the session is closed and Cancel() is called again because the query is still hanging around. This means we try to close the file twice and the second call does not make libhdfs very happy, causing this crash.

#3  0x00007f0e9dd799a2 in jniCheck::validate_handle(JavaThread*, _jobject*) () from /usr/lib/jvm/java-6-sun/jre/lib/amd64/server/libjvm.so
#4  0x00007f0e9dd79ad3 in jniCheck::validate_object(JavaThread*, _jobject*) () from /usr/lib/jvm/java-6-sun/jre/lib/amd64/server/libjvm.so
#5  0x00007f0e9dd848d2 in checked_jni_CallVoidMethodV () from /usr/lib/jvm/java-6-sun/jre/lib/amd64/server/libjvm.so
#6  0x00007f0ea0806329 in invokeMethod (env=0x36e49d0, retval=0x0, methType=<optimized out>, instObj=0x49e1e30, className=0x7f0ea08078f8 "org/apache/hadoop/fs/FSDataOutputStream", methName=0x7f0ea0806dcb "close", 
    methSignature=0x7f0ea0806d1f "()V") at /root/impala-hadoop-update/cdh/build/cdh4/hadoop/2.0.0-cdh4.3.0/source/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/jni_helper.c:242
#7  0x00007f0ea08021c6 in hdfsCloseFile (fs=<optimized out>, file=0x3c6e850) at /root/impala-hadoop-update/cdh/build/cdh4/hadoop/2.0.0-cdh4.3.0/source/hadoop-hdfs-project/hadoop-hdfs/src/main/native/libhdfs/hdfs.c:1027
#8  0x0000000000f34db2 in impala::HdfsTableSink::FinalizePartitionFile (this=0x59ea780, state=0x336f600, partition=0x4de5860) at /home/lskuff/dev/Impala/be/src/exec/hdfs-table-sink.cc:431
#9  0x0000000000f34f76 in impala::HdfsTableSink::Close (this=0x59ea780, state=0x336f600) at /home/lskuff/dev/Impala/be/src/exec/hdfs-table-sink.cc:452
#10 0x0000000000dcc75b in impala::PlanFragmentExecutor::Close (this=0x6a90700) at /home/lskuff/dev/Impala/be/src/runtime/plan-fragment-executor.cc:453
#11 0x0000000000dc8289 in impala::PlanFragmentExecutor::~PlanFragmentExecutor (this=0x6a90700, __in_chrg=<optimized out>) at /home/lskuff/dev/Impala/be/src/runtime/plan-fragment-executor.cc:67
#12 0x0000000000d7a331 in boost::checked_delete<impala::PlanFragmentExecutor> (x=0x6a90700) at /usr/include/boost/checked_delete.hpp:34
#13 0x0000000000d75de1 in boost::scoped_ptr<impala::PlanFragmentExecutor>::~scoped_ptr (this=0x411aa20, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/scoped_ptr.hpp:80
#14 0x0000000000d66ebd in impala::Coordinator::~Coordinator (this=0x411a800, __in_chrg=<optimized out>) at /home/lskuff/dev/Impala/be/src/runtime/coordinator.cc:230
#15 0x0000000000bee765 in boost::checked_delete<impala::Coordinator> (x=0x411a800) at /usr/include/boost/checked_delete.hpp:34
#16 0x0000000000be0e07 in boost::scoped_ptr<impala::Coordinator>::~scoped_ptr (this=0x47ac0d8, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/scoped_ptr.hpp:80
#17 0x0000000000bd9de1 in impala::ImpalaServer::QueryExecState::~QueryExecState (this=0x47ac000, __in_chrg=<optimized out>) at /home/lskuff/dev/Impala/be/src/service/query-exec-state.h:55
#18 0x0000000000c0bda6 in boost::checked_delete<impala::ImpalaServer::QueryExecState> (x=0x47ac000) at /usr/include/boost/checked_delete.hpp:34
#19 0x0000000000c246a8 in boost::detail::sp_counted_impl_p<impala::ImpalaServer::QueryExecState>::dispose (this=0x3b10ee0) at /usr/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
#20 0x0000000000bb194c in boost::detail::sp_counted_base::release (this=0x3b10ee0) at /usr/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:145
#21 0x0000000000bb19c5 in boost::detail::shared_count::~shared_count (this=0x7f0e82b08998, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/detail/shared_count.hpp:217
#22 0x0000000000bda870 in boost::shared_ptr<impala::ImpalaServer::QueryExecState>::~shared_ptr (this=0x7f0e82b08990, __in_chrg=<optimized out>) at /usr/include/boost/smart_ptr/shared_ptr.hpp:168
#23 0x0000000000bc46f7 in impala::ImpalaServer::UnregisterQuery (this=0x3a17500, query_id=...) at /home/lskuff/dev/Impala/be/src/service/impala-server.cc:1046
#24 0x0000000000bc55cf in impala::ImpalaServer::CloseSessionInternal (this=0x3a17500, session_id=...) at /home/lskuff/dev/Impala/be/src/service/impala-server.cc:1119
#25 0x0000000000c3fc47 in impala::ImpalaServer::SessionEnd (this=0x3a17500, session_context=...) at /home/lskuff/dev/Impala/be/src/service/impala-beeswax-server.cc:441
#26 0x0000000000d42793 in impala::ThriftServer::ThriftServerEventProcessor::deleteContext (this=0x29d9680, serverContext=0x34f96c0, input=..., output=...) at /home/lskuff/dev/Impala/be/src/util/thrift-server.cc:266
#27 0x00000000019fd722 in apache::thrift::server::TThreadPoolServer::Task::run (this=0x40a9200) at src/thrift/server/TThreadPoolServer.cpp:89
#28 0x00000000019ede1f in apache::thrift::concurrency::ThreadManager::Task::run (this=0x34f9c40) at src/thrift/concurrency/ThreadManager.cpp:187
#29 0x00000000019f0879 in apache::thrift::concurrency::ThreadManager::Worker::run (this=0x3bad770) at src/thrift/concurrency/ThreadManager.cpp:316
#30 0x0000000001a029b6 in apache::thrift::concurrency::PthreadThread::threadMain (arg=0x3b6ab10) at src/thrift/concurrency/PosixThreadFactory.cpp:208
#31 0x00007f0ea1339e9a in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#32 0x00007f0e9f068ccd in clone () from /lib/x86_64-linux-gnu/libc.so.6