[IMPALA-6165] Crash in kudu::rpc::InboundCall::SerializeResponseBuffer inbound_call.cc:149

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: Not Applicable
• Component/s: Distributed Exec
• Epic Color: ghx-label-3
• Assignee: Michael Ho
• Reporter: Mostafa Mokhtar
• Created: 06/Nov/17 21:33
• Updated: 07/Nov/17 23:26
• Resolved: 07/Nov/17 23:26

Description

While running concurrent TPCDS queries hit the crash below

Stack

#0  0x0000003a7c232625 in raise () from /lib64/libc.so.6
#1  0x0000003a7c233e05 in abort () from /lib64/libc.so.6
#2  0x00007f5dc0c7ea55 in os::abort(bool) ()   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#3  0x00007f5dc0dfef87 in VMError::report_and_die() ()   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#4  0x00007f5dc0c8396f in JVM_handle_linux_signal ()   from /usr/java/jdk1.7.0_67-cloudera/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x0000000000f7986c in kudu::rpc::InboundCall::SerializeResponseBuffer (this=0x7f58a65c6220,    response=..., is_success=true)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/kudu/rpc/inbound_call.cc:149
#7  0x0000000000f79b9f in kudu::rpc::InboundCall::Respond (this=0x7f58a65c6220, response=...,    is_success=Unhandled dwarf expression opcode 0xf3)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/kudu/rpc/inbound_call.cc:137
#8  0x0000000000f79ee4 in kudu::rpc::InboundCall::RespondSuccess (this=0x7f58a65c6220, response=...)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/kudu/rpc/inbound_call.cc:88
#9  0x0000000000f73ae8 in kudu::rpc::RpcContext::RespondSuccess (this=0x7f58707703a0)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/kudu/rpc/rpc_context.cc:68
#10 0x0000000000ae7bcc in impala::KrpcDataStreamMgr::AddData (this=Unhandled dwarf expression opcode 0xf3)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/runtime/krpc-data-stream-mgr.cc:214
#11 0x0000000000b8e0eb in impala::DataStreamService::TransmitData (this=Unhandled dwarf expression opcode 0xf3)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/service/data-stream-service.cc:53
#12 0x0000000000f77685 in operator() (this=Unhandled dwarf expression opcode 0xf3)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/gcc-4.9.2/include/c++/4.9.2/functional:2439
#13 kudu::rpc::GeneratedServiceIf::Handle (this=Unhandled dwarf expression opcode 0xf3)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/kudu/rpc/service_if.cc:134
#14 0x000000000103408d in kudu::rpc::ServicePool::RunThread (this=Unhandled dwarf expression opcode 0xf3)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/kudu/rpc/service_pool.cc:210
#15 0x0000000000f20cd9 in operator() (arg=Unhandled dwarf expression opcode 0xf3)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#16 kudu::Thread::SuperviseThread (arg=Unhandled dwarf expression opcode 0xf3)    at /data/jenkins/workspace/impala-private-build-binaries/repos/Impala/be/src/kudu/util/thread.cc:591
#17 0x0000003a7c607aa1 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a7c2e893d in clone () from /lib64/libc.so.6