[IMPALA-542] Impalad segfaults executing TPCDS-Q7 : in impala::LlvmCodeGen::OptimizeModule (this=0x2177a880) @ llvm-codegen.cc:532

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Works for Me
• Affects Version/s: Impala 1.1.1
• Fix Version/s: Impala 1.2
• Assignee: Nong Li
• Reporter: Lenni Kuff
• Created: 17/Aug/13 00:18
• Updated: 28/Aug/13 20:37
• Resolved: 28/Aug/13 20:37

Description

Impalad segfaults executing TPCDS-Q7. This doesn't repro on my machine after running the query multiple times.

(gdb) bt
#0  0x000000386a4328a5 in raise () from /lib64/libc.so.6
#1  0x000000386a434085 in abort () from /lib64/libc.so.6
#2  0x000000386e0bea5d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3  0x000000386e0bcbe6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x000000386e0bcc13 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x000000386e0bd53f in __cxa_pure_virtual () from /usr/lib64/libstdc++.so.6
#6  0x000000000160d5f0 in llvm::LibCallSimplifierImpl::optimizeCall(llvm::CallInst*) ()
#7  0x0000000001566ec1 in llvm::InstCombiner::tryOptimizeCall(llvm::CallInst*, llvm::DataLayout const*) ()
#8  0x0000000001568067 in llvm::InstCombiner::visitCallSite(llvm::CallSite) ()
#9  0x000000000156824c in llvm::InstCombiner::visitCallInst(llvm::CallInst&) ()
#10 0x0000000001548e9e in llvm::InstCombiner::DoOneIteration(llvm::Function&, unsigned int) ()
#11 0x0000000001549fbf in llvm::InstCombiner::runOnFunction(llvm::Function&) ()
#12 0x000000000180a420 in llvm::FPPassManager::runOnFunction(llvm::Function&) ()
#13 0x0000000001618e2a in (anonymous namespace)::CGPassManager::RunAllPassesOnSCC(llvm::CallGraphSCC&, llvm::CallGraph&, bool&) ()
#14 0x000000000161a869 in (anonymous namespace)::CGPassManager::runOnModule(llvm::Module&) ()
#15 0x000000000180a138 in llvm::MPPassManager::runOnModule(llvm::Module&) ()
#16 0x000000000180acce in llvm::PassManagerImpl::run(llvm::Module&) ()
#17 0x0000000000c1f637 in impala::LlvmCodeGen::OptimizeModule (this=0x2177a880)
    at /data/1/jenkins/workspace/impala-master-64bit/repos/Impala/be/src/codegen/llvm-codegen.cc:532
#18 0x0000000000d8a6ff in impala::PlanFragmentExecutor::OptimizeLlvmModule (this=0x21fff738)
    at /data/1/jenkins/workspace/impala-master-64bit/repos/Impala/be/src/runtime/plan-fragment-executor.cc:194
#19 0x0000000000b5fac9 in impala::ImpalaServer::FragmentExecState::Prepare (this=0x21fff700, exec_params=...)
    at /data/1/jenkins/workspace/impala-master-64bit/repos/Impala/be/src/service/impala-server.cc:226
#20 0x0000000000b6f1ec in impala::ImpalaServer::StartPlanFragmentExecution (this=0x4068380, exec_params=...)
    at /data/1/jenkins/workspace/impala-master-64bit/repos/Impala/be/src/service/impala-server.cc:1339
#21 0x0000000000b6e563 in impala::ImpalaServer::ExecPlanFragment (this=0x4068380, return_val=..., params=...)
    at /data/1/jenkins/workspace/impala-master-64bit/repos/Impala/be/src/service/impala-server.cc:1257
#22 0x0000000000dfd183 in impala::ImpalaInternalServiceProcessor::process_ExecPlanFragment (this=0x43c4d80, 
    seqid=0, iprot=0x4791f80, oprot=0x4791f40, callContext=0x25d1ef00)
    at /data/1/jenkins/workspace/impala-master-64bit/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:949
#23 0x0000000000dfcf28 in impala::ImpalaInternalServiceProcessor::dispatchCall (this=0x43c4d80, 
    iprot=0x4791f80, oprot=0x4791f40, fname="ExecPlanFragment", seqid=0, callContext=0x25d1ef00)
    at /data/1/jenkins/workspace/impala-master-64bit/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:922
#24 0x0000000000b753b0 in apache::thrift::TDispatchProcessor::process (this=0x43c4d80, in=..., out=..., 
    connectionContext=0x25d1ef00)
    at /data/1/jenkins/workspace/impala-master-64bit/repos/Impala/thirdparty/thrift-0.9.0/build/include/thrift/TDispatchProcessor.h:121
#25 0x0000000001891219 in apache::thrift::server::TThreadedServer::Task::run (this=0x5aeed20)
    at src/thrift/server/TThreadedServer.cpp:70
#26 0x0000000001893bc2 in apache::thrift::concurrency::PthreadThread::threadMain (arg=0x22ef8b60)
    at src/thrift/concurrency/PosixThreadFactory.cpp:208
#27 0x000000386ac07851 in start_thread () from /lib64/libpthread.so.0
#28 0x000000386a4e890d in clone () from /lib64/libc.so.6

This seems to be correlated with the "wrong results" that are seen during some TPCDS executions (we expect to get back 100 rows but instead get back a 1 empty row (see Jenkins failure for details: http://sandbox.jenkins.sf.cloudera.com/view/Impala/view/Build/job/impala-master-64bit/1313/consoleFull). I've also seen this with TPCDS-Q19, but it didn't result in a segfault