[IMPALA-5456] impala crashes in impala::Tuple::DeepCopyVarlenData trying to memcpy a null pointer

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: correctness crash
• Target Version: Impala 2.11.0
• Epic Color: ghx-label-6
• Assignee: Tim Armstrong
• Reporter: Anuj Phadke
• Created: 08/Jun/17 00:17
• Updated: 23/Aug/17 22:45
• Resolved: 23/Aug/17 22:45

Description

I can consistently recreate the crash with this query -

select c_custkey, c_mktsegment, o_orderkey, o_orderdate
from customer c,
  (select o1.o_orderkey, o2.o_orderdate
   from c.c_orders o1, c.c_orders o2
   where o1.o_orderkey = o2.o_orderkey limit 10) v limit 500;

Here is the stack trace -

(gdb) bt
#0  0x00007fdc22d49c37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fdc22d4d028 in __GI_abort () at abort.c:89
#2  0x00007fdc25143c55 in os::abort(bool) () from /usr/lib/jvm/java-7-oracle-amd64/jre/lib/amd64/server/libjvm.so
#3  0x00007fdc252c5cd7 in VMError::report_and_die() () from /usr/lib/jvm/java-7-oracle-amd64/jre/lib/amd64/server/libjvm.so
#4  0x00007fdc25148b6f in JVM_handle_linux_signal () from /usr/lib/jvm/java-7-oracle-amd64/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:157
#7  0x000000000146f9d3 in impala::Tuple::DeepCopyVarlenData (this=0x9961542, desc=..., data=0x7fdb8ed75fb0, offset=0x7fdb8ed75fbc, 
    convert_ptrs=true) at /home/anuj/Impala/be/src/runtime/tuple.cc:143
#8  0x000000000146f832 in impala::Tuple::DeepCopy (this=0x8bfa000, desc=..., data=0x7fdb8ed75fb0, offset=0x7fdb8ed75fbc, convert_ptrs=true)
    at /home/anuj/Impala/be/src/runtime/tuple.cc:132
#9  0x000000000145edaf in impala::RowBatch::SerializeInternal (this=0x6fcb680, size=1202, distinct_tuples=0x7fdb8ed762a0, 
    output_batch=0x8a82328) at /home/anuj/Impala/be/src/runtime/row-batch.cc:291
#10 0x000000000145dea1 in impala::RowBatch::Serialize (this=0x6fcb680, output_batch=0x8a82328, full_dedup=true)
    at /home/anuj/Impala/be/src/runtime/row-batch.cc:198
#11 0x000000000145dce7 in impala::RowBatch::Serialize (this=0x6fcb680, output_batch=0x8a82328)
    at /home/anuj/Impala/be/src/runtime/row-batch.cc:174
#12 0x0000000001aedd62 in impala::DataStreamSender::SerializeBatch (this=0x8a82280, src=0x6fcb680, dest=0x8a82328, num_receivers=1)
    at /home/anuj/Impala/be/src/runtime/data-stream-sender.cc:515
#13 0x0000000001aeca39 in impala::DataStreamSender::Send (this=0x8a82280, state=0x8122c00, batch=0x6fcb680)
    at /home/anuj/Impala/be/src/runtime/data-stream-sender.cc:429
#14 0x000000000148cda1 in impala::FragmentInstanceState::ExecInternal (this=0xa04f480)
    at /home/anuj/Impala/be/src/runtime/fragment-instance-state.cc:277
#15 0x000000000148a46d in impala::FragmentInstanceState::Exec (this=0xa04f480)
    at /home/anuj/Impala/be/src/runtime/fragment-instance-state.cc:89
#16 0x00000000014519ca in impala::QueryState::ExecFInstance (this=0x7cf0000, fis=0xa04f480)
    at /home/anuj/Impala/be/src/runtime/query-state.cc:330
#17 0x00000000014506cc in impala::QueryState::<lambda()>::operator()(void) const (__closure=0x7fdb8ed76d28)
    at /home/anuj/Impala/be/src/runtime/query-state.cc:304
#18 0x00000000014523a7 in boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...)
    at /home/anuj/Impala/toolchain/boost-1.57.0-p2/include/boost/function/function_template.hpp:153
#19 0x00000000013b267c in boost::function0<void>::operator() (this=0x7fdb8ed76d20)
    at /home/anuj/Impala/toolchain/boost-1.57.0-p2/include/boost/function/function_template.hpp:767
#20 0x000000000167ebf1 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name="exec-finstance (finst:e343f8b40399501d:5c4d95ff00000001)", category="fragment-execution", functor=..., 
    thread_started=0x7fdb8fd77d80) at /home/anuj/Impala/be/src/util/thread.cc:322
#21 0x0000000001687560 in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x896fbc0, 
    f=@0x896fbb8: 0x167e8d2 <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...) at /home/anuj/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/bind.hpp:457
#22 0x00000000016874a3 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x896fbb8)
    at /home/anuj/Impala/toolchain/boost-1.57.0-p2/include/boost/bind/bind_template.hpp:20
---Type <return> to continue, or q <return> to quit---q
Quit

StringVal has len=10 but is null pointer.

(gdb) frame 7
#7  0x000000000146f9d3 in impala::Tuple::DeepCopyVarlenData (this=0x9961542, desc=..., data=0x7fdb8ed75fb0, offset=0x7fdb8ed75fbc, 
    convert_ptrs=true) at /home/anuj/Impala/be/src/runtime/tuple.cc:143
143	    memcpy(*data, string_v->ptr, string_v->len);
(gdb) p string_v
$1 = (impala::StringValue *) 0x9961542
(gdb) p *string_v
$2 = {static MAX_LENGTH = 1073741824, ptr = 0x0, len = 10, static LLVM_CLASS_NAME = 0x2a22183 "struct.impala::StringValue"}
(gdb) 

Issue Links

• duplicates: IMPALA-3652 Fix resource transfer in subplans with limits
• is blocked by: IMPALA-3652 Fix resource transfer in subplans with limits
• relates to: IMPALA-3652 Fix resource transfer in subplans with limits