[IMPALA-879] impalad crashes while trying to serialize a runtime profile.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 1.3
• Fix Version/s: Impala 1.3
• Assignee: Srinath
• Reporter: Ishaan Joshi
• Created: 13/Mar/14 17:34
• Updated: 20/Dec/15 00:05
• Resolved: 14/Mar/14 23:54

Description

This is a stress environment (20 concurrent clients operating on 1Tb dataset running a range of TPCDS queries)

Stack:

#0  0x00000034c0e328e5 in raise () from /lib64/libc.so.6
#1  0x00000034c0e3404d in abort () from /lib64/libc.so.6
#2  0x00000034c46bea5d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3  0x00000034c46bcbe6 in ?? () from /usr/lib64/libstdc++.so.6
#4  0x00000034c46bcc13 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5  0x00000034c46bcd0e in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6  0x000000000147f89e in cpp_alloc (size=18446744071583318012) at src/tcmalloc.cc:1381
#7  tc_new (size=18446744071583318012) at src/tcmalloc.cc:1530
#8  0x000000000097217d in allocate (this=0x7fc963c88fe0, __position=Cannot access memory at address 0x0
) at /usr/include/c++/4.4.6/ext/new_allocator.h:89
#9  _M_allocate (this=0x7fc963c88fe0, __position=Cannot access memory at address 0x0
) at /usr/include/c++/4.4.6/bits/stl_vector.h:140
#10 std::vector<unsigned char, std::allocator<unsigned char> >::_M_fill_insert (this=0x7fc963c88fe0, __position=Cannot access memory at address 0x0
) at /usr/include/c++/4.4.6/bits/vector.tcc:414
#11 0x0000000000968cb0 in insert (this=<value optimized out>, __new_size=<value optimized out>, __x=0 '\000') at /usr/include/c++/4.4.6/bits/stl_vector.h:851
#12 std::vector<unsigned char, std::allocator<unsigned char> >::resize (this=<value optimized out>, __new_size=<value optimized out>, __x=0 '\000') at /usr/include/c++/4.4.6/bits/stl_vector.h:557
#13 0x00000000009702d8 in impala::RuntimeProfile::SerializeToArchiveString (this=<value optimized out>, out=0x7fc963c89340) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/util/runtime-profile.cc:606
#14 0x0000000000889859 in impala::ImpalaServer::GetRuntimeProfileStr (this=0x1bf7180, query_id=..., base64_encoded=true, output=0x7fc963c89340) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/service/impala-server.cc:421
#15 0x00000000008b84bb in impala::ImpalaServer::QueryProfileEncodedPathHandler (this=0x1bf7180, args=<value optimized out>, output=0x7fc963c89340) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/service/impala-server-callbacks.cc:146
#16 0x000000000098bd9d in operator() (this=0x5adf040, connection=0x77ee000, request_info=<value optimized out>) at /usr/include/boost/function/function_template.hpp:1013
#17 impala::Webserver::BeginRequestCallback (this=0x5adf040, connection=0x77ee000, request_info=<value optimized out>) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/util/webserver.cc:293
#18 0x000000000099a133 in handle_request ()
#19 0x000000000099ba30 in process_new_connection ()
#20 0x000000000099c098 in worker_thread ()
#21 0x00000034c1607851 in start_thread () from /lib64/libpthread.so.0
#22 0x00000034c0ee894d in clone () from /lib64/libc.so.6