Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 1.3
-
None
-
None
Description
60 concurrent clients executing queries from the tpcds 1 Tb dataset. Admission controller was not on.
This is likely a regression, as I had the same cluster executing a larger dataset with 40 concurrent clients over the weekend, without crashing.
(gdb) bt
#0 0x0000003c66e328e5 in raise () from /lib64/libc.so.6
#1 0x0000003c66e3404d in abort () from /lib64/libc.so.6
#2 0x0000003c6aabea5d in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6
#3 0x0000003c6aabcbe6 in ?? () from /usr/lib64/libstdc++.so.6
#4 0x0000003c6aabcc13 in std::terminate() () from /usr/lib64/libstdc++.so.6
#5 0x0000003c6aabcd0e in __cxa_throw () from /usr/lib64/libstdc++.so.6
#6 0x000000000071b3a5 in boost::lock_guard<boost::mutex>::lock_guard(boost::mutex&) ()
#7 0x0000000000856e56 in impala::MemTracker::LogUsage (this=0x2682be840, prefix=<value optimized out>) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/runtime/mem-tracker.cc:215
#8 0x0000000000857093 in impala::MemTracker::LogUsage (prefix=" ", trackers=std::list = {...}) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/runtime/mem-tracker.cc:224
#9 0x0000000000856e79 in impala::MemTracker::LogUsage (this=<value optimized out>, prefix=<value optimized out>) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/runtime/mem-tracker.cc:216
#10 0x0000000000857093 in impala::MemTracker::LogUsage (prefix=" ", trackers=std::list = {...}) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/runtime/mem-tracker.cc:224
#11 0x0000000000856e79 in impala::MemTracker::LogUsage (this=<value optimized out>, prefix=<value optimized out>) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/runtime/mem-tracker.cc:216
#12 0x0000000000861831 in impala::RuntimeState::SetMemLimitExceeded (this=0x51cdde00, tracker=0xa899cccd, failed_allocation_size=<value optimized out>) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/runtime/runtime-state.cc:241
#13 0x0000000000861af4 in impala::RuntimeState::CheckQueryState (this=0x51cdde00) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/runtime/runtime-state.cc:258
#14 0x0000000000abcd66 in impala::HdfsScanner::CommitRows (this=0x195a5c80, num_rows=<value optimized out>) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/exec/hdfs-scanner.cc:149
#15 0x0000000000a85b1c in impala::HdfsParquetScanner::AssembleRows (this=0x195a5c80) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/exec/hdfs-parquet-scanner.cc:692
#16 0x0000000000a874a2 in impala::HdfsParquetScanner::ProcessSplit (this=0x195a5c80) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/exec/hdfs-parquet-scanner.cc:650
#17 0x0000000000a64dbf in impala::HdfsScanNode::ScannerThreadHelper() ()
#18 0x0000000000a5b9e0 in impala::HdfsScanNode::ScannerThread (this=<value optimized out>) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/exec/hdfs-scan-node.cc:766
#19 0x000000000096beda in operator() (name="scanner-thread(1)-15340", category=<value optimized out>, functor=..., thread_started=0x7fe451a30b50) at /usr/include/boost/function/function_template.hpp:1013
#20 impala::Thread::SuperviseThread (name="scanner-thread(1)-15340", category=<value optimized out>, functor=..., thread_started=0x7fe451a30b50) at /usr/src/debug/impala-1.3.0-cdh5.1.0-SNAPSHOT/be/src/util/thread.cc:263
#21 0x000000000096cae0 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
#22 0x0000000000b020f4 in thread_proxy ()
#23 0x0000003c67607851 in start_thread () from /lib64/libpthread.so.0
#24 0x0000003c66ee894d in clone () from /lib64/libc.so.6