Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Invalid
-
Impala 2.9.0
-
None
-
ghx-label-8
Description
While running the TPC-DS workload with 20 concurrent threads on a 140 node cluster I encountered two crashes, both on the same host, that look like some thread limit was reached. In one crash gdb found 32296 threads and in the other it found 31781 threads. I see the following message in the error log:
terminate called after throwing an instance of 'boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::thread_resource_error> >'
what(): boost::thread_resource_error: Resource temporarily unavailable
Backtrace:
Program terminated with signal 6, Aborted. #0 0x0000003c2dc32625 in raise () from /lib64/libc.so.6 Missing separate debuginfos, use: debuginfo-install cyrus-sasl-gssapi-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-lib-2.1.23-15.el6_6.2.x86_64 cyrus-sasl-plain-2.1.23-15.el6_6.2.x86_64 db4-4.7.25-20.el6_7.x86_64 glibc-2.12-1.166.el6_7.7.x86_64 keyutils-libs-1.4-5.el6.x86_64 krb5-libs-1.10.3-42z1.el6_7.x86_64 libcom_err-1.41.12-22.el6.x86_64 libselinux-2.0.94-5.8.el6.x86_64 nss-softokn-freebl-3.14.3-23.el6_7.x86_64 openssl-1.0.1e-42.el6_7.4.x86_64 oracle-j2sdk1.7-1.7.0+update67-1.x86_64 zlib-1.2.3-29.el6.x86_64 (gdb) bt #0 0x0000003c2dc32625 in raise () from /lib64/libc.so.6 #1 0x0000003c2dc33e05 in abort () from /lib64/libc.so.6 #2 0x00007f323cb7707d in __gnu_cxx::__verbose_terminate_handler() () from /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.3/lib/impala/lib/libstdc++.so.6 #3 0x00007f323cb750e6 in ?? () from /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.3/lib/impala/lib/libstdc++.so.6 #4 0x00007f323cb75131 in std::terminate() () from /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.3/lib/impala/lib/libstdc++.so.6 #5 0x00007f323cb75348 in __cxa_throw () from /opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.3/lib/impala/lib/libstdc++.so.6 #6 0x000000000081dd4d in void boost::throw_exception<boost::thread_resource_error>(boost::thread_resource_error const&) () #7 0x0000000000bedda0 in boost::thread::thread<void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()()>, impala::Promise<long>*), std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::function<void ()()>, impala::Promise<long>*>(void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()()>, impala::Promise<long>*), std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::function<void ()()>, impala::Promise<long>*) () #8 0x0000000000bea1b5 in impala::Thread::StartThread(boost::function<void ()()> const&) () #9 0x0000000000df2a63 in impala::DataStreamSender::DataStreamSender(impala::ObjectPool*, int, impala::RowDescriptor const&, impala::TDataStreamSink const&, std::vector<impala::TPlanFragmentDestination, std::allocator<impala::TPlanFragmentDestination> > const&, int) () #10 0x0000000000c461f7 in impala::DataSink::Create(impala::ObjectPool*, impala::TPlanFragmentCtx const&, impala::TPlanFragmentInstanceCtx const&, impala::RowDescriptor const&, impala::DataSink**) () #11 0x0000000000a7cfaf in impala::FragmentInstanceState::Prepare() () #12 0x0000000000a7dae1 in impala::FragmentInstanceState::Exec() () #13 0x0000000000a69bb6 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) () #14 0x0000000000bec7d9 in impala::Thread::SuperviseThread(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()()>, impala::Promise<long>*) () #15 0x0000000000bed194 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::function<void ()()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() () #16 0x0000000000e540ea in ?? () #17 0x0000003c2e007aa1 in start_thread () from /lib64/libpthread.so.0 #18 0x0000003c2dce893d in clone () from /lib64/libc.so.6
Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size unlimited unlimited bytes Max resident set unlimited unlimited bytes Max processes 65536 65536 processes Max open files 262144 262144 files Max locked memory unlimited unlimited bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 1032751 1032751 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Max realtime timeout unlimited unlimited us
/proc/sys/kernel/threads-max
2065503
/proc/sys/vm/max_map_count
65530
/proc/sys/kernel/pid_max
196608