[IMPALA-5302] tcmalloc contention limits CPU utilization on machines with >40 logical processors

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: memory performance scalability
• Epic Color: ghx-label-6
• Assignee: Mostafa Mokhtar
• Reporter: Mostafa Mokhtar
• Created: 11/May/17 01:50
• Updated: 07/Feb/23 11:13
• Resolved: 25/May/17 14:55

Description

On modern hardware with 80 logical processors it is very difficult to saturate the CPU on the server, even where there is high concurrency.

Analysis showed that usage patterns of tcmalloc leads to SpinLock::SlowLock which results in the thread sleeping which results in under utilization of CPU.

From perf with sampling on sched:sched_switch

Samples: 331K of event 'sched:sched_switch', Event count (approx.): 331431
  Children      Self  Trace output                                                                                                          ◆
-    0.76%     0.76%  impalad:47637 [120] S ==> swapper/22:0 [120]                                                                          ▒
     base::internal::SpinLockDelay                                                                                                          ▒
     system_call_fastpath                                                                                                                   ▒
     sys_futex                                                                                                                              ▒
     do_futex                                                                                                                               ▒
     futex_wait                                                                                                                             ▒
     futex_wait_queue_me                                                                                                                    ▒
     schedule                                                                                                                               ▒
     __schedule                                                                                                                             ▒
-    0.76%     0.76%  swapper/22:0 [120] R ==> impalad:47637 [120]                                                                          ▒
     start_secondary                                                                                                                        ▒
     cpu_startup_entry                                                                                                                      ▒
     schedule_preempt_disabled                                                                                                              ▒
     __schedule                                                                                                                             ▒
-    0.73%     0.73%  swapper/35:0 [120] R ==> impalad:47588 [120]                                                                          ▒
     start_secondary                                                                                                                        ▒
     cpu_startup_entry                                                                                                                      ▒
     schedule_preempt_disabled                                                                                                              ▒
     __schedule                                                                                                                             ▒
-    0.73%     0.73%  impalad:47588 [120] S ==> swapper/35:0 [120]                                                                          ▒
     base::internal::SpinLockDelay                                                                                                          ▒
     system_call_fastpath                                                                                                                   ▒
     sys_futex                                                                                                                              ▒
     do_futex                                                                                                                               ▒
     futex_wait                                                                                                                             ▒
     futex_wait_queue_me                                                                                                                    ▒
     schedule                                                                                                                               ▒
     __schedule                                                                                                                             ▒
Tip: Create an archive with symtabs to analyse on other machine: perf archive

Call stack

CPU Time
1 of 101: 27.0% (1.141s of 4.224s)
impalad ! SpinLock::SpinLoop - [unknown source file]
impalad ! SpinLock::SlowLock + 0x70 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::Populate + 0x167 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::FetchFromOneSpansSafe + 0x37 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::RemoveRange + 0x63 - [unknown source file]
impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown source file]
impalad ! malloc + 0x27a - [unknown source file]
impalad ! impala::MemPool::FindChunk + 0x207 - [unknown source file]
impalad ! impala::RowBatch::ResizeAndAllocateTupleBuffer + 0xa1 - [unknown source file]
impalad ! impala::HdfsParquetScanner::AssembleRows + 0x7f - [unknown source file]
impalad ! impala::HdfsParquetScanner::GetNextInternal + 0x277 - [unknown source file]
impalad ! impala::HdfsParquetScanner::ProcessSplit + 0x73 - [unknown source file]
impalad ! impala::HdfsScanNode::ProcessSplit + 0x395 - [unknown source file]
impalad ! impala::HdfsScanNode::ScannerThread + 0xbb8 - [unknown source file]
impalad ! impala::Thread::SuperviseThread + 0x228 - [unknown source file]
impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x73 - [unknown source file]
impalad ! func@0xe4b330 + 0xd9 - [unknown source file]
libpthread-2.17.so ! start_thread + 0xc4 - [unknown source file]
libc-2.17.so ! __clone + 0x6c - [unknown source file]

CPU Time
2 of 101: 21.9% (0.923s of 4.224s)

impalad ! SpinLock::SpinLoop - [unknown source file]
impalad ! SpinLock::SlowLock + 0x1d - [unknown source file]
impalad ! tcmalloc::CentralFreeList::Populate + 0x167 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::FetchFromOneSpansSafe + 0x37 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::RemoveRange + 0x63 - [unknown source file]
impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown source file]
impalad ! malloc + 0x27a - [unknown source file]
impalad ! impala::MemPool::FindChunk + 0x207 - [unknown source file]
impalad ! impala::RowBatch::ResizeAndAllocateTupleBuffer + 0xa1 - [unknown source file]
impalad ! impala::HdfsParquetScanner::AssembleRows + 0x7f - [unknown source file]
impalad ! impala::HdfsParquetScanner::GetNextInternal + 0x277 - [unknown source file]
impalad ! impala::HdfsParquetScanner::ProcessSplit + 0x73 - [unknown source file]
impalad ! impala::HdfsScanNode::ProcessSplit + 0x395 - [unknown source file]
impalad ! impala::HdfsScanNode::ScannerThread + 0xbb8 - [unknown source file]
impalad ! impala::Thread::SuperviseThread + 0x228 - [unknown source file]
impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x73 - [unknown source file]
impalad ! func@0xe4b330 + 0xd9 - [unknown source file]
libpthread-2.17.so ! start_thread + 0xc4 - [unknown source file]
libc-2.17.so ! __clone + 0x6c - [unknown source file]

CPU Time
3 of 101: 7.1% (0.299s of 4.224s)

impalad ! SpinLock::SpinLoop - [unknown source file]
impalad ! SpinLock::SlowLock + 0x70 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::ReleaseToSpans + 0x157 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown source file]
impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source file]
impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown source file]
impalad ! tcmalloc::ThreadCache::ListTooLong + 0x1b - [unknown source file]
impalad ! operator delete + 0x3bf - [unknown source file]
impalad ! impala::MemPool::FreeAll + 0x59 - [unknown source file]
impalad ! impala::RowBatch::Reset + 0x30 - [unknown source file]
impalad ! impala::PartitionedAggregationNode::GetRowsStreaming + 0x1f6 - [unknown source file]
impalad ! impala::PartitionedAggregationNode::GetNextInternal + 0x230 - [unknown source file]
impalad ! impala::PartitionedAggregationNode::GetNext + 0x21 - [unknown source file]
impalad ! impala::FragmentInstanceState::ExecInternal + 0x187 - [unknown source file]
impalad ! impala::FragmentInstanceState::Exec + 0x2b8 - [unknown source file]
impalad ! impala::QueryState::ExecFInstance + 0x1f5 - [unknown source file]
impalad ! impala::Thread::SuperviseThread + 0x228 - [unknown source file]
impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x73 - [unknown source file]
impalad ! func@0xe4b330 + 0xd9 - [unknown source file]
libpthread-2.17.so ! start_thread + 0xc4 - [unknown source file]
libc-2.17.so ! __clone + 0x6c - [unknown source file]

CPU Time
5 of 101: 3.8% (0.162s of 4.224s)

impalad ! SpinLock::SpinLoop - [unknown source file]
impalad ! SpinLock::SlowLock + 0x70 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::Populate + 0x167 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::FetchFromOneSpansSafe + 0x37 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::RemoveRange + 0x63 - [unknown source file]
impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown source file]
impalad ! malloc + 0x27a - [unknown source file]
impalad ! impala::MemPool::FindChunk + 0x353 - [unknown source file]
impalad ! impala::MemPool::Allocate<(bool)0> + 0x51 - [unknown source file]
impalad ! impala::RowBatch::RowBatch + 0x183 - [unknown source file]
impalad ! impala::DataStreamRecvr::SenderQueue::AddBatch + 0x445 - [unknown source file]
impalad ! impala::DataStreamMgr::AddData + 0x115 - [unknown source file]
impalad ! impala::ImpalaServer::TransmitData + 0x142 - [unknown source file]
impalad ! impala::ImpalaInternalServiceProcessor::process_TransmitData + 0x2c0 - [unknown source file]
impalad ! impala::ImpalaInternalServiceProcessor::dispatchCall + 0xe8 - [unknown source file]
impalad ! apache::thrift::TDispatchProcessor::process + 0xab - [unknown source file]
impalad ! apache::thrift::server::TAcceptQueueServer::Task::run + 0x15e - [unknown source file]
impalad ! impala::ThriftThread::RunRunnable + 0x68 - [unknown source file]
impalad ! boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable>>, boost::_bi::value<impala::Promise<unsigned long>*>>>, void>::invoke + 0x61 - [unknown source file]
impalad ! impala::Thread::SuperviseThread + 0x228 - [unknown source file]
impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x73 - [unknown source file]
impalad ! func@0xe4b330 + 0xd9 - [unknown source file]
libpthread-2.17.so ! start_thread + 0xc4 - [unknown source file]
libc-2.17.so ! __clone + 0x6c - [unknown source file]

CPU Time
6 of 101: 3.5% (0.146s of 4.224s)

impalad ! SpinLock::SpinLoop - [unknown source file]
impalad ! SpinLock::SlowLock + 0x1d - [unknown source file]
impalad ! tcmalloc::CentralFreeList::Populate + 0x167 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::FetchFromOneSpansSafe + 0x37 - [unknown source file]
impalad ! tcmalloc::CentralFreeList::RemoveRange + 0x63 - [unknown source file]
impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown source file]
impalad ! malloc + 0x27a - [unknown source file]
impalad ! impala::MemPool::FindChunk + 0x353 - [unknown source file]
impalad ! impala::MemPool::Allocate<(bool)0> + 0x51 - [unknown source file]
impalad ! impala::RowBatch::RowBatch + 0x183 - [unknown source file]
impalad ! impala::DataStreamRecvr::SenderQueue::AddBatch + 0x445 - [unknown source file]
impalad ! impala::DataStreamMgr::AddData + 0x115 - [unknown source file]
impalad ! impala::ImpalaServer::TransmitData + 0x142 - [unknown source file]
impalad ! impala::ImpalaInternalServiceProcessor::process_TransmitData + 0x2c0 - [unknown source file]
impalad ! impala::ImpalaInternalServiceProcessor::dispatchCall + 0xe8 - [unknown source file]
impalad ! apache::thrift::TDispatchProcessor::process + 0xab - [unknown source file]
impalad ! apache::thrift::server::TAcceptQueueServer::Task::run + 0x15e - [unknown source file]
impalad ! impala::ThriftThread::RunRunnable + 0x68 - [unknown source file]
impalad ! boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable>>, boost::_bi::value<impala::Promise<unsigned long>*>>>, void>::invoke + 0x61 - [unknown source file]
impalad ! impala::Thread::SuperviseThread + 0x228 - [unknown source file]
impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x73 - [unknown source file]
impalad ! func@0xe4b330 + 0xd9 - [unknown source file]
libpthread-2.17.so ! start_thread + 0xc4 - [unknown source file]
libc-2.17.so ! __clone + 0x6c - [unknown source file]

#0  0x0000000001b75ba7 in base::internal::SpinLockDelay(int volatile*, int, int) ()
#1  0x0000000001bad949 in SpinLock::SlowLock() ()
#2  0x0000000001bf02c8 in tcmalloc::CentralFreeList::Populate() ()
#3  0x0000000001bf03c8 in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) ()
#4  0x0000000001bf0454 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) ()
#5  0x0000000001bfdbb3 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) ()
#6  0x0000000001c0aadb in tc_malloc ()
#7  0x0000000000a26b88 in impala::MemPool::FindChunk(long, bool) ()
#8  0x0000000000842c35 in unsigned char* impala::MemPool::Allocate<true>(long, int) ()
#9  0x0000000000cbc313 in impala::BaseScalarColumnReader::ReadDataPage() ()
#10 0x0000000000cbca3f in impala::BaseScalarColumnReader::NextPage() ()
#11 0x0000000000ccc8a8 in impala::ScalarColumnReader<int, true>::ReadNonRepeatedValueBatch(impala::MemPool*, int, int, unsigned char*, int*) ()
#12 0x0000000000c8d254 in impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*, std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*, bool*) ()
#13 0x0000000000c91e48 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) ()
#14 0x0000000000c8ca74 in impala::HdfsParquetScanner::ProcessSplit() ()
#15 0x0000000000c5d946 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::DiskIoMgr::ScanRange*) ()
#16 0x0000000000c5f5a9 in impala::HdfsScanNode::ScannerThread() ()
#17 0x0000000000bea1b9 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
#18 0x0000000000beab74 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
#19 0x0000000000e4b40a in thread_proxy ()
#20 0x00007fc0ed6c8dc5 in start_thread () from /lib64/libpthread.so.0
#21 0x00007fc0ed3f773d in clone () from /lib64/libc.so.6

Thread 1 (process 20304):
#0  0x0000000001b75ba7 in base::internal::SpinLockDelay(int volatile*, int, int) ()
#1  0x0000000001bad949 in SpinLock::SlowLock() ()
#2  0x0000000001bf02c8 in tcmalloc::CentralFreeList::Populate() ()
#3  0x0000000001bf03c8 in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) ()
#4  0x0000000001bf0454 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) ()
#5  0x0000000001bfdbb3 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) ()
#6  0x0000000001c0aadb in tc_malloc ()
#7  0x0000000000a26b88 in impala::MemPool::FindChunk(long, bool) ()
#8  0x0000000000a39d92 in impala::RowBatch::ResizeAndAllocateTupleBuffer(impala::RuntimeState*, long*, unsigned char**) ()
#9  0x0000000000c8d0d0 in impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*, std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*, bool*) ()
#10 0x0000000000c91e48 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) ()
#11 0x0000000000c8ca74 in impala::HdfsParquetScanner::ProcessSplit() ()
#12 0x0000000000c5d946 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::DiskIoMgr::ScanRange*) ()
#13 0x0000000000c5f5a9 in impala::HdfsScanNode::ScannerThread() ()
#14 0x0000000000bea1b9 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
#15 0x0000000000beab74 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
#16 0x0000000000e4b40a in thread_proxy ()
#17 0x00007f0c99962dc5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f0c9969173d in clone () from /lib64/libc.so.6

#0  0x0000000001b75ba7 in base::internal::SpinLockDelay(int volatile*, int, int) ()
#1  0x0000000001bad949 in SpinLock::SlowLock() ()
#2  0x0000000001befd38 in tcmalloc::CentralFreeList::ReleaseToSpans(void*) ()
#3  0x0000000001befd7b in tcmalloc::CentralFreeList::ReleaseListToSpans(void*) ()
#4  0x0000000001beffec in tcmalloc::CentralFreeList::InsertRange(void*, void*, int) ()
#5  0x0000000001bfdd54 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) ()
#6  0x0000000001bfde1c in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) ()
#7  0x0000000001c0aff0 in tc_free ()
#8  0x0000000000a25cea in impala::MemPool::FreeAll() ()
#9  0x0000000000a39821 in impala::RowBatch::Reset() ()
#10 0x0000000000cdda97 in impala::PartitionedAggregationNode::GetRowsStreaming(impala::RuntimeState*, impala::RowBatch*) ()
#11 0x0000000000ce34f1 in impala::PartitionedAggregationNode::GetNextInternal(impala::RuntimeState*, impala::RowBatch*, bool*) ()
#12 0x0000000000ce35b2 in impala::PartitionedAggregationNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) ()
#13 0x0000000000a738d8 in impala::FragmentInstanceState::ExecInternal() ()
#14 0x0000000000a76dd9 in impala::FragmentInstanceState::Exec() ()
#15 0x0000000000a29516 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) ()
#16 0x0000000000bea1b9 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
#17 0x0000000000beab74 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
#18 0x0000000000e4b40a in thread_proxy ()
#19 0x00007f5f74b03dc5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f5f7483273d in clone () from /lib64/libc.so.6

#0  0x0000000001b75ba7 in base::internal::SpinLockDelay(int volatile*, int, int) ()
#1  0x0000000001bad949 in SpinLock::SlowLock() ()
#2  0x0000000001bf02c8 in tcmalloc::CentralFreeList::Populate() ()
#3  0x0000000001bf03c8 in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) ()
#4  0x0000000001bf0454 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) ()
#5  0x0000000001bfdbb3 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) ()
#6  0x0000000001c0aadb in tc_malloc ()
#7  0x0000000000a26b88 in impala::MemPool::FindChunk(long, bool) ()
#8  0x0000000000a39d92 in impala::RowBatch::ResizeAndAllocateTupleBuffer(impala::RuntimeState*, long*, unsigned char**) ()
#9  0x0000000000c8d0d0 in impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*, std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*, bool*) ()
#10 0x0000000000c91e48 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) ()
#11 0x0000000000c8ca74 in impala::HdfsParquetScanner::ProcessSplit() ()
#12 0x0000000000c5d946 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::DiskIoMgr::ScanRange*) ()
#13 0x0000000000c5f5a9 in impala::HdfsScanNode::ScannerThread() ()
#14 0x0000000000bea1b9 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
#15 0x0000000000beab74 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
#16 0x0000000000e4b40a in thread_proxy ()
#17 0x00007f0c99962dc5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f0c9969173d in clone () from /lib64/libc.so.6

Issue Links

• is related to: IMPALA-4923 Operators running on top of selective Parquet scans spend a lot of time calling impala::MemPool::FreeAll on empty batches
• is related to: IMPALA-5150 Uneven load distribution of work across NUMA nodes
• relates to: IMPALA-4923 Operators running on top of selective Parquet scans spend a lot of time calling impala::MemPool::FreeAll on empty batches