Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5302

tcmalloc contention limits CPU utilization on machines with >40 logical processors

    Details

    • Epic Color:
      ghx-label-6

      Description

      On modern hardware with 80 logical processors it is very difficult to saturate the CPU on the server, even where there is high concurrency.

      Analysis showed that usage patterns of tcmalloc leads to SpinLock::SlowLock which results in the thread sleeping which results in under utilization of CPU.

      From perf with sampling on sched:sched_switch

      Samples: 331K of event 'sched:sched_switch', Event count (approx.): 331431
        Children      Self  Trace output                                                                                                          ◆
      -    0.76%     0.76%  impalad:47637 [120] S ==> swapper/22:0 [120]                                                                          ▒
           base::internal::SpinLockDelay                                                                                                          ▒
           system_call_fastpath                                                                                                                   ▒
           sys_futex                                                                                                                              ▒
           do_futex                                                                                                                               ▒
           futex_wait                                                                                                                             ▒
           futex_wait_queue_me                                                                                                                    ▒
           schedule                                                                                                                               ▒
           __schedule                                                                                                                             ▒
      -    0.76%     0.76%  swapper/22:0 [120] R ==> impalad:47637 [120]                                                                          ▒
           start_secondary                                                                                                                        ▒
           cpu_startup_entry                                                                                                                      ▒
           schedule_preempt_disabled                                                                                                              ▒
           __schedule                                                                                                                             ▒
      -    0.73%     0.73%  swapper/35:0 [120] R ==> impalad:47588 [120]                                                                          ▒
           start_secondary                                                                                                                        ▒
           cpu_startup_entry                                                                                                                      ▒
           schedule_preempt_disabled                                                                                                              ▒
           __schedule                                                                                                                             ▒
      -    0.73%     0.73%  impalad:47588 [120] S ==> swapper/35:0 [120]                                                                          ▒
           base::internal::SpinLockDelay                                                                                                          ▒
           system_call_fastpath                                                                                                                   ▒
           sys_futex                                                                                                                              ▒
           do_futex                                                                                                                               ▒
           futex_wait                                                                                                                             ▒
           futex_wait_queue_me                                                                                                                    ▒
           schedule                                                                                                                               ▒
           __schedule                                                                                                                             ▒
      Tip: Create an archive with symtabs to analyse on other machine: perf archive
      

      Call stack

      CPU Time
      1 of 101: 27.0% (1.141s of 4.224s)
      impalad ! SpinLock::SpinLoop - [unknown source file]
      impalad ! SpinLock::SlowLock + 0x70 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::Populate + 0x167 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::FetchFromOneSpansSafe + 0x37 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::RemoveRange + 0x63 - [unknown source file]
      impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown source file]
      impalad ! malloc + 0x27a - [unknown source file]
      impalad ! impala::MemPool::FindChunk + 0x207 - [unknown source file]
      impalad ! impala::RowBatch::ResizeAndAllocateTupleBuffer + 0xa1 - [unknown source file]
      impalad ! impala::HdfsParquetScanner::AssembleRows + 0x7f - [unknown source file]
      impalad ! impala::HdfsParquetScanner::GetNextInternal + 0x277 - [unknown source file]
      impalad ! impala::HdfsParquetScanner::ProcessSplit + 0x73 - [unknown source file]
      impalad ! impala::HdfsScanNode::ProcessSplit + 0x395 - [unknown source file]
      impalad ! impala::HdfsScanNode::ScannerThread + 0xbb8 - [unknown source file]
      impalad ! impala::Thread::SuperviseThread + 0x228 - [unknown source file]
      impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x73 - [unknown source file]
      impalad ! func@0xe4b330 + 0xd9 - [unknown source file]
      libpthread-2.17.so ! start_thread + 0xc4 - [unknown source file]
      libc-2.17.so ! __clone + 0x6c - [unknown source file]
      
      CPU Time
      2 of 101: 21.9% (0.923s of 4.224s)
      
      impalad ! SpinLock::SpinLoop - [unknown source file]
      impalad ! SpinLock::SlowLock + 0x1d - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::Populate + 0x167 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::FetchFromOneSpansSafe + 0x37 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::RemoveRange + 0x63 - [unknown source file]
      impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown source file]
      impalad ! malloc + 0x27a - [unknown source file]
      impalad ! impala::MemPool::FindChunk + 0x207 - [unknown source file]
      impalad ! impala::RowBatch::ResizeAndAllocateTupleBuffer + 0xa1 - [unknown source file]
      impalad ! impala::HdfsParquetScanner::AssembleRows + 0x7f - [unknown source file]
      impalad ! impala::HdfsParquetScanner::GetNextInternal + 0x277 - [unknown source file]
      impalad ! impala::HdfsParquetScanner::ProcessSplit + 0x73 - [unknown source file]
      impalad ! impala::HdfsScanNode::ProcessSplit + 0x395 - [unknown source file]
      impalad ! impala::HdfsScanNode::ScannerThread + 0xbb8 - [unknown source file]
      impalad ! impala::Thread::SuperviseThread + 0x228 - [unknown source file]
      impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x73 - [unknown source file]
      impalad ! func@0xe4b330 + 0xd9 - [unknown source file]
      libpthread-2.17.so ! start_thread + 0xc4 - [unknown source file]
      libc-2.17.so ! __clone + 0x6c - [unknown source file]
      
      CPU Time
      3 of 101: 7.1% (0.299s of 4.224s)
      
      impalad ! SpinLock::SpinLoop - [unknown source file]
      impalad ! SpinLock::SlowLock + 0x70 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::ReleaseToSpans + 0x157 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::ReleaseListToSpans + 0x1a - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::InsertRange + 0x3b - [unknown source file]
      impalad ! tcmalloc::ThreadCache::ReleaseToCentralCache + 0x103 - [unknown source file]
      impalad ! tcmalloc::ThreadCache::ListTooLong + 0x1b - [unknown source file]
      impalad ! operator delete + 0x3bf - [unknown source file]
      impalad ! impala::MemPool::FreeAll + 0x59 - [unknown source file]
      impalad ! impala::RowBatch::Reset + 0x30 - [unknown source file]
      impalad ! impala::PartitionedAggregationNode::GetRowsStreaming + 0x1f6 - [unknown source file]
      impalad ! impala::PartitionedAggregationNode::GetNextInternal + 0x230 - [unknown source file]
      impalad ! impala::PartitionedAggregationNode::GetNext + 0x21 - [unknown source file]
      impalad ! impala::FragmentInstanceState::ExecInternal + 0x187 - [unknown source file]
      impalad ! impala::FragmentInstanceState::Exec + 0x2b8 - [unknown source file]
      impalad ! impala::QueryState::ExecFInstance + 0x1f5 - [unknown source file]
      impalad ! impala::Thread::SuperviseThread + 0x228 - [unknown source file]
      impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x73 - [unknown source file]
      impalad ! func@0xe4b330 + 0xd9 - [unknown source file]
      libpthread-2.17.so ! start_thread + 0xc4 - [unknown source file]
      libc-2.17.so ! __clone + 0x6c - [unknown source file]
      
      CPU Time
      5 of 101: 3.8% (0.162s of 4.224s)
      
      impalad ! SpinLock::SpinLoop - [unknown source file]
      impalad ! SpinLock::SlowLock + 0x70 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::Populate + 0x167 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::FetchFromOneSpansSafe + 0x37 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::RemoveRange + 0x63 - [unknown source file]
      impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown source file]
      impalad ! malloc + 0x27a - [unknown source file]
      impalad ! impala::MemPool::FindChunk + 0x353 - [unknown source file]
      impalad ! impala::MemPool::Allocate<(bool)0> + 0x51 - [unknown source file]
      impalad ! impala::RowBatch::RowBatch + 0x183 - [unknown source file]
      impalad ! impala::DataStreamRecvr::SenderQueue::AddBatch + 0x445 - [unknown source file]
      impalad ! impala::DataStreamMgr::AddData + 0x115 - [unknown source file]
      impalad ! impala::ImpalaServer::TransmitData + 0x142 - [unknown source file]
      impalad ! impala::ImpalaInternalServiceProcessor::process_TransmitData + 0x2c0 - [unknown source file]
      impalad ! impala::ImpalaInternalServiceProcessor::dispatchCall + 0xe8 - [unknown source file]
      impalad ! apache::thrift::TDispatchProcessor::process + 0xab - [unknown source file]
      impalad ! apache::thrift::server::TAcceptQueueServer::Task::run + 0x15e - [unknown source file]
      impalad ! impala::ThriftThread::RunRunnable + 0x68 - [unknown source file]
      impalad ! boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable>>, boost::_bi::value<impala::Promise<unsigned long>*>>>, void>::invoke + 0x61 - [unknown source file]
      impalad ! impala::Thread::SuperviseThread + 0x228 - [unknown source file]
      impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x73 - [unknown source file]
      impalad ! func@0xe4b330 + 0xd9 - [unknown source file]
      libpthread-2.17.so ! start_thread + 0xc4 - [unknown source file]
      libc-2.17.so ! __clone + 0x6c - [unknown source file]
      
      CPU Time
      6 of 101: 3.5% (0.146s of 4.224s)
      
      impalad ! SpinLock::SpinLoop - [unknown source file]
      impalad ! SpinLock::SlowLock + 0x1d - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::Populate + 0x167 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::FetchFromOneSpansSafe + 0x37 - [unknown source file]
      impalad ! tcmalloc::CentralFreeList::RemoveRange + 0x63 - [unknown source file]
      impalad ! tcmalloc::ThreadCache::FetchFromCentralCache + 0x62 - [unknown source file]
      impalad ! malloc + 0x27a - [unknown source file]
      impalad ! impala::MemPool::FindChunk + 0x353 - [unknown source file]
      impalad ! impala::MemPool::Allocate<(bool)0> + 0x51 - [unknown source file]
      impalad ! impala::RowBatch::RowBatch + 0x183 - [unknown source file]
      impalad ! impala::DataStreamRecvr::SenderQueue::AddBatch + 0x445 - [unknown source file]
      impalad ! impala::DataStreamMgr::AddData + 0x115 - [unknown source file]
      impalad ! impala::ImpalaServer::TransmitData + 0x142 - [unknown source file]
      impalad ! impala::ImpalaInternalServiceProcessor::process_TransmitData + 0x2c0 - [unknown source file]
      impalad ! impala::ImpalaInternalServiceProcessor::dispatchCall + 0xe8 - [unknown source file]
      impalad ! apache::thrift::TDispatchProcessor::process + 0xab - [unknown source file]
      impalad ! apache::thrift::server::TAcceptQueueServer::Task::run + 0x15e - [unknown source file]
      impalad ! impala::ThriftThread::RunRunnable + 0x68 - [unknown source file]
      impalad ! boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable>>, boost::_bi::value<impala::Promise<unsigned long>*>>>, void>::invoke + 0x61 - [unknown source file]
      impalad ! impala::Thread::SuperviseThread + 0x228 - [unknown source file]
      impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x73 - [unknown source file]
      impalad ! func@0xe4b330 + 0xd9 - [unknown source file]
      libpthread-2.17.so ! start_thread + 0xc4 - [unknown source file]
      libc-2.17.so ! __clone + 0x6c - [unknown source file]
      
      #0  0x0000000001b75ba7 in base::internal::SpinLockDelay(int volatile*, int, int) ()
      #1  0x0000000001bad949 in SpinLock::SlowLock() ()
      #2  0x0000000001bf02c8 in tcmalloc::CentralFreeList::Populate() ()
      #3  0x0000000001bf03c8 in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) ()
      #4  0x0000000001bf0454 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) ()
      #5  0x0000000001bfdbb3 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) ()
      #6  0x0000000001c0aadb in tc_malloc ()
      #7  0x0000000000a26b88 in impala::MemPool::FindChunk(long, bool) ()
      #8  0x0000000000842c35 in unsigned char* impala::MemPool::Allocate<true>(long, int) ()
      #9  0x0000000000cbc313 in impala::BaseScalarColumnReader::ReadDataPage() ()
      #10 0x0000000000cbca3f in impala::BaseScalarColumnReader::NextPage() ()
      #11 0x0000000000ccc8a8 in impala::ScalarColumnReader<int, true>::ReadNonRepeatedValueBatch(impala::MemPool*, int, int, unsigned char*, int*) ()
      #12 0x0000000000c8d254 in impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*, std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*, bool*) ()
      #13 0x0000000000c91e48 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) ()
      #14 0x0000000000c8ca74 in impala::HdfsParquetScanner::ProcessSplit() ()
      #15 0x0000000000c5d946 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::DiskIoMgr::ScanRange*) ()
      #16 0x0000000000c5f5a9 in impala::HdfsScanNode::ScannerThread() ()
      #17 0x0000000000bea1b9 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
      #18 0x0000000000beab74 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
      #19 0x0000000000e4b40a in thread_proxy ()
      #20 0x00007fc0ed6c8dc5 in start_thread () from /lib64/libpthread.so.0
      #21 0x00007fc0ed3f773d in clone () from /lib64/libc.so.6
      
      Thread 1 (process 20304):
      #0  0x0000000001b75ba7 in base::internal::SpinLockDelay(int volatile*, int, int) ()
      #1  0x0000000001bad949 in SpinLock::SlowLock() ()
      #2  0x0000000001bf02c8 in tcmalloc::CentralFreeList::Populate() ()
      #3  0x0000000001bf03c8 in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) ()
      #4  0x0000000001bf0454 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) ()
      #5  0x0000000001bfdbb3 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) ()
      #6  0x0000000001c0aadb in tc_malloc ()
      #7  0x0000000000a26b88 in impala::MemPool::FindChunk(long, bool) ()
      #8  0x0000000000a39d92 in impala::RowBatch::ResizeAndAllocateTupleBuffer(impala::RuntimeState*, long*, unsigned char**) ()
      #9  0x0000000000c8d0d0 in impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*, std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*, bool*) ()
      #10 0x0000000000c91e48 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) ()
      #11 0x0000000000c8ca74 in impala::HdfsParquetScanner::ProcessSplit() ()
      #12 0x0000000000c5d946 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::DiskIoMgr::ScanRange*) ()
      #13 0x0000000000c5f5a9 in impala::HdfsScanNode::ScannerThread() ()
      #14 0x0000000000bea1b9 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
      #15 0x0000000000beab74 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
      #16 0x0000000000e4b40a in thread_proxy ()
      #17 0x00007f0c99962dc5 in start_thread () from /lib64/libpthread.so.0
      #18 0x00007f0c9969173d in clone () from /lib64/libc.so.6
      
      #0  0x0000000001b75ba7 in base::internal::SpinLockDelay(int volatile*, int, int) ()
      #1  0x0000000001bad949 in SpinLock::SlowLock() ()
      #2  0x0000000001befd38 in tcmalloc::CentralFreeList::ReleaseToSpans(void*) ()
      #3  0x0000000001befd7b in tcmalloc::CentralFreeList::ReleaseListToSpans(void*) ()
      #4  0x0000000001beffec in tcmalloc::CentralFreeList::InsertRange(void*, void*, int) ()
      #5  0x0000000001bfdd54 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int) ()
      #6  0x0000000001bfde1c in tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long) ()
      #7  0x0000000001c0aff0 in tc_free ()
      #8  0x0000000000a25cea in impala::MemPool::FreeAll() ()
      #9  0x0000000000a39821 in impala::RowBatch::Reset() ()
      #10 0x0000000000cdda97 in impala::PartitionedAggregationNode::GetRowsStreaming(impala::RuntimeState*, impala::RowBatch*) ()
      #11 0x0000000000ce34f1 in impala::PartitionedAggregationNode::GetNextInternal(impala::RuntimeState*, impala::RowBatch*, bool*) ()
      #12 0x0000000000ce35b2 in impala::PartitionedAggregationNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*) ()
      #13 0x0000000000a738d8 in impala::FragmentInstanceState::ExecInternal() ()
      #14 0x0000000000a76dd9 in impala::FragmentInstanceState::Exec() ()
      #15 0x0000000000a29516 in impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) ()
      #16 0x0000000000bea1b9 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
      #17 0x0000000000beab74 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
      #18 0x0000000000e4b40a in thread_proxy ()
      #19 0x00007f5f74b03dc5 in start_thread () from /lib64/libpthread.so.0
      #20 0x00007f5f7483273d in clone () from /lib64/libc.so.6
      
      #0  0x0000000001b75ba7 in base::internal::SpinLockDelay(int volatile*, int, int) ()
      #1  0x0000000001bad949 in SpinLock::SlowLock() ()
      #2  0x0000000001bf02c8 in tcmalloc::CentralFreeList::Populate() ()
      #3  0x0000000001bf03c8 in tcmalloc::CentralFreeList::FetchFromOneSpansSafe(int, void**, void**) ()
      #4  0x0000000001bf0454 in tcmalloc::CentralFreeList::RemoveRange(void**, void**, int) ()
      #5  0x0000000001bfdbb3 in tcmalloc::ThreadCache::FetchFromCentralCache(unsigned long, unsigned long) ()
      #6  0x0000000001c0aadb in tc_malloc ()
      #7  0x0000000000a26b88 in impala::MemPool::FindChunk(long, bool) ()
      #8  0x0000000000a39d92 in impala::RowBatch::ResizeAndAllocateTupleBuffer(impala::RuntimeState*, long*, unsigned char**) ()
      #9  0x0000000000c8d0d0 in impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*, std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*, bool*) ()
      #10 0x0000000000c91e48 in impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) ()
      #11 0x0000000000c8ca74 in impala::HdfsParquetScanner::ProcessSplit() ()
      #12 0x0000000000c5d946 in impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::DiskIoMgr::ScanRange*) ()
      #13 0x0000000000c5f5a9 in impala::HdfsScanNode::ScannerThread() ()
      #14 0x0000000000bea1b9 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) ()
      #15 0x0000000000beab74 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() ()
      #16 0x0000000000e4b40a in thread_proxy ()
      #17 0x00007f0c99962dc5 in start_thread () from /lib64/libpthread.so.0
      #18 0x00007f0c9969173d in clone () from /lib64/libc.so.6
      
      1. Screen Shot 2017-05-10 at 6.44.32 PM.png
        249 kB
        Mostafa Mokhtar
      2. thread_stack.txt
        105 kB
        Mostafa Mokhtar

        Issue Links

          Activity

          Hide
          mmokhtar Mostafa Mokhtar added a comment -

          Maybe we are seeing symptoms of this https://github.com/gperftools/gperftools/issues/494

          Show
          mmokhtar Mostafa Mokhtar added a comment - Maybe we are seeing symptoms of this https://github.com/gperftools/gperftools/issues/494
          Hide
          mmokhtar Mostafa Mokhtar added a comment -

          Already on gperftools-2.5 so unlikely.

          Show
          mmokhtar Mostafa Mokhtar added a comment - Already on gperftools-2.5 so unlikely.
          Hide
          marcelk Marcel Kornacker added a comment -

          Some observations:
          a) we're doing quite a bit of cross-thread allocations here (scanner threads, TransmitData handler threads)
          b) when a MemPool goes above a certain size, the underlying memory goes back to the central cache when the MemPool is destroyed; that's the case even for intra-thread allocations

          We should probably cache MemPools within a threads, instead of freeing them.

          Show
          marcelk Marcel Kornacker added a comment - Some observations: a) we're doing quite a bit of cross-thread allocations here (scanner threads, TransmitData handler threads) b) when a MemPool goes above a certain size, the underlying memory goes back to the central cache when the MemPool is destroyed; that's the case even for intra-thread allocations We should probably cache MemPools within a threads, instead of freeing them.
          Hide
          tarmstrong Tim Armstrong added a comment -

          Assigned it back to Mostafa to confirm that IMPALA-4923 solves it

          Show
          tarmstrong Tim Armstrong added a comment - Assigned it back to Mostafa to confirm that IMPALA-4923 solves it

            People

            • Assignee:
              mmokhtar Mostafa Mokhtar
              Reporter:
              mmokhtar Mostafa Mokhtar
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development