[IMPALA-4893] Sequence scanner increments RuntimeProfile rows read counter per row wasting 30% CPU

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0
• Fix Version/s: Impala 2.9.0
• Component/s: Backend
• Labels: performance ramp-up
• Assignee: Anuj Phadke
• Reporter: Mostafa Mokhtar
• Created: 07/Feb/17 00:22
• Updated: 04/May/17 20:33
• Resolved: 04/May/17 20:33

Description

HdfsSequenceScanner::ProcessRange increments the number of scanned rows every row by calling COUNTER_ADD(scan_node_->rows_read_counter(), 1);.

Updating the number of scanned rows should be moved outside the while loop https://github.com/apache/incubator-impala/blob/b7eeb8bf85ef24b730c9ba2891f5a8075ba9605e/be/src/exec/hdfs-sequence-scanner.cc#L298

impalad ! Barrier_AtomicIncrement - atomicops-internals-x86.h
impalad ! impala::internal::AtomicInt<long>::Add - atomic.h:91
impalad ! impala::RuntimeProfile::Counter::Add - atomicops-internals-x86.h:276
impalad ! impala::HdfsSequenceScanner::ProcessRange + 0x265 - hdfs-sequence-scanner.cc:340
impalad ! impala::BaseSequenceScanner::ProcessSplit + 0x14c - base-sequence-scanner.cc:178
impalad ! impala::HdfsScanNode::ProcessSplit + 0x395 - hdfs-scan-node.cc:539
impalad ! impala::HdfsScanNode::ScannerThread + 0xbb8 - hdfs-scan-node.cc:430
impalad ! boost::function0<void>::operator() + 0x1a - function_template.hpp:767
impalad ! impala::Thread::SuperviseThread + 0x20e - thread.cc:317
impalad ! operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0> + 0x5a - bind.hpp:457
impalad ! boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>::operator() - bind_template.hpp:20
impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x19 - thread.hpp:116
impalad ! thread_proxy + 0xd9 - [unknown source file]
libpthread.so.0 ! start_thread + 0xd0 - [unknown source file]
libc.so.6 ! clone + 0x6c - [unknown source file]