Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4893

Sequence scanner increments RuntimeProfile rows read counter per row wasting 30% CPU

    Details

      Description

      HdfsSequenceScanner::ProcessRange increments the number of scanned rows every row by calling COUNTER_ADD(scan_node_->rows_read_counter(), 1);.

      Updating the number of scanned rows should be moved outside the while loop https://github.com/apache/incubator-impala/blob/b7eeb8bf85ef24b730c9ba2891f5a8075ba9605e/be/src/exec/hdfs-sequence-scanner.cc#L298

      impalad ! Barrier_AtomicIncrement - atomicops-internals-x86.h
      impalad ! impala::internal::AtomicInt<long>::Add - atomic.h:91
      impalad ! impala::RuntimeProfile::Counter::Add - atomicops-internals-x86.h:276
      impalad ! impala::HdfsSequenceScanner::ProcessRange + 0x265 - hdfs-sequence-scanner.cc:340
      impalad ! impala::BaseSequenceScanner::ProcessSplit + 0x14c - base-sequence-scanner.cc:178
      impalad ! impala::HdfsScanNode::ProcessSplit + 0x395 - hdfs-scan-node.cc:539
      impalad ! impala::HdfsScanNode::ScannerThread + 0xbb8 - hdfs-scan-node.cc:430
      impalad ! boost::function0<void>::operator() + 0x1a - function_template.hpp:767
      impalad ! impala::Thread::SuperviseThread + 0x20e - thread.cc:317
      impalad ! operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0> + 0x5a - bind.hpp:457
      impalad ! boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>::operator() - bind_template.hpp:20
      impalad ! boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void (void)>>, boost::_bi::value<impala::Promise<long>*>>>>::run + 0x19 - thread.hpp:116
      impalad ! thread_proxy + 0xd9 - [unknown source file]
      libpthread.so.0 ! start_thread + 0xd0 - [unknown source file]
      libc.so.6 ! clone + 0x6c - [unknown source file]
      

        Activity

        Show
        anujphadke Anuj Phadke added a comment - https://github.com/apache/incubator-impala/commit/5809317c9a202159ac28e0565d04151eae843d09

          People

          • Assignee:
            anujphadke Anuj Phadke
            Reporter:
            mmokhtar Mostafa Mokhtar
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development