XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.3.0
    • Fix Version/s: Impala 2.5.0, Impala 2.3.4
    • Component/s: None
    • Labels:

      Description

      By default Thrift transport has 512 bytes of buffer space which results in massive amount of small memory copies as Thrift goes through writeSlow and readSlow which ends up breaking the payload into N chunks of 512 bytes.

      The fix is to increase the default buffer size, or use the constructor which passes in a non-default R/W buffer size.

      Increasing the buffer size to 512KB removed Thrift from the hot code path.

      TBufferTransports.h

      class TBufferedTransport
        : public TVirtualTransport<TBufferedTransport, TBufferBase> {
       public:
      
        static const int DEFAULT_BUFFER_SIZE = 512;
      

      The call stack below takes about 30% of the CPU for a broadcast join.

      apache::thrift::transport::TSocket::read
      apache::thrift::transport::TTransport::read
      apache::thrift::transport::TBufferedTransport::readSlow
      apache::thrift::transport::TBufferBase::read
      apache::thrift::transport::readAll<apache::thrift::transport::TBufferBase>
      apache::thrift::transport::TTransport::readAll
      apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport>::readStringBody<std::string>
      readString<std::basic_string<char> >
      apache::thrift::protocol::TVirtualProtocol<apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport>, apache::thrift::protocol::TProtocolDefaults>::readString_virt
      apache::thrift::protocol::TProtocol::readString
      impala::TRowBatch::read
      impala::TTransmitDataParams::read
      impala::ImpalaInternalService_TransmitData_args::read
      impala::ImpalaInternalServiceProcessor::process_TransmitData
      impala::ImpalaInternalServiceProcessor::dispatchCall
      apache::thrift::TDispatchProcessor::process
      apache::thrift::server::TThreadedServer::Task::run
      impala::ThriftThread::RunRunnable
      boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>::operator()
      operator()<boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<long unsigned int>*>, boost::_bi::list0>
      boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala:
      boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned
      boost::function0<void>::operator()
      impala::Thread::SuperviseThread
      operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0>
      boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::str
      boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>,
      func@0xba00
      start_thread
      __clone
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                henryr Henry Robinson
                Reporter:
                mmokhtar Mostafa Mokhtar
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: