Details
-
Sub-task
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 2.3.0
-
None
Description
By default Thrift transport has 512 bytes of buffer space which results in massive amount of small memory copies as Thrift goes through writeSlow and readSlow which ends up breaking the payload into N chunks of 512 bytes.
The fix is to increase the default buffer size, or use the constructor which passes in a non-default R/W buffer size.
Increasing the buffer size to 512KB removed Thrift from the hot code path.
TBufferTransports.h
class TBufferedTransport : public TVirtualTransport<TBufferedTransport, TBufferBase> { public: static const int DEFAULT_BUFFER_SIZE = 512;
The call stack below takes about 30% of the CPU for a broadcast join.
apache::thrift::transport::TSocket::read apache::thrift::transport::TTransport::read apache::thrift::transport::TBufferedTransport::readSlow apache::thrift::transport::TBufferBase::read apache::thrift::transport::readAll<apache::thrift::transport::TBufferBase> apache::thrift::transport::TTransport::readAll apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport>::readStringBody<std::string> readString<std::basic_string<char> > apache::thrift::protocol::TVirtualProtocol<apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport>, apache::thrift::protocol::TProtocolDefaults>::readString_virt apache::thrift::protocol::TProtocol::readString impala::TRowBatch::read impala::TTransmitDataParams::read impala::ImpalaInternalService_TransmitData_args::read impala::ImpalaInternalServiceProcessor::process_TransmitData impala::ImpalaInternalServiceProcessor::dispatchCall apache::thrift::TDispatchProcessor::process apache::thrift::server::TThreadedServer::Task::run impala::ThriftThread::RunRunnable boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>::operator() operator()<boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<long unsigned int>*>, boost::_bi::list0> boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala: boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned boost::function0<void>::operator() impala::Thread::SuperviseThread operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0> boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::str boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, func@0xba00 start_thread __clone
Attachments
Issue Links
- is related to
-
IMPALA-3260 Investigate EXCHANGE / DataStreamSender performance
-
- Closed
-