[IMPALA-2554] Increase Thrift default buffers size from 512 Bytes

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.5.0, Impala 2.3.4
• Labels: performance
• Target Version: Impala 2.5.0, Impala 2.3.4
• Assignee: Henry Robinson
• Reporter: Mostafa Mokhtar
• Created: 15/Oct/15 02:06
• Updated: 21/May/16 16:01
• Resolved: 30/Nov/15 18:19

Description

By default Thrift transport has 512 bytes of buffer space which results in massive amount of small memory copies as Thrift goes through writeSlow and readSlow which ends up breaking the payload into N chunks of 512 bytes.

The fix is to increase the default buffer size, or use the constructor which passes in a non-default R/W buffer size.

Increasing the buffer size to 512KB removed Thrift from the hot code path.

TBufferTransports.h

class TBufferedTransport
  : public TVirtualTransport<TBufferedTransport, TBufferBase> {
 public:

  static const int DEFAULT_BUFFER_SIZE = 512;

The call stack below takes about 30% of the CPU for a broadcast join.

apache::thrift::transport::TSocket::read
apache::thrift::transport::TTransport::read
apache::thrift::transport::TBufferedTransport::readSlow
apache::thrift::transport::TBufferBase::read
apache::thrift::transport::readAll<apache::thrift::transport::TBufferBase>
apache::thrift::transport::TTransport::readAll
apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport>::readStringBody<std::string>
readString<std::basic_string<char> >
apache::thrift::protocol::TVirtualProtocol<apache::thrift::protocol::TBinaryProtocolT<apache::thrift::transport::TTransport>, apache::thrift::protocol::TProtocolDefaults>::readString_virt
apache::thrift::protocol::TProtocol::readString
impala::TRowBatch::read
impala::TTransmitDataParams::read
impala::ImpalaInternalService_TransmitData_args::read
impala::ImpalaInternalServiceProcessor::process_TransmitData
impala::ImpalaInternalServiceProcessor::dispatchCall
apache::thrift::TDispatchProcessor::process
apache::thrift::server::TThreadedServer::Task::run
impala::ThriftThread::RunRunnable
boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>::operator()
operator()<boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<long unsigned int>*>, boost::_bi::list0>
boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala:
boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned
boost::function0<void>::operator()
impala::Thread::SuperviseThread
operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0>
boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::str
boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void (void)>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>,
func@0xba00
start_thread
__clone

Issue Links

• is related to: IMPALA-3260 Investigate EXCHANGE / DataStreamSender performance