Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3494

Thrift buffer overflows when serialize more than 3355443200 bytes in impala

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 2.5.0
    • Impala 2.6.0
    • Backend

    Description

      Back trace:

      (gdb) bt
#0  0x0000003bf5832625 in raise () from s/lib64/libc.so.6
#1  0x0000003bf5833e05 in abort () from s/lib64/libc.so.6
#2  0x00007fa6cc9517b5 in os::abort(bool) () from s/usr/java/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so
#3  0x00007fa6ccaeeea3 in VMError::report_and_die() () from s/usr/java/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so
#4  0x00007fa6cc956cdf in JVM_handle_linux_signal () from s/usr/java/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so
#5  0x00007fa6cc94d593 in signalHandler(int, siginfo*, void*) () from s/usr/java/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so
#6  <signal handler called>
#7  0x0000000001f7cc55 in tcmalloc::PageHeap::MergeIntoFreeList (this=0x333c000, span=0x7fa24d38c340) at src/page_heap.cc:227
#8  0x0000000001f7cf2d in tcmalloc::PageHeap::Delete (this=0x333c000, span=<optimized out>) at src/page_heap.cc:197
#9  0x0000000001f927b5 in (anonymous namespace)::do_free_with_callback (ptr=<optimized out>, invalid_free_fn=<optimized out>) at src/tcmalloc.cc:1147
#10 (anonymous namespace)::do_free (ptr=<optimized out>) at src/tcmalloc.cc:1153
#11 tc_free (ptr=<optimized out>) at src/tcmalloc.cc:1499
#12 0x000000000111b006 in apache::thrift::transport::TMemoryBuffer::~TMemoryBuffer (this=0x7fa44f4b03c0, __in_chrg=<optimized out>) at /usr/src/debug/impala-2.2.0-cdh5.4.5/thirdparty/thrift-0.9.0/build/include/thrift/transport/TBufferTransports.h:556
#13 0x000000000111b042 in apache::thrift::transport::TMemoryBuffer::~TMemoryBuffer (this=0x7fa44f4b03c0, __in_chrg=<optimized out>) at /usr/src/debug/impala-2.2.0-cdh5.4.5/thirdparty/thrift-0.9.0/build/include/thrift/transport/TBufferTransports.h:558
#14 0x0000000000ef2ae8 in boost::checked_delete<apache::thrift::transport::TMemoryBuffer> (x=0x7fa44f4b03c0) at /opt/toolchain/boost-pic-1.55.0/include/boost/checked_delete.hpp:34
#15 0x0000000000ef3fb4 in boost::detail::sp_counted_impl_p<apache::thrift::transport::TMemoryBuffer>::dispose (this=0x7fa3c15d8620) at /opt/toolchain/boost-pic-1.55.0/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
#16 0x0000000000e6f1dc in boost::detail::sp_counted_base::release (this=0x7fa3c15d8620) at /opt/toolchain/boost-pic-1.55.0/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
#17 0x0000000000e6f26b in boost::detail::shared_count::~shared_count (this=0x7fa57821ac08, __in_chrg=<optimized out>) at /opt/toolchain/boost-pic-1.55.0/include/boost/smart_ptr/detail/shared_count.hpp:371
#18 0x0000000000ef1258 in boost::shared_ptr<apache::thrift::transport::TMemoryBuffer>::~shared_ptr (this=0x7fa57821ac00, __in_chrg=<optimized out>) at /opt/toolchain/boost-pic-1.55.0/include/boost/smart_ptr/shared_ptr.hpp:329
#19 0x0000000000f9bedc in impala::ThriftSerializer::~ThriftSerializer (this=0x7fa57821ac00, __in_chrg=<optimized out>) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/rpc/thrift-util.h:39
#20 0x0000000000fa4648 in impala::SerializeThriftMsg<impala::TUpdateCatalogCacheRequest const> (env=0x7fa40d40a9f8, msg=0x7fa57821bb70, serialized_msg=0x7fa57821acf8) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/rpc/jni-thrift-util.h:41
#21 0x0000000000fa08b1 in impala::JniUtil::CallJniMethod<impala::TUpdateCatalogCacheRequest, impala::TUpdateCatalogCacheResponse> (obj=@0xace86c8: 0x9dc9878, method=@0xace86f8: 0xb844600, arg=..., response=0x7fa57821bbd0)
    at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/util/jni-util.h:261
#22 0x0000000000f9ec78 in impala::Frontend::UpdateCatalogCache (this=0xace86c0, req=..., resp=0x7fa57821bbd0) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/service/frontend.cc:107
Python Exception <type 'exceptions.ValueError'> Cannot find type const impala::StatestoreSubscriber::TopicDeltaMap::_Rep_type: 
#23 0x0000000000fc9711 in impala::ImpalaServer::CatalogUpdateCallback (this=0x9f5c580, incoming_topic_deltas=std::map with 2 elements, subscriber_topic_updates=0x7fa57821c4c0) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/service/impala-server.cc:1206
Python Exception <type 'exceptions.ValueError'> Cannot find type const impala::StatestoreSubscriber::TopicDeltaMap::_Rep_type: 
#24 0x000000000102c606 in boost::_mfi::mf2<void, impala::ImpalaServer, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>::operator() (this=0xaa79b88, p=0x9f5c580, a1=std::map with 2 elements, a2=0x7fa57821c4c0) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/mem_fn_template.hpp:280
#25 0x0000000001028c38 in boost::_bi::list3<boost::_bi::value<impala::ImpalaServer*>, boost::arg<1>, boost::arg<2> >::operator()<boost::_mfi::mf2<void, impala::ImpalaServer, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>, boost::_bi::list2<std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*&> > (this=0xaa79b98, f=..., a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:392
Python Exception <type 'exceptions.ValueError'> Cannot find type const std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, impala::TTopicDelta, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, impala::TTopicDelta> > >::_Rep_type: 
#26 0x000000000102301c in boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ImpalaServer, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>, boost::_bi::list3<boost::_bi::value<impala::ImpalaServer*>, boost::arg<1>, boost::arg<2> > >::operator()<std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > >, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*> (this=0xaa79b88, a1=std::map with 2 elements, a2=@0x7fa57821c008: 0x7fa57821c4c0) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:76
Python Exception <type 'exceptions.ValueError'> Cannot find type const impala::StatestoreSubscriber::TopicDeltaMap::_Rep_type: 
#27 0x00000000010162ad in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ImpalaServer, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>, boost::_bi::list3<boost::_bi::value<impala::ImpalaServer*>, boost::arg<1>, boost::arg<2> > >, void, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>::invoke (function_obj_ptr=..., a0=std::map with 2 elements, a1=0x7fa57821c4c0) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:153
#28 0x000000000110423e in boost::function2<void, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>::operator() (
Python Exception <type 'exceptions.ValueError'> Cannot find type const impala::StatestoreSubscriber::TopicDeltaMap::_Rep_type: 
    this=0xaa79b80, a0=std::map with 2 elements, a1=0x7fa57821c4c0) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:767
Python Exception <type 'exceptions.ValueError'> Cannot find type const impala::StatestoreSubscriber::TopicDeltaMap::_Rep_type: 
#29 0x0000000001101aa6 in impala::StatestoreSubscriber::UpdateState (this=0xb832a00, incoming_topic_deltas=std::map with 2 elements, registration_id=..., subscriber_topic_updates=0x7fa57821c4c0, skipped=0x7fa57821c4d8)
    at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/statestore/statestore-subscriber.cc:359
#30 0x00000000011029bb in impala::StatestoreSubscriberThriftIf::UpdateState (this=0xaa065a0, response=..., params=...) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/statestore/statestore-subscriber.cc:76
#31 0x0000000001401fff in impala::StatestoreSubscriberProcessor::process_UpdateState (this=0xb314d80, seqid=0, iprot=0xa38d140, oprot=0xa38d100, callContext=0xa38d480) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/generated-sources/gen-cpp/StatestoreSubscriber.cpp:501
#32 0x0000000001401da4 in impala::StatestoreSubscriberProcessor::dispatchCall (this=0xb314d80, iprot=0xa38d140, oprot=0xa38d100, fname="UpdateState", seqid=0, callContext=0xa38d480) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/generated-sources/gen-cpp/StatestoreSubscriber.cpp:474
#33 0x0000000000fd0ede in apache::thrift::TDispatchProcessor::process (this=0xb314d80, in=..., out=..., connectionContext=0xa38d480) at /usr/src/debug/impala-2.2.0-cdh5.4.5/thirdparty/thrift-0.9.0/build/include/thrift/TDispatchProcessor.h:121
#34 0x0000000001ed9399 in apache::thrift::server::TThreadedServer::Task::run (this=0xb315a40) at src/thrift/server/TThreadedServer.cpp:70
#35 0x0000000000f0d06b in impala::ThriftThread::RunRunnable (this=0xa38d400, runnable=..., promise=0x7fa5cb4a1470) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/rpc/thrift-thread.cc:61
#36 0x0000000000f0e94d in boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>::operator() (this=0xa398120, p=0xa38d400, a1=..., a2=0x7fa5cb4a1470)
    at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/mem_fn_template.hpp:280
#37 0x0000000000f0e7a6 in boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long>*> >::operator()<boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list0> (this=0xa398130, f=..., a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:392
#38 0x0000000000f0e523 in boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long>*> > >::operator() (this=0xa398120) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:20
#39 0x0000000000f0e431 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long>*> > >, void>::invoke (function_obj_ptr=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:153
#40 0x0000000000f33c51 in boost::function0<void>::operator() (this=0x7fa57821ccc0) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:767
#41 0x0000000001192840 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name="StatestoreSubscriber-1", category="thrift-server", functor=..., thread_started=0x7fa5cb4a1270)
    at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/util/thread.cc:311
#42 0x000000000119b9dc in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0xa394dc0, 
    f=@0xa394db8: 0x11924ee <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:457
#43 0x000000000119b923 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0xa394db8) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:20
#44 0x000000000119b8e2 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0xa394c00) at /opt/toolchain/boost-pic-1.55.0/include/boost/thread/detail/thread.hpp:117
#45 0x000000000160d8b3 in ?? ()
#46 0x0000003bf5c079d1 in start_thread () from s/lib64/libpthread.so.0
#47 0x0000003bf58e88fd in clone () from s/lib64/libc.so.6

      And

      (gdb) f 13
#13 0x000000000111b042 in apache::thrift::transport::TMemoryBuffer::~TMemoryBuffer (this=0x7fa44f4b03c0, __in_chrg=<optimized out>) at /usr/src/debug/impala-2.2.0-cdh5.4.5/thirdparty/thrift-0.9.0/build/include/thrift/transport/TBufferTransports.h:558
558	  }
(gdb) p *this
$158 = {
  <apache::thrift::transport::TVirtualTransport<apache::thrift::transport::TMemoryBuffer, apache::thrift::transport::TBufferBase>> = {
    <apache::thrift::transport::TBufferBase> = {
      <apache::thrift::transport::TVirtualTransport<apache::thrift::transport::TBufferBase, apache::thrift::transport::TTransportDefaults>> = {
        <apache::thrift::transport::TTransportDefaults> = {
          <apache::thrift::transport::TTransport> = {
            _vptr.TTransport = 0x2847570 <vtable for apache::thrift::transport::TMemoryBuffer+16>
          }, <No data fields>}, <No data fields>}, 
      members of apache::thrift::transport::TBufferBase: 
      rBase_ = 0x7f9e8b05e000 "\002", 
      rBound_ = 0x7f9e8b05e000 "\002", 
      wBase_ = 0x7f9fc2b45f7c "\002", 
      wBound_ = 0x7f9f1b05e000 "vice/globalhiliter/data/ccb_ci_rwip/productname=cbf/process=6400/storeday=2014-09-20\b"
    }, <No data fields>}, 
  members of apache::thrift::transport::TMemoryBuffer: 
  static defaultSize = 1024, 
  buffer_ = 0x7f9e8b05e000 "\002", 
  bufferSize_ = 2415919104, 
  owner_ = true
}
(gdb) p wBound_-wBase_
$159 = -2813230972

      So wBound_ is lower then wBase_.
      In Thrift library: 0.9.0-p2

        uint32_t available_write() const {
    return static_cast<uint32_t>(wBound_ - wBase_);
  }


void TMemoryBuffer::writeSlow(const uint8_t* buf, uint32_t len) {
  ensureCanWrite(len);

  // Copy into the buffer and increment wBase_.
  memcpy(wBase_, buf, len);
  wBase_ += len;
}

and

void TMemoryBuffer::ensureCanWrite(uint32_t len) {
  // Check available space
  uint32_t avail = available_write();
  if (len <= avail) {
    return;
  }

  if (!owner_) {
    throw TTransportException("Insufficient space in external MemoryBuffer");
  }

  // Grow the buffer as necessary.
  uint32_t new_size = bufferSize_;    <---- Old buffer was 3355443200, a litter more than 3GB
  while (len > avail) {
    new_size = new_size > 0 ? new_size * 2 : 1;<----- New buffer overflowed and became 2415919104, which is consistent with the core dump
    avail = available_write() + (new_size - bufferSize_);<----- (new_size - bufferSize_) = 3355443200 (overflowed, but subtraction returns the expected result)
  }

  // Allocate into a new pointer so we don't bork ours if it fails.
  void* new_buffer = std::realloc(buffer_, new_size);
  if (new_buffer == NULL) {
    throw std::bad_alloc();
  }
  bufferSize_ = new_size;

  ptrdiff_t offset = (uint8_t*)new_buffer - buffer_;
  buffer_ += offset;                                                             
  rBase_ += offset;                                                             
  rBound_ += offset;                                                           
  wBase_ += offset;                                                            <----------- wBase_ was at an offseted location
  wBound_ = buffer_ + bufferSize_;                                   <----------- wBound_ shrinked because bufferSize_ was less than the old limit.

      Bug:
      1. It is unsafe to use value passed to realloc as Tim pointed out.
      2. once wBase_ += offset > wBound_ = buffer_ + bufferSize_, then available_write() may return very large result since it returns unsigned int. thus Thrift will continue writing even to invalid addresses. In this case wBase_-buffer=5229150076, but tcmalloc only allocated 2415919104 to thrift. so thrift wrote 5229150076 - 2415919104 = 2813230972 bytes to other places.

      This patch from Thrift fixes the first problem: https://github.com/apache/thrift/commit/6077481139933b927397c7da0088aa4678f9fb3c

      related link https://issues.apache.org/jira/browse/THRIFT-1248, https://issues.apache.org/jira/browse/THRIFT-3821

      I did not try to reproduce this.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-3552 Make incremental stats max serialized size configurable or dramatically increase limit.

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-3571 Thrift serialization can be very inefficient

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-3499 Backend cannot write catalog-update topic more than 2GB to jByteArray

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-3568 Possible Impala hangs and leak current catalog cache when state store sends full update

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-3605 Frontend OOM during catalog update may end up infinite catalog update loop

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-2649 improve incremental stats scalability

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-3625 Show JVM heap size in the logs and webUI

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Activity

            People

              HuaisiXu Huaisi Xu
              HuaisiXu Huaisi Xu
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: