Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3494

Thrift buffer overflows when serialize more than 3355443200 bytes in impala

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: Impala 2.6.0
    • Component/s: Backend
    • Labels:

      Description

      Back trace:

      (gdb) bt
      #0  0x0000003bf5832625 in raise () from s/lib64/libc.so.6
      #1  0x0000003bf5833e05 in abort () from s/lib64/libc.so.6
      #2  0x00007fa6cc9517b5 in os::abort(bool) () from s/usr/java/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so
      #3  0x00007fa6ccaeeea3 in VMError::report_and_die() () from s/usr/java/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so
      #4  0x00007fa6cc956cdf in JVM_handle_linux_signal () from s/usr/java/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so
      #5  0x00007fa6cc94d593 in signalHandler(int, siginfo*, void*) () from s/usr/java/jdk1.8.0_40/jre/lib/amd64/server/libjvm.so
      #6  <signal handler called>
      #7  0x0000000001f7cc55 in tcmalloc::PageHeap::MergeIntoFreeList (this=0x333c000, span=0x7fa24d38c340) at src/page_heap.cc:227
      #8  0x0000000001f7cf2d in tcmalloc::PageHeap::Delete (this=0x333c000, span=<optimized out>) at src/page_heap.cc:197
      #9  0x0000000001f927b5 in (anonymous namespace)::do_free_with_callback (ptr=<optimized out>, invalid_free_fn=<optimized out>) at src/tcmalloc.cc:1147
      #10 (anonymous namespace)::do_free (ptr=<optimized out>) at src/tcmalloc.cc:1153
      #11 tc_free (ptr=<optimized out>) at src/tcmalloc.cc:1499
      #12 0x000000000111b006 in apache::thrift::transport::TMemoryBuffer::~TMemoryBuffer (this=0x7fa44f4b03c0, __in_chrg=<optimized out>) at /usr/src/debug/impala-2.2.0-cdh5.4.5/thirdparty/thrift-0.9.0/build/include/thrift/transport/TBufferTransports.h:556
      #13 0x000000000111b042 in apache::thrift::transport::TMemoryBuffer::~TMemoryBuffer (this=0x7fa44f4b03c0, __in_chrg=<optimized out>) at /usr/src/debug/impala-2.2.0-cdh5.4.5/thirdparty/thrift-0.9.0/build/include/thrift/transport/TBufferTransports.h:558
      #14 0x0000000000ef2ae8 in boost::checked_delete<apache::thrift::transport::TMemoryBuffer> (x=0x7fa44f4b03c0) at /opt/toolchain/boost-pic-1.55.0/include/boost/checked_delete.hpp:34
      #15 0x0000000000ef3fb4 in boost::detail::sp_counted_impl_p<apache::thrift::transport::TMemoryBuffer>::dispose (this=0x7fa3c15d8620) at /opt/toolchain/boost-pic-1.55.0/include/boost/smart_ptr/detail/sp_counted_impl.hpp:78
      #16 0x0000000000e6f1dc in boost::detail::sp_counted_base::release (this=0x7fa3c15d8620) at /opt/toolchain/boost-pic-1.55.0/include/boost/smart_ptr/detail/sp_counted_base_gcc_x86.hpp:146
      #17 0x0000000000e6f26b in boost::detail::shared_count::~shared_count (this=0x7fa57821ac08, __in_chrg=<optimized out>) at /opt/toolchain/boost-pic-1.55.0/include/boost/smart_ptr/detail/shared_count.hpp:371
      #18 0x0000000000ef1258 in boost::shared_ptr<apache::thrift::transport::TMemoryBuffer>::~shared_ptr (this=0x7fa57821ac00, __in_chrg=<optimized out>) at /opt/toolchain/boost-pic-1.55.0/include/boost/smart_ptr/shared_ptr.hpp:329
      #19 0x0000000000f9bedc in impala::ThriftSerializer::~ThriftSerializer (this=0x7fa57821ac00, __in_chrg=<optimized out>) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/rpc/thrift-util.h:39
      #20 0x0000000000fa4648 in impala::SerializeThriftMsg<impala::TUpdateCatalogCacheRequest const> (env=0x7fa40d40a9f8, msg=0x7fa57821bb70, serialized_msg=0x7fa57821acf8) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/rpc/jni-thrift-util.h:41
      #21 0x0000000000fa08b1 in impala::JniUtil::CallJniMethod<impala::TUpdateCatalogCacheRequest, impala::TUpdateCatalogCacheResponse> (obj=@0xace86c8: 0x9dc9878, method=@0xace86f8: 0xb844600, arg=..., response=0x7fa57821bbd0)
          at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/util/jni-util.h:261
      #22 0x0000000000f9ec78 in impala::Frontend::UpdateCatalogCache (this=0xace86c0, req=..., resp=0x7fa57821bbd0) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/service/frontend.cc:107
      Python Exception <type 'exceptions.ValueError'> Cannot find type const impala::StatestoreSubscriber::TopicDeltaMap::_Rep_type: 
      #23 0x0000000000fc9711 in impala::ImpalaServer::CatalogUpdateCallback (this=0x9f5c580, incoming_topic_deltas=std::map with 2 elements, subscriber_topic_updates=0x7fa57821c4c0) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/service/impala-server.cc:1206
      Python Exception <type 'exceptions.ValueError'> Cannot find type const impala::StatestoreSubscriber::TopicDeltaMap::_Rep_type: 
      #24 0x000000000102c606 in boost::_mfi::mf2<void, impala::ImpalaServer, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>::operator() (this=0xaa79b88, p=0x9f5c580, a1=std::map with 2 elements, a2=0x7fa57821c4c0) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/mem_fn_template.hpp:280
      #25 0x0000000001028c38 in boost::_bi::list3<boost::_bi::value<impala::ImpalaServer*>, boost::arg<1>, boost::arg<2> >::operator()<boost::_mfi::mf2<void, impala::ImpalaServer, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>, boost::_bi::list2<std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*&> > (this=0xaa79b98, f=..., a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:392
      Python Exception <type 'exceptions.ValueError'> Cannot find type const std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, impala::TTopicDelta, std::less<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, impala::TTopicDelta> > >::_Rep_type: 
      #26 0x000000000102301c in boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ImpalaServer, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>, boost::_bi::list3<boost::_bi::value<impala::ImpalaServer*>, boost::arg<1>, boost::arg<2> > >::operator()<std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > >, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*> (this=0xaa79b88, a1=std::map with 2 elements, a2=@0x7fa57821c008: 0x7fa57821c4c0) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:76
      Python Exception <type 'exceptions.ValueError'> Cannot find type const impala::StatestoreSubscriber::TopicDeltaMap::_Rep_type: 
      #27 0x00000000010162ad in boost::detail::function::void_function_obj_invoker2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ImpalaServer, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>, boost::_bi::list3<boost::_bi::value<impala::ImpalaServer*>, boost::arg<1>, boost::arg<2> > >, void, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>::invoke (function_obj_ptr=..., a0=std::map with 2 elements, a1=0x7fa57821c4c0) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:153
      #28 0x000000000110423e in boost::function2<void, std::map<std::string, impala::TTopicDelta, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TTopicDelta> > > const&, std::vector<impala::TTopicDelta, std::allocator<impala::TTopicDelta> >*>::operator() (
      Python Exception <type 'exceptions.ValueError'> Cannot find type const impala::StatestoreSubscriber::TopicDeltaMap::_Rep_type: 
          this=0xaa79b80, a0=std::map with 2 elements, a1=0x7fa57821c4c0) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:767
      Python Exception <type 'exceptions.ValueError'> Cannot find type const impala::StatestoreSubscriber::TopicDeltaMap::_Rep_type: 
      #29 0x0000000001101aa6 in impala::StatestoreSubscriber::UpdateState (this=0xb832a00, incoming_topic_deltas=std::map with 2 elements, registration_id=..., subscriber_topic_updates=0x7fa57821c4c0, skipped=0x7fa57821c4d8)
          at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/statestore/statestore-subscriber.cc:359
      #30 0x00000000011029bb in impala::StatestoreSubscriberThriftIf::UpdateState (this=0xaa065a0, response=..., params=...) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/statestore/statestore-subscriber.cc:76
      #31 0x0000000001401fff in impala::StatestoreSubscriberProcessor::process_UpdateState (this=0xb314d80, seqid=0, iprot=0xa38d140, oprot=0xa38d100, callContext=0xa38d480) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/generated-sources/gen-cpp/StatestoreSubscriber.cpp:501
      #32 0x0000000001401da4 in impala::StatestoreSubscriberProcessor::dispatchCall (this=0xb314d80, iprot=0xa38d140, oprot=0xa38d100, fname="UpdateState", seqid=0, callContext=0xa38d480) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/generated-sources/gen-cpp/StatestoreSubscriber.cpp:474
      #33 0x0000000000fd0ede in apache::thrift::TDispatchProcessor::process (this=0xb314d80, in=..., out=..., connectionContext=0xa38d480) at /usr/src/debug/impala-2.2.0-cdh5.4.5/thirdparty/thrift-0.9.0/build/include/thrift/TDispatchProcessor.h:121
      #34 0x0000000001ed9399 in apache::thrift::server::TThreadedServer::Task::run (this=0xb315a40) at src/thrift/server/TThreadedServer.cpp:70
      #35 0x0000000000f0d06b in impala::ThriftThread::RunRunnable (this=0xa38d400, runnable=..., promise=0x7fa5cb4a1470) at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/rpc/thrift-thread.cc:61
      #36 0x0000000000f0e94d in boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>::operator() (this=0xa398120, p=0xa38d400, a1=..., a2=0x7fa5cb4a1470)
          at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/mem_fn_template.hpp:280
      #37 0x0000000000f0e7a6 in boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long>*> >::operator()<boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list0> (this=0xa398130, f=..., a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:392
      #38 0x0000000000f0e523 in boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long>*> > >::operator() (this=0xa398120) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:20
      #39 0x0000000000f0e431 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long>*> > >, void>::invoke (function_obj_ptr=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:153
      #40 0x0000000000f33c51 in boost::function0<void>::operator() (this=0x7fa57821ccc0) at /opt/toolchain/boost-pic-1.55.0/include/boost/function/function_template.hpp:767
      #41 0x0000000001192840 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name="StatestoreSubscriber-1", category="thrift-server", functor=..., thread_started=0x7fa5cb4a1270)
          at /usr/src/debug/impala-2.2.0-cdh5.4.5/be/src/util/thread.cc:311
      #42 0x000000000119b9dc in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0xa394dc0, 
          f=@0xa394db8: 0x11924ee <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind.hpp:457
      #43 0x000000000119b923 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0xa394db8) at /opt/toolchain/boost-pic-1.55.0/include/boost/bind/bind_template.hpp:20
      #44 0x000000000119b8e2 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0xa394c00) at /opt/toolchain/boost-pic-1.55.0/include/boost/thread/detail/thread.hpp:117
      #45 0x000000000160d8b3 in ?? ()
      #46 0x0000003bf5c079d1 in start_thread () from s/lib64/libpthread.so.0
      #47 0x0000003bf58e88fd in clone () from s/lib64/libc.so.6
      

      And

      (gdb) f 13
      #13 0x000000000111b042 in apache::thrift::transport::TMemoryBuffer::~TMemoryBuffer (this=0x7fa44f4b03c0, __in_chrg=<optimized out>) at /usr/src/debug/impala-2.2.0-cdh5.4.5/thirdparty/thrift-0.9.0/build/include/thrift/transport/TBufferTransports.h:558
      558	  }
      (gdb) p *this
      $158 = {
        <apache::thrift::transport::TVirtualTransport<apache::thrift::transport::TMemoryBuffer, apache::thrift::transport::TBufferBase>> = {
          <apache::thrift::transport::TBufferBase> = {
            <apache::thrift::transport::TVirtualTransport<apache::thrift::transport::TBufferBase, apache::thrift::transport::TTransportDefaults>> = {
              <apache::thrift::transport::TTransportDefaults> = {
                <apache::thrift::transport::TTransport> = {
                  _vptr.TTransport = 0x2847570 <vtable for apache::thrift::transport::TMemoryBuffer+16>
                }, <No data fields>}, <No data fields>}, 
            members of apache::thrift::transport::TBufferBase: 
            rBase_ = 0x7f9e8b05e000 "\002", 
            rBound_ = 0x7f9e8b05e000 "\002", 
            wBase_ = 0x7f9fc2b45f7c "\002", 
            wBound_ = 0x7f9f1b05e000 "vice/globalhiliter/data/ccb_ci_rwip/productname=cbf/process=6400/storeday=2014-09-20\b"
          }, <No data fields>}, 
        members of apache::thrift::transport::TMemoryBuffer: 
        static defaultSize = 1024, 
        buffer_ = 0x7f9e8b05e000 "\002", 
        bufferSize_ = 2415919104, 
        owner_ = true
      }
      (gdb) p wBound_-wBase_
      $159 = -2813230972
      

      So wBound_ is lower then wBase_.
      In Thrift library: 0.9.0-p2

        uint32_t available_write() const {
          return static_cast<uint32_t>(wBound_ - wBase_);
        }
      
      
      void TMemoryBuffer::writeSlow(const uint8_t* buf, uint32_t len) {
        ensureCanWrite(len);
      
        // Copy into the buffer and increment wBase_.
        memcpy(wBase_, buf, len);
        wBase_ += len;
      }
      
      and
      
      void TMemoryBuffer::ensureCanWrite(uint32_t len) {
        // Check available space
        uint32_t avail = available_write();
        if (len <= avail) {
          return;
        }
      
        if (!owner_) {
          throw TTransportException("Insufficient space in external MemoryBuffer");
        }
      
        // Grow the buffer as necessary.
        uint32_t new_size = bufferSize_;    <---- Old buffer was 3355443200, a litter more than 3GB
        while (len > avail) {
          new_size = new_size > 0 ? new_size * 2 : 1;<----- New buffer overflowed and became 2415919104, which is consistent with the core dump
          avail = available_write() + (new_size - bufferSize_);<----- (new_size - bufferSize_) = 3355443200 (overflowed, but subtraction returns the expected result)
        }
      
        // Allocate into a new pointer so we don't bork ours if it fails.
        void* new_buffer = std::realloc(buffer_, new_size);
        if (new_buffer == NULL) {
          throw std::bad_alloc();
        }
        bufferSize_ = new_size;
      
        ptrdiff_t offset = (uint8_t*)new_buffer - buffer_;
        buffer_ += offset;                                                             
        rBase_ += offset;                                                             
        rBound_ += offset;                                                           
        wBase_ += offset;                                                            <----------- wBase_ was at an offseted location
        wBound_ = buffer_ + bufferSize_;                                   <----------- wBound_ shrinked because bufferSize_ was less than the old limit.
      

      Bug:
      1. It is unsafe to use value passed to realloc as Tim pointed out.
      2. once wBase_ += offset > wBound_ = buffer_ + bufferSize_, then available_write() may return very large result since it returns unsigned int. thus Thrift will continue writing even to invalid addresses. In this case wBase_-buffer=5229150076, but tcmalloc only allocated 2415919104 to thrift. so thrift wrote 5229150076 - 2415919104 = 2813230972 bytes to other places.

      This patch from Thrift fixes the first problem: https://github.com/apache/thrift/commit/6077481139933b927397c7da0088aa4678f9fb3c

      related link https://issues.apache.org/jira/browse/THRIFT-1248, https://issues.apache.org/jira/browse/THRIFT-3821

      I did not try to reproduce this.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                HuaisiXu Huaisi Xu
                Reporter:
                HuaisiXu Huaisi Xu
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: