Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4408

ASAN detected heap-buffer-overflow in Kudu scanner.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Backend
    • Labels:

      Description

      Repro:

      select * from functional_kudu.tinyinttable
      

      The problem is that Kudu does not allocate any null bits if all slots are non-nullable.

      Asan report:

      ASAN output:
      

      =================================================================
      ==17557==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6040017af378 at pc 0x000000fa1e55 bp 0x7f8358008ac0 sp 0x7f8358008270
      READ of size 5 at 0x6040017af378 thread T6829
      #0 0xfa1e54 in __asan_memcpy /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:393
      #1 0x1398fcc in impala::Tuple::DeepCopy(impala::Tuple*, impala::TupleDescriptor const&, impala::MemPool*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/tuple.cc:90:3
      #2 0x1945ed0 in impala::KuduScanner::DecodeRowsIntoRowBatch(impala::RowBatch*, impala::Tuple*, bool) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/kudu-scanner.cc:191:5
      #3 0x19455e6 in impala::KuduScanner::GetNext(impala::RowBatch*, bool*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/kudu-scanner.cc:104:33
      #4 0x18b7a64 in impala::KuduScanNode::ProcessScanToken(impala::KuduScanner*, std::string const&) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/kudu-scan-node.cc:265:31
      #5 0x18b71f5 in impala::KuduScanNode::RunScannerThread(std::string const&, std::string const*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/kudu-scan-node.cc:292:16
      #6 0x18ba017 in boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::KuduScanNode, std::string const&, std::string const*>, boost::_bi::list3<boost::_bi::value<impala::KuduScanNode*>, boost::_bi::value<std::string>, boost::_bi::value<std::string const*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20:16
      #7 0x12ad182 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/function/function_template.hpp:766:14
      #8 0x167af45 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:317:3
      #9 0x1683d1a in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>> >::operator()<void (std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>), boost::_bi::list0>(boost::_bi::type<void>, void (&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>), boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind.hpp:457:9
      #10 0x1683ba7 in boost::_bi::bind_t<void, void (std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20:16
      #11 0x1cf0099 in thread_proxy (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cf0099)
      #12 0x382f407850 in start_thread (/lib64/libpthread.so.0+0x382f407850)
      #13 0x382f0e894c in clone (/lib64/libc.so.6+0x382f0e894c)

      0x6040017af378 is located 0 bytes to the right of 40-byte region [0x6040017af350,0x6040017af378)
      allocated by thread T6827 (rpc reactor-855) here:
      #0 0xfe90c0 in operator new[](unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:64
      #1 0x7f889e1dd29a in kudu::faststring::GrowArray(unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/util/faststring.cc:41
      #2 0x7f889e1681b0 in kudu::faststring::reserve(unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/util/faststring.h:103
      #3 0x7f889e1681b0 in kudu::faststring::resize(unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/util/faststring.h:73
      #4 0x7f889e1681b0 in kudu::rpc::InboundTransfer::ReceiveBuffer(kudu::Socket&) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/rpc/transfer.cc:109
      #5 0x7f889e158ee9 in kudu::rpc::Connection::ReadHandler(ev::io&, int) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/rpc/connection.cc:453
      #6 0x7f889e158ee9 in void ev::base<ev_io, ev::io>::method_thunk<kudu::rpc::Connection, &kudu::rpc::Connection::ReadHandler>(ev_loop*, ev_io*, int) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/thirdparty/installed/include/ev++.h:479
      #7 0x7f889e2e6c1a in ev_invoke_pending /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/thirdparty/libev-4.20/ev.c:3155
      #8 0x7f889e2ea323 in ev_run /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/thirdparty/libev-4.20/ev.c:3555
      #9 0x7f889e1420fa in ev::loop_ref::run(int) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/thirdparty/installed/include/ev++.h:211
      #10 0x7f889e1420fa in kudu::rpc::ReactorThread::RunThread() /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/rpc/reactor.cc:306
      #11 0x7f889e210ce9 in boost::function0<void>::operator()() const /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/thirdparty/installed/include/boost/function/function_template.hpp:771
      #12 0x7f889e210ce9 in kudu::Thread::SuperviseThread(void*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/util/thread.cc:588
      #13 0x382f407850 in start_thread (/lib64/libpthread.so.0+0x382f407850)

      Thread T6829 created by T6819 here:
      #0 0xf230b9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      #1 0x1cef479 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cef479)

      Thread T6819 created by T4457 here:
      #0 0xf230b9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      #1 0x1cef479 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cef479)

      Thread T4457 created by T74 here:
      #0 0xf230b9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      #1 0x1cef479 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cef479)

      Thread T74 created by T73 here:
      #0 0xf230b9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      #1 0x1cef479 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cef479)

      Thread T73 created by T0 here:
      #0 0xf230b9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      #1 0x1cef479 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cef479)

      Thread T6827 (rpc reactor-855) created by T6819 here:
      #0 0xf230b9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
      #1 0x7f889e20d26b in kudu::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()> const&, unsigned long, scoped_refptr<kudu::Thread>*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/util/thread.cc:511
      #2 0x7f889e142392 in Create<void (kudu::rpc::ReactorThread::)(), kudu::rpc::ReactorThread> /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/util/thread.h:158
      #3 0x7f889e142392 in kudu::rpc::ReactorThread::Init() /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/rpc/reactor.cc:108
      #4 0x7f889e1437e4 in kudu::rpc::Reactor::Init() /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/rpc/reactor.cc:488
      #5 0x7f889e140bdb in kudu::rpc::Messenger::Init() /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/rpc/messenger.cc:266
      #6 0x7f889e140bdb in kudu::rpc::MessengerBuilder::Build(kudu::rpc::Messenger**) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/rpc/messenger.cc:106
      #7 0x7f889e140cef in kudu::rpc::MessengerBuilder::Build(std::shared_ptr<kudu::rpc::Messenger>*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/rpc/messenger.cc:113
      #8 0x7f889e0dc1db in kudu::client::KuduClientBuilder::Build(std::tr1::shared_ptr<kudu::client::KuduClient>*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/kudu/kudu-1.0.0-RC1/src/kudu/client/client.cc:230
      #9 0x18b4c90 in impala::KuduScanNode::Open(impala::RuntimeState*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/kudu-scan-node.cc:129:27
      #10 0x185f24c in impala::PartitionedAggregationNode::Open(impala::RuntimeState*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/partitioned-aggregation-node.cc:317:29
      #11 0x1c789b2 in impala::PlanFragmentExecutor::OpenInternal() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:322:31
      #12 0x1c784c5 in impala::PlanFragmentExecutor::Open() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/plan-fragment-executor.cc:295:19
      #13 0x1555fb1 in impala::FragmentMgr::FragmentExecState::Exec() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/fragment-exec-state.cc:58:5
      #14 0x1548bf5 in impala::FragmentMgr::FragmentThread(impala::TUniqueId) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/fragment-mgr.cc:86:3
      #15 0x154e26d in boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>::operator()(impala::FragmentMgr*, impala::TUniqueId) const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/mem_fn_template.hpp:165:16
      #16 0x154e0c7 in void boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> >::operator()<boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>&, boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind.hpp:313:9
      #17 0x154df77 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20:16
      #18 0x12ad182 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/function/function_template.hpp:766:14
      #19 0x167af45 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:317:3
      #20 0x1683d1a in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>> >::operator()<void (std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>), boost::_bi::list0>(boost::_bi::type<void>, void (&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>), boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind.hpp:457:9
      #21 0x1683ba7 in boost::_bi::bind_t<void, void (std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20:16
      #22 0x1cf0099 in thread_proxy (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cf0099)

      SUMMARY: AddressSanitizer: heap-buffer-overflow /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:393 in __asan_memcpy
      Shadow bytes around the buggy address:
      0x0c08802ede10: fa fa fd fd fd fd fd fd fa fa fd fd fd fd fd fa
      0x0c08802ede20: fa fa fd fd fd fd fd fd fa fa fa fa fa fa fa fa
      0x0c08802ede30: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c08802ede40: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c08802ede50: fa fa fd fd fd fd fd fa fa fa fa fa fa fa fa fa
      =>0x0c08802ede60: fa fa fa fa fa fa fa fa fa fa 00 00 00 00 00[fa]
      0x0c08802ede70: fa fa fa fa fa fa fa fa fa fa fd fd fd fd fd fa
      0x0c08802ede80: fa fa fa fa fa fa fa fa fa fa 00 00 00 00 00 fa
      0x0c08802ede90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      0x0c08802edea0: fa fa fd fd fd fd fd fa fa fa fd fd fd fd fd fd
      0x0c08802edeb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      Shadow byte legend (one shadow byte represents 8 application bytes):
      Addressable: 00
      Partially addressable: 01 02 03 04 05 06 07
      Heap left redzone: fa
      Heap right redzone: fb
      Freed heap region: fd
      Stack left redzone: f1
      Stack mid redzone: f2
      Stack right redzone: f3
      Stack partial redzone: f4
      Stack after return: f5
      Stack use after scope: f8
      Global redzone: f9
      Global init order: f6
      Poisoned by user: f7
      Container overflow: fc
      Array cookie: ac
      Intra object redzone: bb
      ASan internal: fe
      Left alloca redzone: ca
      Right alloca redzone: cb
      ==17557==ABORTING

      
      

        Activity

        Hide
        lv Lars Volker added a comment -

        IMPALA-4408: Omit null bytes for Kudu scans with no nullable slots.

        Kudu does not allocate null bytes if all projected columns are
        non-nullable. Otherwise, Kudu allocates a null bit for all columns,
        even the non-nullable ones. The bug was that Impala's memory layout
        did not match the first requirement.

        Change-Id: I762ad9d5cc4198922ea4b5218c504fde355c49a5
        Reviewed-on: http://gerrit.cloudera.org:8080/4892
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        lv Lars Volker added a comment - IMPALA-4408 : Omit null bytes for Kudu scans with no nullable slots. Kudu does not allocate null bytes if all projected columns are non-nullable. Otherwise, Kudu allocates a null bit for all columns, even the non-nullable ones. The bug was that Impala's memory layout did not match the first requirement. Change-Id: I762ad9d5cc4198922ea4b5218c504fde355c49a5 Reviewed-on: http://gerrit.cloudera.org:8080/4892 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            alex.behm Alexander Behm
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development