Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4707

Heap use-after-free in QueryExecMgr

    Details

      Description

      ASAN found a use-after-free in QueryExecMgr:

      =================================================================
      ==27045==ERROR: AddressSanitizer: heap-use-after-free on address 0x6180004549f0 at pc 0x0000015c6729 bp 0x7f3b20539f30 sp 0x7f3b20539f28
      READ of size 8 at 0x6180004549f0 thread T20705
          #0 0x15c6728 in impala::PrintId(impala::TUniqueId const&, std::string const&) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/debug-util.cc:107:20
          #1 0x1c60551 in impala::QueryExecMgr::ReleaseQueryState(impala::QueryState*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:138:599
          #2 0x1c6026e in impala::QueryExecMgr::ExecFInstance(impala::FragmentInstanceState*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:119:3
          #3 0x1c639b7 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20:16
          #4 0x12c4362 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/function/function_template.hpp:766:14
          #5 0x1681c95 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:317:3
          #6 0x168aa6a in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind.hpp:457:9
          #7 0x168a8f7 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20:16
          #8 0x1cd8539 in thread_proxy (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cd8539)
          #9 0x3591607850 in start_thread (/lib64/libpthread.so.0+0x3591607850)
          #10 0x35912e894c in clone (/lib64/libc.so.6+0x35912e894c)
      
      0x6180004549f0 is located 368 bytes inside of 864-byte region [0x618000454880,0x618000454be0)
      freed by thread T83 here:
          #0 0xffdd90 in operator delete(void*) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:94
          #1 0x1c607e4 in impala::QueryExecMgr::ReleaseQueryState(impala::QueryState*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:163:3
          #2 0x1c0fd44 in impala::Coordinator::TearDown() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/coordinator.cc:1929:5
          #3 0x1506d8a in impala::ImpalaServer::QueryExecState::Done() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/query-exec-state.cc:561:5
          #4 0x146e739 in impala::ImpalaServer::UnregisterQuery(impala::TUniqueId const&, bool, impala::Status const*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/impala-server.cc:923:3
          #5 0x14f6cc3 in impala::ImpalaServer::close(beeswax::QueryHandle const&) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/impala-beeswax-server.cc:235:29
          #6 0x1b6bc63 in beeswax::BeeswaxServiceProcessor::process_close(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:3543:5
          #7 0x1b648e9 in beeswax::BeeswaxServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::string const&, int, void*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:2952:3
          #8 0x1b3db0a in impala::ImpalaServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::string const&, int, void*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/generated-sources/gen-cpp/ImpalaService.cpp:1673:12
          #9 0x100340a in apache::thrift::TDispatchProcessor::process(boost::shared_ptr<apache::thrift::protocol::TProtocol>, boost::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/thrift-0.9.0-p8/include/thrift/TDispatchProcessor.h:121:12
          #10 0x2a5c62a in apache::thrift::server::TThreadPoolServer::Task::run() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x2a5c62a)
      
      previously allocated by thread T341 here:
          #0 0xffd790 in operator new(unsigned long) /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_new_delete.cc:62
          #1 0x1c5f624 in impala::QueryExecMgr::StartFInstance(impala::TExecPlanFragmentParams const&) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/runtime/query-exec-mgr.cc:67:12
          #2 0x14fe413 in impala::ImpalaInternalService::ExecPlanFragment(impala::TExecPlanFragmentResult&, impala::TExecPlanFragmentParams const&) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/service/impala-internal-service.cc:44:3
          #3 0x1b2d78e in impala::ImpalaInternalServiceProcessor::process_ExecPlanFragment(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:1397:5
          #4 0x1b2d269 in impala::ImpalaInternalServiceProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::string const&, int, void*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/generated-sources/gen-cpp/ImpalaInternalService.cpp:1370:3
          #5 0x100340a in apache::thrift::TDispatchProcessor::process(boost::shared_ptr<apache::thrift::protocol::TProtocol>, boost::shared_ptr<apache::thrift::protocol::TProtocol>, void*) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/thrift-0.9.0-p8/include/thrift/TDispatchProcessor.h:121:12
          #6 0x12b6465 in apache::thrift::server::TAcceptQueueServer::Task::run() /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/rpc/TAcceptQueueServer.cpp:76:14
          #7 0x12afb69 in impala::ThriftThread::RunRunnable(boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/rpc/thrift-thread.cc:64:3
          #8 0x12b220c in boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>::operator()(impala::ThriftThread*, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*) const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/mem_fn_template.hpp:280:16
          #9 0x12b2053 in void boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long>*> >::operator()<boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>&, boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind.hpp:392:9
          #10 0x12b1ef7 in boost::_bi::bind_t<void, boost::_mfi::mf2<void, impala::ThriftThread, boost::shared_ptr<apache::thrift::concurrency::Runnable>, impala::Promise<unsigned long>*>, boost::_bi::list3<boost::_bi::value<impala::ThriftThread*>, boost::_bi::value<boost::shared_ptr<apache::thrift::concurrency::Runnable> >, boost::_bi::value<impala::Promise<unsigned long>*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20:16
          #11 0x12c4362 in boost::function0<void>::operator()() const /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/function/function_template.hpp:766:14
          #12 0x1681c95 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:317:3
          #13 0x168aa6a in void boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind.hpp:457:9
          #14 0x168a8f7 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20:16
          #15 0x1cd8539 in thread_proxy (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cd8539)
      
      Thread T20705 created by T341 here:
          #0 0xf378e9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
          #1 0x1cd7919 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cd7919)
      
      Thread T341 created by T74 here:
          #0 0xf378e9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
          #1 0x1cd7919 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cd7919)
      
      Thread T74 created by T73 here:
          #0 0xf378e9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
          #1 0x1cd7919 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cd7919)
      
      Thread T73 created by T0 here:
          #0 0xf378e9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
          #1 0x1cd7919 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cd7919)
      
      Thread T83 created by T0 here:
          #0 0xf378e9 in __interceptor_pthread_create /data/jenkins/workspace/verify-impala-toolchain-package-build/label/ec2-package-centos-6/toolchain/source/llvm/llvm-3.8.0.src-p1/projects/compiler-rt/lib/asan/asan_interceptors.cc:238
          #1 0x1cd7919 in boost::thread::start_thread_noexcept() (/data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/build/debug/service/impalad+0x1cd7919)
      
      SUMMARY: AddressSanitizer: heap-use-after-free /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/debug-util.cc:107:20 in impala::PrintId(impala::TUniqueId const&, std::string const&)
      Shadow bytes around the buggy address:
        0x0c30800828e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c30800828f0: fd fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c3080082900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
        0x0c3080082910: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c3080082920: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
      =>0x0c3080082930: fd fd fd fd fd fd fd fd fd fd fd fd fd fd[fd]fd
        0x0c3080082940: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c3080082950: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c3080082960: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
        0x0c3080082970: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
        0x0c3080082980: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
      Shadow byte legend (one shadow byte represents 8 application bytes):
        Addressable:           00
        Partially addressable: 01 02 03 04 05 06 07 
        Heap left redzone:       fa
        Heap right redzone:      fb
        Freed heap region:       fd
        Stack left redzone:      f1
        Stack mid redzone:       f2
        Stack right redzone:     f3
        Stack partial redzone:   f4
        Stack after return:      f5
        Stack use after scope:   f8
        Global redzone:          f9
        Global init order:       f6
        Poisoned by user:        f7
        Container overflow:      fc
        Array cookie:            ac
        Intra object redzone:    bb
        ASan internal:           fe
        Left alloca redzone:     ca
        Right alloca redzone:    cb
      ==27045==ABORTING
      

      The following tests were running:

      23:14:16 [gw2] SKIPPED query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: string | decimal_type: (6, 6)] 
      23:14:16 query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: number | decimal_type: (6, 6)] 
      23:14:16 [gw2] SKIPPED query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: number | decimal_type: (6, 6)] 
      23:14:16 query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: number | decimal_type: (16, 0)] 
      23:14:16 [gw2] PASSED query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: number | decimal_type: (16, 0)] 
      23:14:16 query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: string | decimal_type: (16, 0)] 
      23:14:16 [gw2] PASSED query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: string | decimal_type: (16, 0)] 
      23:14:16 query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: string | decimal_type: (16, 1)] 
      23:14:16 [gw2] PASSED query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: string | decimal_type: (16, 1)] 
      23:14:16 query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: number | decimal_type: (16, 1)] 
      23:14:16 [gw2] PASSED query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: number | decimal_type: (16, 1)] 
      23:14:16 query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: number | decimal_type: (16, 2)] 
      23:14:16 [gw3] FAILED query_test/test_aggregation.py::TestAggregationQueries::test_aggregation[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      23:14:16 query_test/test_aggregation.py::TestAggregationQueries::test_aggregation[exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      23:14:16 [gw0] FAILED query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: number | decimal_type: (16, 2)] 
      23:14:16 query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: string | decimal_type: (16, 2)] 
      23:14:16 [gw1] FAILED metadata/test_compute_stats.py::TestComputeStats::test_compute_stats[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      23:14:16 [gw0] FAILED query_test/test_aggregation.py::TestAggregationQueries::test_aggregation[exec_option: {'disable_codegen': True, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      23:14:16 query_test/test_aggregation.py::TestAggregationQueries::test_distinct[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      23:14:16 [gw2] ERROR query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: string | decimal_type: (16, 2)] 
      23:14:16 query_test/test_decimal_casting.py::TestDecimalCasting::test_underflow[cast_from: string | decimal_type: (16, 3)] 
      23:14:16 [gw1] ERROR metadata/test_compute_stats.py::TestComputeStats::test_compute_stats[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      23:14:16 metadata/test_compute_stats.py::TestComputeStats::test_compute_stats_incremental[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] query_test/test_aggregation.py::TestAggregationQueries::test_distinct[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] INTERNALERROR> Traceback (most recent call last):
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: