Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5887

Hung union query

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Cannot Reproduce
    • Impala 2.10.0
    • None
    • None
    • None
    • ghx-label-9

    Description

      During an exhaustive test run on CentOS 7.0, I noticed the following query hung for 2.5 hours:

      select count(c) from ( select bigint_col + 1 as c from functional.alltypes limit 15 union all select bigint_col as c from functional.alltypes limit 15 union all select bigint_col + 1 as c from functional.alltypes limit 15 union all (select bigint_col as c from functional.alltypes limit 15)) t

      There was one fragment instance still running which was waiting on some hdfs scanner threads to complete. Unfortunately taking the pstack caused the threads to unblock themselves, and the query completed.

      Thread 1 (process 18704):
#0  0x00007f47592a9705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000013648e5 in boost::condition_variable::wait (this=0x1b303c858, m=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/pthread/condition_variable.hpp:73
#2  0x0000000001d89d8c in boost::thread::join_noexcept() ()
#3  0x00000000015148ab in boost::thread::join (this=0x181d458f0) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:767
#4  0x0000000001514f3e in impala::Thread::Join (this=0x1a983220) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/util/thread.h:106
#5  0x00000000017fa67e in impala::ThreadGroup::JoinAll (this=0x181d856c8) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/util/thread.cc:338
#6  0x000000000189be87 in impala::HdfsScanNode::Close (this=0x181d85000, state=0xbf38800) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:236
#7  0x0000000001a515e1 in impala::UnionNode::GetNextMaterialized (this=0x171f5680, state=0xbf38800, row_batch=0x7f469d2561f0) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/exec/union-node.cc:242
#8  0x0000000001a5272b in impala::UnionNode::GetNext (this=0x171f5680, state=0xbf38800, row_batch=0x7f469d2561f0, eos=0x7f469d25639f) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/exec/union-node.cc:297
#9  0x00000000019e4f09 in impala::PartitionedAggregationNode::Open (this=0x25251900, state=0xbf38800) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/exec/partitioned-aggregation-node.cc:302
#10 0x00000000015f74ab in impala::FragmentInstanceState::Open (this=0x150b3480) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/runtime/fragment-instance-state.cc:256
#11 0x00000000015f4fd7 in impala::FragmentInstanceState::Exec (this=0x150b3480) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/runtime/fragment-instance-state.cc:80
#12 0x00000000015bb5d2 in impala::QueryState::ExecFInstance (this=0x16cad600, fis=0x150b3480) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/runtime/query-state.cc:351
#13 0x00000000015ba102 in impala::QueryState::<lambda()>::operator()(void) const (__closure=0x7f469d256c28) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/runtime/query-state.cc:319
#14 0x00000000015bc283 in boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
#15 0x000000000152d542 in boost::function0<void>::operator() (this=0x7f469d256c20) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#16 0x00000000017fa4eb in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=..., thread_started=0x7f468003ec40) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/util/thread.cc:329
#17 0x0000000001802e26 in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x13f7cdfc0, f=@0x13f7cdfb8: 0x17fa1cc <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
#18 0x0000000001802d69 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x13f7cdfb8) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#19 0x0000000001802d2c in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0x13f7cde00) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
#20 0x0000000001d8935a in thread_proxy ()
#21 0x00007f47592a5df3 in start_thread () from /lib64/libpthread.so.0
#22 0x00007f4758fd31ad in clone () from /lib64/libc.so.6

      [root@impala-boost-static-burst-slave-el7-18a2 ~]# pstack 18717
Thread 1 (process 18717):
#0  0x00007f47592a9705 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x0000000001521f0f in impala::ConditionVariable::Wait (this=0x82f4940, lock=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/util/condition-variable.h:40
#2  0x00000000018883ad in impala::BlockingQueue<std::unique_ptr<impala::RowBatch, std::default_delete<impala::RowBatch> > >::BlockingPut<std::unique_ptr<impala::RowBatch, std::default_delete<impala::RowBatch> > >(std::unique_ptr<impala::RowBatch, std::default_delete<impala::RowBatch> >&&) (this=0x82f48c0, val=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/util/blocking-queue.h:120
#3  0x0000000001881d13 in impala::ExecNode::RowBatchQueue::AddBatch (this=0x82f48c0, batch=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/exec/exec-node.cc:94
#4  0x000000000189c0e5 in impala::HdfsScanNode::AddMaterializedRowBatch (this=0x181d85000, row_batch=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:257
#5  0x00000000018c5d44 in impala::HdfsScanner::ProcessSplit (this=0x145709180) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scanner.cc:126
#6  0x000000000189d833 in impala::HdfsScanNode::ProcessSplit (this=0x181d85000, filter_ctxs=..., scan_range=0xc898b180) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:528
#7  0x000000000189cc9a in impala::HdfsScanNode::ScannerThread (this=0x181d85000) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:418
#8  0x000000000189c407 in impala::HdfsScanNode::<lambda()>::operator()(void) const (__closure=0x7f4692026c28) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/exec/hdfs-scan-node.cc:355
#9  0x000000000189e161 in boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourceMgr::ResourcePool*)::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
#10 0x000000000152d542 in boost::function0<void>::operator() (this=0x7f4692026c20) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#11 0x00000000017fa4eb in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=..., thread_started=0x7f469d255310) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/util/thread.cc:329
#12 0x0000000001802e26 in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x1b303c9c0, f=@0x1b303c9b8: 0x17fa1cc <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
#13 0x0000000001802d69 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x1b303c9b8) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#14 0x0000000001802d2c in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0x1b303c800) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
#15 0x0000000001d8935a in thread_proxy ()
#16 0x00007f47592a5df3 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f4758fd31ad in clone () from /lib64/libc.so.6

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-5724 test_union hangs in exhaustive test run

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-6185 disk-io-mgr backend test gets stuck on RHEL7

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              sailesh Sailesh Mukil
              henryr Henry Robinson
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: