Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5188

DCHECK in UnionNode::GetNextPassThrough with GROUP BY, AVG

    Details

      Description

      This query:

      SELECT
      1,
      1
      UNION ALL
      SELECT
      AVG(id),
      id
      FROM functional.alltypes
      GROUP BY
      id
      

      ... is sufficient enough to hit the DCHECK, which is:

      union-node.cc:113] Check failed: child(child_idx_)->row_desc().LayoutEquals(row_batch->row_desc())
      
      #6  0x00000000028c502e in google::LogMessageFatal::~LogMessageFatal() ()
      #7  0x0000000001801da4 in impala::UnionNode::GetNextPassThrough (this=0xadff900, state=0xa704000, row_batch=0x7e03960) at /home/mikeb/Impala/be/src/exec/union-node.cc:113
      #8  0x00000000018043d7 in impala::UnionNode::GetNext (this=0xadff900, state=0xa704000, row_batch=0x7e03960, eos=0x7eff74b2cebf) at /home/mikeb/Impala/be/src/exec/union-node.cc:255
      #9  0x0000000001a73940 in impala::PlanFragmentExecutor::ExecInternal (this=0xa3c33d0) at /home/mikeb/Impala/be/src/runtime/plan-fragment-executor.cc:360
      #10 0x0000000001a735ec in impala::PlanFragmentExecutor::Exec (this=0xa3c33d0) at /home/mikeb/Impala/be/src/runtime/plan-fragment-executor.cc:337
      #11 0x0000000001a6cd8b in impala::FragmentInstanceState::Exec (this=0xa3c3100) at /home/mikeb/Impala/be/src/runtime/fragment-instance-state.cc:68
      #12 0x0000000001a783ef in impala::QueryExecMgr::ExecFInstance (this=0x9e01020, fis=0xa3c3100) at /home/mikeb/Impala/be/src/runtime/query-exec-mgr.cc:110
      #13 0x0000000001a7b218 in boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>::operator() (this=0x8ee6920, p=0x9e01020, a1=0xa3c3100)
          at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/bind/mem_fn_template.hpp:165
      #14 0x0000000001a7b0a1 in boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> >::operator()<boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list0> (this=0x8ee6930, f=..., a=...) at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:313
      #15 0x0000000001a7abd5 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >::operator() (this=0x8ee6920) at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
      #16 0x0000000001a7a7e0 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, impala::FragmentInstanceState*>, boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, boost::_bi::value<impala::FragmentInstanceState*> > >, void>::invoke (function_obj_ptr=...)
          at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:153
      #17 0x000000000137bd48 in boost::function0<void>::operator() (this=0x7eff74b2dd20) at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/function/function_template.hpp:767
      #18 0x000000000162e363 in impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*) (name=..., category=..., functor=...,
          thread_started=0x7eff7632fa80) at /home/mikeb/Impala/be/src/util/thread.cc:325
      #19 0x0000000001636d3e in boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> >::operator()<void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list0&, int) (this=0x9eb77c0,
          f=@0x9eb77b8: 0x162e09e <impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*)>, a=...)
          at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/bind/bind.hpp:457
      #20 0x0000000001636c81 in boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > >::operator()() (this=0x9eb77b8)
          at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/bind/bind_template.hpp:20
      #21 0x0000000001636c44 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()>, impala::Promise<long>*), boost::_bi::list4<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, boost::_bi::value<impala::Promise<long>*> > > >::run() (this=0x9eb7600)
          at /home/mikeb/Impala/toolchain/boost-1.57.0-p1/include/boost/thread/detail/thread.hpp:116
      #22 0x0000000001af270a in thread_proxy ()
      #23 0x00007f0000172184 in start_thread (arg=0x7eff74b2e700) at pthread_create.c:312
      #24 0x00007effffe9f37d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
      

      Plan:

      +-----------------------------------------------------------+
      | Explain String                                            |
      +-----------------------------------------------------------+
      | Estimated Per-Host Requirements: Memory=180.00MB VCores=2 |
      |                                                           |
      | PLAN-ROOT SINK                                            |
      | |                                                         |
      | 05:EXCHANGE [UNPARTITIONED]                               |
      | |                                                         |
      | 00:UNION                                                  |
      | |  constant-operands=1                                    |
      | |  pass-through-operands: all                             |
      | |                                                         |
      | 04:AGGREGATE [FINALIZE]                                   |
      | |  output: avg:merge(id)                                  |
      | |  group by: id                                           |
      | |                                                         |
      | 03:EXCHANGE [HASH(id)]                                    |
      | |                                                         |
      | 02:AGGREGATE [STREAMING]                                  |
      | |  output: avg(id)                                        |
      | |  group by: id                                           |
      | |                                                         |
      | 01:SCAN HDFS [functional.alltypes]                        |
      |    partitions=24/24 files=24 size=478.45KB                |
      +-----------------------------------------------------------+
      

      That's as simple a query I could get. The AVG() and GROUP BY together seemed to be needed. For example, this query creates a similar plan but does not hit the dcheck:

      SELECT
      1
      UNION ALL
      SELECT
      id
      FROM functional.alltypes
      GROUP BY
      id
      

      Original query generator query:

      USE tpch_kudu;
      SELECT DISTINCT
      a1.n_nationkey,
      CAST(COALESCE(a1.n_nationkey, a1.n_regionkey, a1.n_nationkey) AS STRING) AS char_col,
      COALESCE(a1.n_regionkey, GREATEST(COALESCE(a1.n_nationkey, -70), COALESCE(a1.n_nationkey, -909)), a1.n_nationkey) AS int_col,
      COALESCE(a1.n_regionkey, a1.n_nationkey, COALESCE(a1.n_nationkey, a1.n_nationkey, LEAST(COALESCE(a1.n_nationkey, -273), COALESCE(a1.n_nationkey, 967)))) AS int_col_1
      FROM nation a1
      UNION ALL
      SELECT
      AVG(COALESCE(a2.s_suppkey, a2.s_nationkey, LEAST(COALESCE(a2.s_suppkey, 547), COALESCE(a2.s_suppkey, -534)))) AS float_col,
      CAST(COALESCE(a3.p_partkey, a3.p_partkey, a2.s_nationkey) AS STRING) AS char_col,
      AVG(a2.s_suppkey) AS float_col_1,
      GREATEST(COALESCE(LEAST(COALESCE(a2.s_suppkey, -764), 178.6745398063), 455.7368346861), COALESCE(a2.s_suppkey, 720)) AS decimal_col
      FROM supplier a2
      INNER JOIN part a3 ON (a2.s_nationkey) = (a3.p_size)
      GROUP BY
      CAST(COALESCE(a3.p_partkey, a3.p_partkey, a2.s_nationkey) AS STRING),
      GREATEST(COALESCE(LEAST(COALESCE(a2.s_suppkey, -764), 178.6745398063), 455.7368346861), COALESCE(a2.s_suppkey, 720))
      

        Issue Links

          Activity

          Hide
          mikesbrown Michael Brown added a comment -

          Also, is it right that the union node in the plan is numbered as zero? (00:UNION) That's doesn't seem correct to me.

          Show
          mikesbrown Michael Brown added a comment - Also, is it right that the union node in the plan is numbered as zero? ( 00:UNION ) That's doesn't seem correct to me.
          Hide
          mikesbrown Michael Brown added a comment -

          Taras Bobrovytsky I assigned this to you under the assumption that this bug is a regression caused by IMPALA-3586: Implement union passthrough.

          Show
          mikesbrown Michael Brown added a comment - Taras Bobrovytsky I assigned this to you under the assumption that this bug is a regression caused by IMPALA-3586 : Implement union passthrough .
          Hide
          tarasbob Taras Bobrovytsky added a comment -

          Nice catch! I agree, this is most likely a regression that's caused by my change. I'll investigate.

          Show
          tarasbob Taras Bobrovytsky added a comment - Nice catch! I agree, this is most likely a regression that's caused by my change. I'll investigate.
          Hide
          tarasbob Taras Bobrovytsky added a comment -
          commit 50e3abdc3daf89c76d0d40e7f8c926477c2a1db4
          Author: Taras Bobrovytsky <tbobrovytsky@cloudera.com>
          Date:   Tue Apr 11 14:07:19 2017 -0700
          
              IMPALA-5188: Add slot sorting in TupleDescriptor::LayoutEquals()
              
              The slot descriptor vectors are not guaranteed to be sorted on the slot
              index within a tuple. As a result, TupleDescriptor::LayoutEquals()
              sometimes returned a wrong result.
              
              In this patch, we sort the vectors of slot descriptors on the slot index
              within the tuple before comparing the vectors.
              
              Testing:
              - ran EE tests locally.
              
              Change-Id: I426ad244678dbfe517262dfb7bbf4adc0247a35e
              Reviewed-on: http://gerrit.cloudera.org:8080/6610
              Reviewed-by: Dan Hecht <dhecht@cloudera.com>
              Reviewed-by: Alex Behm <alex.behm@cloudera.com>
              Tested-by: Impala Public Jenkins
          
          Show
          tarasbob Taras Bobrovytsky added a comment - commit 50e3abdc3daf89c76d0d40e7f8c926477c2a1db4 Author: Taras Bobrovytsky <tbobrovytsky@cloudera.com> Date: Tue Apr 11 14:07:19 2017 -0700 IMPALA-5188: Add slot sorting in TupleDescriptor::LayoutEquals() The slot descriptor vectors are not guaranteed to be sorted on the slot index within a tuple. As a result, TupleDescriptor::LayoutEquals() sometimes returned a wrong result. In this patch, we sort the vectors of slot descriptors on the slot index within the tuple before comparing the vectors. Testing: - ran EE tests locally. Change-Id: I426ad244678dbfe517262dfb7bbf4adc0247a35e Reviewed-on: http: //gerrit.cloudera.org:8080/6610 Reviewed-by: Dan Hecht <dhecht@cloudera.com> Reviewed-by: Alex Behm <alex.behm@cloudera.com> Tested-by: Impala Public Jenkins

            People

            • Assignee:
              tarasbob Taras Bobrovytsky
              Reporter:
              mikesbrown Michael Brown
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development