Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4369

DCHECK in Parquet scanner with MT_DOP > 1 when reading file with bad metadata.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.8.0
    • Component/s: Backend
    • Labels:

      Description

      Repro:

      1. Start impala cluster with MT_DOP > 0
      2. ./run-tests.py query_test/test_scanners.py -k test_parquet --table_formats=parquet/none
      

      DCHECK:

      F1025 17:31:47.367564 29896 hdfs-scanner.h:155] Check failed: scan_node_->HasRowBatchQueue() 
      

      Stack:

      #7  0x000000000165d3ac in impala::HdfsScanner::batch (this=0xd1b5180) at /home/abehm/impala/be/src/exec/hdfs-scanner.h:155
      #8  0x0000000001669a21 in impala::HdfsScanNodeBase::CreateAndOpenScanner (this=0xc0e9200, partition=0xf5f2600, context=0x8122870, scanner=0xc0e96c0)
          at /home/abehm/impala/be/src/exec/hdfs-scan-node-base.cc:638
      #9  0x000000000167f495 in impala::HdfsScanNodeMt::GetNext (this=0xc0e9200, state=0xe546400, row_batch=0x10bcd960, eos=0x7f7df7726e7f)
          at /home/abehm/impala/be/src/exec/hdfs-scan-node-mt.cc:90
      #10 0x00000000019b1780 in impala::PlanFragmentExecutor::ExecInternal (this=0xc9cca88) at /home/abehm/impala/be/src/runtime/plan-fragment-executor.cc:351
      #11 0x00000000019b145f in impala::PlanFragmentExecutor::Exec (this=0xc9cca88) at /home/abehm/impala/be/src/runtime/plan-fragment-executor.cc:327
      #12 0x000000000150f99a in impala::FragmentMgr::FragmentExecState::Exec (this=0xc9cc700) at /home/abehm/impala/be/src/service/fragment-exec-state.cc:59
      #13 0x00000000015070e8 in impala::FragmentMgr::FragmentThread (this=0x83b2cc0, fragment_instance_id=...) at /home/abehm/impala/be/src/service/fragment-mgr.cc:86
      #14 0x000000000150ae6a in boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>::operator() (this=0xe0b41e0, p=0x83b2cc0, a1=...)
          at /home/abehm/impala/toolchain/boost-1.57.0/include/boost/bind/mem_fn_template.hpp:165
      #15 0x000000000150ac27 in boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> >::operator()<boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list0> (this=0xe0b41f0, f=..., a=...)
          at /home/abehm/impala/toolchain/boost-1.57.0/include/boost/bind/bind.hpp:313
      #16 0x000000000150a551 in boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> > >::operator() (this=0xe0b41e0)
          at /home/abehm/impala/toolchain/boost-1.57.0/include/boost/bind/bind_template.hpp:20
      #17 0x0000000001509ee4 in boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::FragmentMgr, impala::TUniqueId>, boost::_bi::list2<boost::_bi::value<impala::FragmentMgr*>, boost::_bi::value<impala::TUniqueId> > >, void>::invoke (function_obj_ptr=...)
          at /home/abehm/impala/toolchain/boost-1.57.0/include/boost/function/function_template.hpp:153
      #18 0x000000000130ca1a in boost::function0<void>::operator() (this=0x7f7df7727d30)
          at /home/abehm/impala/toolchain/boost-1.57.0/include/boost/function/function_template.hpp:767
      

        Activity

        Hide
        alex.behm Alexander Behm added a comment -

        commit f7d71950e3e2ebda07f90c48d6b93b1335eaa25e
        Author: Alex Behm <alex.behm@cloudera.com>
        Date: Tue Oct 25 17:53:59 2016 -0700

        IMPALA-4369: Avoid DCHECK in Parquet scanner with MT_DOP > 0.

        When HdfsParquetScanner::Open() failed we used to hit a DCHECK
        when trying to access HdfsParquetScanner::batch() which is
        only valid to call for non-MT scan nodes.

        Change-Id: Ifbfdde505dbbd2742e7ab79a2415ff317a9bfa2f
        Reviewed-on: http://gerrit.cloudera.org:8080/4851
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Internal Jenkins

        Show
        alex.behm Alexander Behm added a comment - commit f7d71950e3e2ebda07f90c48d6b93b1335eaa25e Author: Alex Behm <alex.behm@cloudera.com> Date: Tue Oct 25 17:53:59 2016 -0700 IMPALA-4369 : Avoid DCHECK in Parquet scanner with MT_DOP > 0. When HdfsParquetScanner::Open() failed we used to hit a DCHECK when trying to access HdfsParquetScanner::batch() which is only valid to call for non-MT scan nodes. Change-Id: Ifbfdde505dbbd2742e7ab79a2415ff317a9bfa2f Reviewed-on: http://gerrit.cloudera.org:8080/4851 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Internal Jenkins

          People

          • Assignee:
            alex.behm Alexander Behm
            Reporter:
            alex.behm Alexander Behm
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development