Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6099

DCHECK in runtime filters: "Tried to increment unknown counters group"

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 2.11.0
    • Impala 2.11.0
    • Backend

    Description

      F1021 06:50:31.885411 20811 filter-context.cc:49] Check failed: it != counters.end() Tried to increment unknown counter group
*** Check failure stack trace: ***
F1021 06:50:31.885462 20822 filter-context.cc:49] Check failed: it != counters.end() Tried to increment unknown counter group
*** Check failure stack trace: ***
    @          0x2ec387d  google::LogMessage::Fail()
    @          0x2ec387d  google::LogMessage::Fail()
    @          0x2ec5122  google::LogMessage::SendToLog()
    @          0x2ec5122  google::LogMessage::SendToLog()
    @          0x2ec3257  google::LogMessage::Flush()
    @          0x2ec3257  google::LogMessage::Flush()
    @          0x2ec681e  google::LogMessageFatal::~LogMessageFatal()
    @          0x1c4b48d  impala::FilterStats::IncrCounters()
    @          0x2ec681e  google::LogMessageFatal::~LogMessageFatal()
    @          0x1c4b48d  impala::FilterStats::IncrCounters()
    @          0x1c4d998  impala::FilterContext::CheckForAlwaysFalse()
    @          0x1a4cc51  impala::HdfsScanNodeBase::PartitionPassesFilters()
    @          0x1c4d998  impala::FilterContext::CheckForAlwaysFalse()
    @          0x1ab2f20  impala::HdfsParquetScanner::GetNextInternal()
    @          0x1a4cc51  impala::HdfsScanNodeBase::PartitionPassesFilters()
    @          0x1ab1224  impala::HdfsParquetScanner::ProcessSplit()
    @          0x1ab2f20  impala::HdfsParquetScanner::GetNextInternal()
    @          0x1a41594  impala::HdfsScanNode::ProcessSplit()
    @          0x1ab1224  impala::HdfsParquetScanner::ProcessSplit()
    @          0x1a40a0a  impala::HdfsScanNode::ScannerThread()
    @          0x1a3fea7  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_17ThreadResourceMgr12ResourcePoolEENKUlvE_clEv
    @          0x1a41594  impala::HdfsScanNode::ProcessSplit()
    @          0x1a41dd5  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_17ThreadResourceMgr12ResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
    @          0x1a40a0a  impala::HdfsScanNode::ScannerThread()
    @          0x16c3c6a  boost::function0<>::operator()()
    @          0x1a3fea7  _ZZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS_17ThreadResourceMgr12ResourcePoolEENKUlvE_clEv
    @          0x1a41dd5  _ZN5boost6detail8function26void_function_obj_invoker0IZN6impala12HdfsScanNode22ThreadTokenAvailableCbEPNS3_17ThreadResourceMgr12ResourcePoolEEUlvE_vE6invokeERNS1_15function_bufferE
    @          0x199afe3  impala::Thread::SuperviseThread()
    @          0x16c3c6a  boost::function0<>::operator()()
    @          0x19a3b76  boost::_bi::list4<>::operator()<>()
    @          0x19a3ab9  boost::_bi::bind_t<>::operator()()
    @          0x199afe3  impala::Thread::SuperviseThread()
    @          0x19a3a7c  boost::detail::thread_data<>::run()
    @          0x204e3aa  thread_proxy
    @       0x38a7407851  (unknown)

      	info "(gdb)Auto-loading safe path"
#0  0x00000038a70328e5 in raise () from /lib64/libc.so.6
#1  0x00000038a70340c5 in abort () from /lib64/libc.so.6
#2  0x0000000002eccdd4 in google::DumpStackTraceAndExit() ()
#3  0x0000000002ec387d in google::LogMessage::Fail() ()
#4  0x0000000002ec5122 in google::LogMessage::SendToLog() ()
#5  0x0000000002ec3257 in google::LogMessage::Flush() ()
#6  0x0000000002ec681e in google::LogMessageFatal::~LogMessageFatal() ()
#7  0x0000000001c4b48d in impala::FilterStats::IncrCounters (this=0x266f9080, key=..., total=1, processed=1, rejected=1) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/filter-context.cc:49
#8  0x0000000001c4d998 in impala::FilterContext::CheckForAlwaysFalse (stats_name=..., ctxs=...) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/filter-context.cc:382
#9  0x0000000001a4cc51 in impala::HdfsScanNodeBase::PartitionPassesFilters (this=0xf4e9800, partition_id=5285, stats_name=..., filter_ctxs=...) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:649
#10 0x0000000001ab2f20 in impala::HdfsParquetScanner::GetNextInternal (this=0x1963ae00, row_batch=0x3197cb50) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:502
#11 0x0000000001ab1224 in impala::HdfsParquetScanner::ProcessSplit (this=0x1963ae00) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-parquet-scanner.cc:405
#12 0x0000000001a41594 in impala::HdfsScanNode::ProcessSplit (this=0xf4e9800, filter_ctxs=..., expr_results_pool=0x7f2dd25ee6e0, scan_range=0xaa48700) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node.cc:533
#13 0x0000000001a40a0a in impala::HdfsScanNode::ScannerThread (this=0xf4e9800) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node.cc:442
#14 0x0000000001a3fea7 in impala::HdfsScanNode::<lambda()>::operator()(void) const (__closure=0x7f2dd25eebc8) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/exec/hdfs-scan-node.cc:356
#15 0x0000000001a41dd5 in boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourceMgr::ResourcePool*)::<lambda()>, void>::invoke(boost::detail::function::function_buffer &) (function_obj_ptr=...) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:153
#16 0x00000000016c3c6a in boost::function0<void>::operator() (this=0x7f2dd25eebc0) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
#17 0x000000000199afe3 in impala::Thread::SuperviseThread (name=..., category=..., functor=..., thread_started=0x7f2dec7f3eb0) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:352
#18 0x00000000019a3b76 in boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> >::operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, boost::function<void()>, impala::Promise<long> *), boost::_bi::list0 &, int) (this=0x2faf7bc0, f=@0x2faf7bb8, a=...) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
#19 0x00000000019a3ab9 in boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > >::operator()(void) (this=0x2faf7bb8) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
#20 0x00000000019a3a7c in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > > >::run(void) (this=0x2faf7a00) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
#21 0x000000000204e3aa in thread_proxy ()
#22 0x00000038a7407851 in start_thread () from /lib64/libpthread.so.0
#23 0x00000038a70e894d in clone () from /lib64/libc.so.6

      Tests running were:

      06:50:30 [gw2] PASSED query_test/test_scanners.py::TestTextSplitDelimiters::test_text_split_across_buffers_delimiter[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
06:50:30 query_test/test_scanners.py::TestScanTruncatedFiles::test_scan_truncated_file_empty[vector0] 
06:50:30 [gw2] SKIPPED query_test/test_scanners.py::TestScanTruncatedFiles::test_scan_truncated_file_empty[vector0] 
06:50:30 query_test/test_scanners.py::TestScanTruncatedFiles::test_scan_truncated_file[vector0] 
06:50:30 [gw2] SKIPPED query_test/test_scanners.py::TestScanTruncatedFiles::test_scan_truncated_file[vector0] 
06:50:41 query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option: {'mem_limit': '512m', 'abort_on_error': False, 'num_nodes': 0} | table_format: text/none] 
06:50:41 [gw3] FAILED query_test/test_join_queries.py::TestJoinQueries::test_empty_build_joins[batch_size: 0 | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:41 query_test/test_join_queries.py::TestTPCHJoinQueries::test_outer_joins[batch_size: 0 | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:41 [gw1] FAILED query_test/test_sort.py::TestQueryFullSort::test_multiple_mem_limits[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:41 query_test/test_sort.py::TestQueryFullSort::test_multiple_mem_limits_full_output[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:41 [gw0] FAILED query_test/test_runtime_filters.py::TestRuntimeFilters::test_basic_filters[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: avro/snap/block] 
06:50:42 query_test/test_runtime_filters.py::TestRuntimeFilters::test_wait_time[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
06:50:42 [gw2] FAILED query_test/test_scanners_fuzz.py::TestScannersFuzzing::test_fuzz_alltypes[exec_option: {'mem_limit': '512m', 'abort_on_error': False, 'num_nodes': 0} | table_format: text/none] 
06:50:42 [gw3] ERROR query_test/test_join_queries.py::TestTPCHJoinQueries::test_outer_joins[batch_size: 0 | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:42 query_test/test_join_queries.py::TestSemiJoinQueries::test_semi_joins[batch_size: 0 | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:42 [gw1] FAILED query_test/test_sort.py::TestQueryFullSort::test_multiple_mem_limits_full_output[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:42 query_test/test_sort.py::TestQueryFullSort::test_sort_join[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:42 [gw0] FAILED query_test/test_runtime_filters.py::TestRuntimeFilters::test_wait_time[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] 
06:50:42 query_test/test_runtime_filters.py::TestRuntimeFilters::test_wait_time[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: seq/snap/block] 
06:50:42 [gw3] ERROR query_test/test_join_queries.py::TestSemiJoinQueries::test_semi_joins[batch_size: 0 | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:42 query_test/test_join_queries.py::TestSemiJoinQueries::test_semi_joins_exhaustive[batch_size: 0 | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:42 [gw1] FAILED query_test/test_sort.py::TestQueryFullSort::test_sort_join[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:42 query_test/test_sort.py::TestQueryFullSort::test_sort_union[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: parquet/none] 
06:50:44 [gw0] FAILED query_test/test_runtime_filters.py::TestRuntimeFilters::test_wait_time[exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: seq/snap/block] 
06:50:44

      Impala hash is 78776e9b5f03d122dbc6ddda81aec6e292f75ab4

      Attachments

        Issue Links

          is broken by

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5789 Prune all partitions if an always-false runtime filter is received

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. IMPALA-5789 Prune all partitions if an always-false runtime filter is received

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              tianyiwang Tianyi Wang
              tarmstrong Tim Armstrong
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: