Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3502

impala-cdh5-trunk-exhaustive-rhel7 test failures and crashes

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.5.0
    • Fix Version/s: Impala 2.6.0
    • Component/s: Backend
    • Labels:

      Description

      http://sandbox.jenkins.cloudera.com/job/impala-cdh5-trunk-exhaustive-rhel7/312

      There are a few different failures/crashes happening in this build, and I can't tell if they're related or not. Henry Robinson I've assigned to you because of the runtime filters DCHECK, please file other JIRAs if you think some of these problems are unrelated.

      Failure 1:
      metadata.test_compute_stats.TestComputeStats.test_compute_stats
      metadata.test_compute_stats.TestCorruptTableStats.test_compute_stats (I think py.test may have screwed up and mislabeled this one, it looks like it's another test_compute_stats vector)

      http://sandbox.jenkins.cloudera.com/job/impala-cdh5-trunk-exhaustive-rhel7/312/testReport/junit/metadata.test_compute_stats/TestComputeStats/test_compute_stats_exec_option____disable_codegen___False___abort_on_error___1___exec_single_node_rows_threshold___0___batch_size___0___num_nodes___0____table_format__text_none_/

      Stacktrace
      
      metadata/test_compute_stats.py:41: in test_compute_stats
          self.run_test_case('QueryTest/compute-stats', vector, unique_database)
      common/impala_test_suite.py:311: in run_test_case
          pytest.config.option.update_results)
      common/test_result_verifier.py:357: in verify_raw_results
          VERIFIER_MAP[verifier](expected, actual)
      common/test_result_verifier.py:203: in verify_query_result_is_equal
          assert expected_results == actual_results
      E   assert Comparing QueryTestResults (expected vs actual):
      E     100,3,'7.73KB','NOT CACHED','NOT CACHED','TEXT','false',regex:.* != 100,2,'7.73KB','NOT CACHED','NOT CACHED','TEXT','false','hdfs://localhost:20500/test-warehouse/test_compute_stats_650013a0.db/alltypesnopart'
      

      Why are there only two files instead of one? Maybe because an impalad crashed? (see below failures)

      Failure 2:
      impalad with process ID 20640 hit DCHECK:

      F0506 15:17:08.873009  3054 coordinator.cc:1988] Check failed: filter_routing_table_complete_ Filter received before routing table complete
      

      Failure 3:
      impalad with process ID 20921 crashed with following stack:

      ...
      #5  <signal handler called>
      #6  0x000000000125306a in std::_Rb_tree<std::string, std::pair<std::string const, impala::RuntimeProfile::Counter*>, std::_Select1st<std::pair<std::string const, impala::RuntimeProfile::Counter*> >, std::less<std::string>, std::allocator<std::pair<std::string const, impala::RuntimeProfile::Counter*> > >::_M_begin (this=0x306b6e7568635f9b) at /tmp/impala-deps/gcc-4.9.2/include/c++/4.9.2/bits/stl_tree.h:518
      #7  0x0000000001251a66 in std::_Rb_tree<std::string, std::pair<std::string const, impala::RuntimeProfile::Counter*>, std::_Select1st<std::pair<std::string const, impala::RuntimeProfile::Counter*> >, std::less<std::string>, std::allocator<std::pair<std::string const, impala::RuntimeProfile::Counter*> > >::lower_bound (this=0x306b6e7568635f9b, __k=...) at /tmp/impala-deps/gcc-4.9.2/include/c++/4.9.2/bits/stl_tree.h:927
      #8  0x0000000001250473 in std::map<std::string, impala::RuntimeProfile::Counter*, std::less<std::string>, std::allocator<std::pair<std::string const, impala::RuntimeProfile::Counter*> > >::lower_bound (this=0x306b6e7568635f9b, __x=...) at /tmp/impala-deps/gcc-4.9.2/include/c++/4.9.2/bits/stl_map.h:902
      #9  0x000000000124f204 in std::map<std::string, impala::RuntimeProfile::Counter*, std::less<std::string>, std::allocator<std::pair<std::string const, impala::RuntimeProfile::Counter*> > >::operator[] (this=0x306b6e7568635f9b, __k=...) at /tmp/impala-deps/gcc-4.9.2/include/c++/4.9.2/bits/stl_map.h:496
      #10 0x000000000124d884 in impala::RuntimeProfile::total_time_counter (this=0x306b6e7568635f73) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/util/runtime-profile.h:523
      #11 0x00000000018083f5 in impala::InstanceComparator::operator() (this=0x7fc498105f90, a=..., b=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/runtime/coordinator.cc:1641
      #12 0x000000000181b76c in __gnu_cxx::__ops::_Val_comp_iter<impala::InstanceComparator>::operator()<std::pair<impala::RuntimeProfile*, bool>, __gnu_cxx::__normal_iterator<std::pair<impala::RuntimeProfile*, bool>*, std::vector<std::pair<impala::RuntimeProfile*, bool>, std::allocator<std::pair<impala::RuntimeProfile*, bool> > > > > (this=0x7fc498105f90, __val=..., __it=...) at /tmp/impala-deps/gcc-4.9.2/include/c++/4.9.2/bits/predefined_ops.h:166
      #13 0x000000000181a28c in std::__unguarded_linear_insert<__gnu_cxx::__normal_iterator<std::pair<impala::RuntimeProfile*, bool>*, std::vector<std::pair<impala::RuntimeProfile*, bool>, std::allocator<std::pair<impala::RuntimeProfile*, bool> > > >, __gnu_cxx::__ops::_Val_comp_iter<impala::InstanceComparator> > (__last=..., __comp=...) at /tmp/impala-deps/gcc-4.9.2/include/c++/4.9.2/bits/stl_algo.h:1827
      #14 0x0000000001817bd9 in std::__insertion_sort<__gnu_cxx::__normal_iterator<std::pair<impala::RuntimeProfile*, bool>*, std::vector<std::pair<impala::RuntimeProfile*, bool>, std::allocator<std::pair<impala::RuntimeProfile*, bool> > > >, __gnu_cxx::__ops::_Iter_comp_iter<impala::InstanceComparator> > (__first=..., __last=..., __comp=...) at /tmp/impala-deps/gcc-4.9.2/include/c++/4.9.2/bits/stl_algo.h:1854
      #15 0x00000000018149e7 in std::__final_insertion_sort<__gnu_cxx::__normal_iterator<std::pair<impala::RuntimeProfile*, bool>*, std::vector<std::pair<impala::RuntimeProfile*, bool>, std::allocator<std::pair<impala::RuntimeProfile*, bool> > > >, __gnu_cxx::__ops::_Iter_comp_iter<impala::InstanceComparator> > (__first=..., __last=..., __comp=...) at /tmp/impala-deps/gcc-4.9.2/include/c++/4.9.2/bits/stl_algo.h:1889
      #16 0x000000000181111f in std::__sort<__gnu_cxx::__normal_iterator<std::pair<impala::RuntimeProfile*, bool>*, std::vector<std::pair<impala::RuntimeProfile*, bool>, std::allocator<std::pair<impala::RuntimeProfile*, bool> > > >, __gnu_cxx::__ops::_Iter_comp_iter<impala::InstanceComparator> > (__first=..., __last=..., __comp=...) at /tmp/impala-deps/gcc-4.9.2/include/c++/4.9.2/bits/stl_algo.h:1970
      #17 0x000000000180ddf9 in std::sort<__gnu_cxx::__normal_iterator<std::pair<impala::RuntimeProfile*, bool>*, std::vector<std::pair<impala::RuntimeProfile*, bool>, std::allocator<std::pair<impala::RuntimeProfile*, bool> > > >, impala::InstanceComparator> (__first=..., __last=..., __comp=...) at /tmp/impala-deps/gcc-4.9.2/include/c++/4.9.2/bits/stl_algo.h:4716
      #18 0x000000000180ab85 in impala::RuntimeProfile::SortChildren<impala::InstanceComparator> (this=0x9b75b00, cmp=...) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/util/runtime-profile.h:445
      #19 0x00000000018043a7 in impala::Coordinator::ReportQuerySummary (this=0xc015500) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/runtime/coordinator.cc:1740
      #20 0x00000000017fd6c3 in impala::Coordinator::Wait (this=0xc015500) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/runtime/coordinator.cc:1050
      #21 0x00000000013ca7a1 in impala::ImpalaServer::QueryExecState::WaitInternal (this=0xe0c6000) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/service/query-exec-state.cc:618
      #22 0x00000000013ca474 in impala::ImpalaServer::QueryExecState::Wait (this=0xe0c6000) at /data/jenkins/workspace/impala-cdh5-trunk-exhaustive-rhel7/repos/Impala/be/src/service/query-exec-state.cc:594
      ...
      

        Attachments

          Activity

            People

            • Assignee:
              dtsirogiannis Dimitris Tsirogiannis
              Reporter:
              skye Skye Wanderman-Milne
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: