Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8319

ThreadPoolTest.SynchronousThreadPoolTest failure + SIGSEGV on Centos 6

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: Impala 3.2.0
    • Fix Version/s: Impala 3.2.0
    • Component/s: Backend
    • Labels:
    • Epic Color:
      ghx-label-2

      Description

      One run of Centos 6 tests saw this failure for ThreadPoolTest.SynchronousPoolTest:

      [==========] Running 2 tests from 1 test case.
      [----------] Global test environment set-up.
      [----------] 2 tests from ThreadPoolTest
      [ RUN ] ThreadPoolTest.BasicTest
      19/03/17 16:26:21 INFO util.JvmPauseMonitor: Starting JVM pause monitor
      [ OK ] ThreadPoolTest.BasicTest (38 ms)
      [ RUN ] ThreadPoolTest.SynchronousThreadPoolTest
      /data/jenkins/workspace/impala-cdh6.x-exhaustive-centos6/repos/Impala/be/src/util/thread-pool-test.cc:132: Failure
      Value of: TErrorCode::THREAD_POOL_TASK_TIMED_OUT
      Actual: 124
      Expected: queued_task_status.code()
      Which is: 123
      [ FAILED ] ThreadPoolTest.SynchronousThreadPoolTest (109 ms)
      [----------] 2 tests from ThreadPoolTest (148 ms total)

      This also produced a minidump indicating a SIGSEGV. It has the following stack:

      C [unifiedbetests+0x4846c93] tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned long, int)+0x133
      C [unifiedbetests+0x4846d2c] tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned long)+0x1c
      C [unifiedbetests+0x499e5a0] operator delete[](void*)+0x3c0
      C [unifiedbetests+0x19b7232] __gnu_cxx::new_allocator<std::string>::deallocate(std::string*, unsigned long)+0x20
      C [unifiedbetests+0x19a9bb3] std::allocator_traits<std::allocator<std::string> >::deallocate(std::allocator<std::string>&, std::string*, unsigned long)+0x2b
      C [unifiedbetests+0x199d5ce] std::_Vector_base<std::string, std::allocator<std::string> >::_M_deallocate(std::string*, unsigned long)+0x32
      C [unifiedbetests+0x1996037] std::_Vector_base<std::string, std::allocator<std::string> >::~_Vector_base()+0x41
      C [unifiedbetests+0x1990791] std::vector<std::string, std::allocator<std::string> >::~vector()+0x41
      C [unifiedbetests+0x28264de] impala::TMetricDef::~TMetricDef()+0x4e
      C [unifiedbetests+0x2549370] std::pair<std::string const, impala::TMetricDef>::~pair()+0x1c
      C [unifiedbetests+0x254939a] void __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<std::string const, impala::TMetricDef> > >::destroy<std::pair<std::string const, impala::TMetricDef> >(std::pair<std::string const, impala::TMetricDef>*)+0x1c
      C [unifiedbetests+0x25474a3] std::enable_if<std::__and_<std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::string const, impala::TMetricDef> > > >::__destroy_helper<std::pair<std::string const, impala::TMetricDef> >::type>::value, void>::type std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::string const, impala::TMetricDef> > > >::_S_destroy<std::pair<std::string const, impala::TMetricDef> >(std::allocator<std::_Rb_tree_node<std::pair<std::string const, impala::TMetricDef> > >&, std::pair<std::string const, impala::TMetricDef>*)+0x23
      C [unifiedbetests+0x2544d87] void std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<std::string const, impala::TMetricDef> > > >::destroy<std::pair<std::string const, impala::TMetricDef> >(std::allocator<std::_Rb_tree_node<std::pair<std::string const, impala::TMetricDef> > >&, std::pair<std::string const, impala::TMetricDef>*)+0x23
      C [unifiedbetests+0x2540841] std::_Rb_tree<std::string, std::pair<std::string const, impala::TMetricDef>, std::_Select1st<std::pair<std::string const, impala::TMetricDef> >, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TMetricDef> > >::_M_destroy_node(std::_Rb_tree_node<std::pair<std::string const, impala::TMetricDef> >*)+0x37
      C [unifiedbetests+0x253ab1f] std::_Rb_tree<std::string, std::pair<std::string const, impala::TMetricDef>, std::_Select1st<std::pair<std::string const, impala::TMetricDef> >, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TMetricDef> > >::_M_erase(std::_Rb_tree_node<std::pair<std::string const, impala::TMetricDef> >*)+0x53
      C [unifiedbetests+0x253aafc] std::_Rb_tree<std::string, std::pair<std::string const, impala::TMetricDef>, std::_Select1st<std::pair<std::string const, impala::TMetricDef> >, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TMetricDef> > >::_M_erase(std::_Rb_tree_node<std::pair<std::string const, impala::TMetricDef> >*)+0x30
      C [unifiedbetests+0x253aafc] std::_Rb_tree<std::string, std::pair<std::string const, impala::TMetricDef>, std::_Select1st<std::pair<std::string const, impala::TMetricDef> >, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TMetricDef> > >::_M_erase(std::_Rb_tree_node<std::pair<std::string const, impala::TMetricDef> >*)+0x30
      C [unifiedbetests+0x2825e56] std::_Rb_tree<std::string, std::pair<std::string const, impala::TMetricDef>, std::_Select1st<std::pair<std::string const, impala::TMetricDef> >, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TMetricDef> > >::~_Rb_tree()+0x2a
      C [unifiedbetests+0x2824962] std::map<std::string, impala::TMetricDef, std::less<std::string>, std::allocator<std::pair<std::string const, impala::TMetricDef> > >::~map()+0x18
      C [unifiedbetests+0x282648e] impala::MetricDefsConstants::~MetricDefsConstants()+0x18

      This stack doesn't make much sense, because thread-pool-test doesn't use MetricDefs. When I tested locally on Ubuntu 16.04, MetricDefs is not instantiated. This is using the unified backend executable.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                joemcdonnell Joe McDonnell
                Reporter:
                joemcdonnell Joe McDonnell
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: