Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5756

Impala crashes on startup in impala::SpinLock::lock()

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 2.10.0
    • Impala 2.10.0
    • Backend
    • ghx-label-1

    Description

      There were several build failures with the following backtrace:

      #0  0x000000348ea328e5 in raise () from /lib64/libc.so.6
      #1  0x000000348ea340c5 in abort () from /lib64/libc.so.6
      #2  0x00007f9235dd5c55 in os::abort(bool) () from /opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #3  0x00007f9235f57cd7 in VMError::report_and_die() () from /opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #4  0x00007f9235ddab6f in JVM_handle_linux_signal () from /opt/toolchain/sun-jdk-64bit-1.7.0.75/jre/lib/amd64/server/libjvm.so
      #5  <signal handler called>
      #6  0x00000000011e100c in base::subtle::NoBarrier_CompareAndSwap (ptr=0x1c, old_value=0, new_value=1) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/gutil/atomicops-internals-x86.h:85
      #7  0x00000000011e108c in base::subtle::Acquire_CompareAndSwap (ptr=0x1c, old_value=0, new_value=1) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/gutil/atomicops-internals-x86.h:138
      #8  0x00000000011e10d2 in base::SpinLock::Lock (this=0x1c) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/gutil/spinlock.h:74
      #9  0x00000000011e115c in impala::SpinLock::lock (this=0x1c) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/spinlock.h:34
      #10 0x00000000011e22ea in boost::lock_guard<impala::SpinLock>::lock_guard (this=0x7f92331dd9b0, m_=...) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/lock_guard.hpp:38
      #11 0x00000000015e6f64 in impala::SimpleMetric<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, (impala::TMetricKind::type)2>::set_value (this=0x0, value=...) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/metrics.h:153
      #12 0x000000000163b50b in impala::AggregateMemoryMetrics::Refresh () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/memory-metrics.cc:147
      #13 0x0000000001232629 in MemoryMaintenanceThread () at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/common/init.cc:151
      #14 0x0000000001234f29 in boost::detail::function::void_function_invoker0<void (*)(), void>::invoke (function_ptr=...) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:112
      #15 0x00000000013b72ca in boost::function0<void>::operator() (this=0x7f92331ddc20) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/function/function_template.hpp:767
      #16 0x000000000168ed67 in impala::Thread::SuperviseThread (name=..., category=..., functor=..., thread_started=0x7fff0445e450) at /data/jenkins/workspace/impala-umbrella-build-and-test/repos/Impala/be/src/util/thread.cc:330
      #17 0x000000000169786e in boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> >::operator()<void (*)(const std::basic_string<char>&, const std::basic_string<char>&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, const std::basic_string<char, std::char_traits<char>, std::allocator<char> > &, boost::function<void()>, impala::Promise<long> *), boost::_bi::list0 &, int) (this=0x62325c0, f=@0x62325b8, a=...) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind.hpp:457
      #18 0x00000000016977b1 in boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > >::operator()(void) (this=0x62325b8) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/bind/bind_template.hpp:20
      #19 0x0000000001697774 in boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&, boost::function<void()>, impala::Promise<long int>*), boost::_bi::list4<boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<std::basic_string<char, std::char_traits<char>, std::allocator<char> > >, boost::_bi::value<boost::function<void()> >, boost::_bi::value<impala::Promise<long int>*> > > >::run(void) (this=0x6232400) at /data/jenkins/workspace/impala-umbrella-build-and-test/Impala-Toolchain/boost-1.57.0-p3/include/boost/thread/detail/thread.hpp:116
      #20 0x0000000001b6a21a in thread_proxy ()
      #21 0x000000348ee07851 in start_thread () from /lib64/libpthread.so.0
      #22 0x000000348eae894d in clone () from /lib64/libc.so.6
      

      Attachments

        Issue Links

          Activity

            I'm not sure what is causing it yet, but maybe it has something to do with your recent change, tarmstrong, so assigning it to you. Feel free to reassign.

            tarasbob Taras Bobrovytsky added a comment - I'm not sure what is causing it yet, but maybe it has something to do with your recent change, tarmstrong , so assigning it to you. Feel free to reassign.
            tarmstrong Tim Armstrong added a comment -

            Yeah it's a race during startup caused by that patch. It might be best just to revert it then figure out the best way to avoid it: https://gerrit.cloudera.org/#/c/7571/

            tarmstrong Tim Armstrong added a comment - Yeah it's a race during startup caused by that patch. It might be best just to revert it then figure out the best way to avoid it: https://gerrit.cloudera.org/#/c/7571/
            tarmstrong Tim Armstrong added a comment -

            IMPALA-5756: start memory maintenance thread after metric creation

            Daemons were sometimes crashing on startup if the memory maintenance
            thread started running before the memory metrics were created.

            This changes the startup sequence so that the thread is started
            after the metrics are registered.

            Change-Id: Ibb7337a68547b4d660f3100b04df2c20befb044c
            Reviewed-on: http://gerrit.cloudera.org:8080/7573
            Reviewed-by: Henry Robinson <henry@cloudera.com>
            Tested-by: Impala Public Jenkins

            M be/src/catalog/catalogd-main.cc
            M be/src/common/init.cc
            M be/src/common/init.h
            M be/src/service/impalad-main.cc
            M be/src/statestore/statestored-main.cc
            5 files changed, 13 insertions, 8 deletions

            Approvals:
            Impala Public Jenkins: Verified
            Henry Robinson: Looks good to me, approved

            tarmstrong Tim Armstrong added a comment - IMPALA-5756 : start memory maintenance thread after metric creation Daemons were sometimes crashing on startup if the memory maintenance thread started running before the memory metrics were created. This changes the startup sequence so that the thread is started after the metrics are registered. Change-Id: Ibb7337a68547b4d660f3100b04df2c20befb044c Reviewed-on: http://gerrit.cloudera.org:8080/7573 Reviewed-by: Henry Robinson <henry@cloudera.com> Tested-by: Impala Public Jenkins — M be/src/catalog/catalogd-main.cc M be/src/common/init.cc M be/src/common/init.h M be/src/service/impalad-main.cc M be/src/statestore/statestored-main.cc 5 files changed, 13 insertions , 8 deletions Approvals: Impala Public Jenkins: Verified Henry Robinson: Looks good to me, approved

            People

              tarmstrong Tim Armstrong
              tarasbob Taras Bobrovytsky
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: