Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1124

DebugUtilTest.TestSignalStackTrace flaky due to likely timeout of stacktrace function

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Feature Complete
    • Fix Version/s: 1.1.0
    • Component/s: test, util
    • Labels:
      None

      Description

      DebugUtilTest.TestSignalStackTrace is flaky, likely because the stacktrace code can timeout and return an empty string.

      Failure log:

      [ RUN      ] DebugUtilTest.TestSignalStackTrace
      /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/debug-util-test.cc:91: Failure
      Failed
      Expected to find substring 'SleeperThread'. Got: ''
      ==================
      WARNING: ThreadSanitizer: data race (pid=20054)
        Write of size 8 at 0x7fff39ba2820 by main thread:
          #0 pthread_cond_destroy <null> (debug-util-test+0x0000000580ec)
          #1 kudu::ConditionVariable::~ConditionVariable() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/condition_variable.cc:46:12 (libkudu_util.so+0x000000148efe)
          #2 kudu::CountDownLatch::~CountDownLatch() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/countdown_latch.h:18:7 (debug-util-test+0x0000000adddb)
          #3 kudu::DebugUtilTest_TestSignalStackTrace_Test::TestBody() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/debug-util-test.cc:117:1 (debug-util-test+0x0000000ac2dc)
          #4 HandleSehExceptionsInMethodIfSupported<testing::Test, void> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2078 (libgmock.so+0x00000003ff2a)
          #5 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2114 (libgmock.so+0x00000003ff2a)
          #6 main /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/test_main.cc:38:13 (libkudu_test_main.so+0x0000000058a1)
      
        Previous read of size 8 at 0x7fff39ba2820 by thread T2:
          #0 pthread_cond_timedwait <null> (debug-util-test+0x000000055901)
          #1 kudu::ConditionVariable::TimedWait(kudu::MonoDelta const&) const /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/condition_variable.cc:107:12 (libkudu_util.so+0x000000149185)
          #2 kudu::CountDownLatch::WaitFor(kudu::MonoDelta const&) const /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/countdown_latch.h:79:12 (debug-util-test+0x0000000ae96b)
          #3 kudu::(anonymous namespace)::SleeperThread(kudu::CountDownLatch*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/debug-util-test.cc:51:11 (debug-util-test+0x0000000ac928)
          #4 void boost::_bi::list1<boost::_bi::value<kudu::CountDownLatch*> >::operator()<void (*)(kudu::CountDownLatch*), boost::_bi::list0>(boost::_bi::type<void>, void (*&)(kudu::CountDownLatch*), boost::_bi::list0&, int) /usr/include/boost/bind/bind.hpp:246:9 (debug-util-test+0x0000000b1480)
          #5 boost::_bi::bind_t<void, void (*)(kudu::CountDownLatch*), boost::_bi::list1<boost::_bi::value<kudu::CountDownLatch*> > >::operator()() /usr/include/boost/bind/bind_template.hpp:20:27 (debug-util-test+0x0000000b1403)
          #6 boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, void (*)(kudu::CountDownLatch*), boost::_bi::list1<boost::_bi::value<kudu::CountDownLatch*> > >, void>::invoke(boost::detail::function::function_buffer&) /usr/include/boost/function/function_template.hpp:153:11 (debug-util-test+0x0000000b11d9)
          #7 boost::function0<void>::operator()() const /usr/include/boost/function/function_template.hpp:1012:12 (libkudu_util.so+0x0000001ba4a1)
          #8 kudu::Thread::SuperviseThread(void*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:485:3 (libkudu_util.so+0x000000200759)
      
        Location is stack of main thread.
      
        Thread T2 'test thread-200' (tid=20065, running) created by main thread at:
          #0 pthread_create <null> (debug-util-test+0x000000055d23)
          #1 kudu::Thread::StartThread(std::string const&, std::string const&, boost::function<void ()()> const&, scoped_refptr<kudu::Thread>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.cc:413:13 (libkudu_util.so+0x000000200216)
          #2 kudu::Status kudu::Thread::Create<void (*)(kudu::CountDownLatch*), kudu::CountDownLatch*>(std::string const&, std::string const&, void (* const&)(kudu::CountDownLatch*), kudu::CountDownLatch* const&, scoped_refptr<kudu::Thread>*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/thread.h:126:12 (debug-util-test+0x0000000ad900)
          #3 kudu::DebugUtilTest_TestSignalStackTrace_Test::TestBody() /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/debug-util-test.cc:67:3 (debug-util-test+0x0000000ab4f4)
          #4 HandleSehExceptionsInMethodIfSupported<testing::Test, void> /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2078 (libgmock.so+0x00000003ff2a)
          #5 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/thirdparty/gmock-1.7.0/gtest/src/gtest.cc:2114 (libgmock.so+0x00000003ff2a)
          #6 main /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/src/kudu/util/test_main.cc:38:13 (libkudu_test_main.so+0x0000000058a1)
      
      SUMMARY: ThreadSanitizer: data race (/data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/TSAN/label/kudu-gerrit-slaves/build/fastdebug/debug-util-test+0x580ec) in pthread_cond_destroy
      ==================
      WARNING: Logging before InitGoogleLogging() is written to STDERR
      F0908 16:05:43.714090 20054 condition_variable.cc:47] Check failed: 0 == rv (0 vs. 16)
      *** Check failure stack trace: ***
      *** Aborted at 1441753543 (unix time) try "date -d @1441753543" if you are using GNU date ***
      PC: @     0x7ff30a5cf8e5 (unknown)
      *** SIGABRT (@0xfa000004e56) received by PID 20054 (TID 0x7ff30e36f700) from PID 20054; stack trace: ***
          @     0x7ff30e3bd9b9 __tsan::CallUserSignalHandler() at ??:0
          @     0x7ff30e3be64d rtl_sigaction() at ??:0
          @     0x7ff30ab55500 (unknown) at ??:0
          @     0x7ff30a5cf8e5 (unknown) at ??:0
          @     0x7ff30a5d10c5 (unknown) at ??:0
          @     0x7ff30e3e6a78 __tsan::cur_thread_placeholder at ??:0
          @     0x7ff30b962cc9 google::logging_fail() at ??:0
          @     0x7ff30b962bcd google::LogMessage::Fail() at ??:0
          @     0x7ff30b9670ad google::LogMessage::SendToLog() at ??:0
          @     0x7ff30b964f1b google::LogMessage::Flush() at ??:0
          @     0x7ff30b965241 google::LogMessageFatal::~LogMessageFatal() at ??:0
          @     0x7ff30d5aaf71 kudu::ConditionVariable::~ConditionVariable() at ??:0
          @     0x7ff30e43cddc kudu::CountDownLatch::~CountDownLatch() at ??:0
          @     0x7ff30e43b2dd kudu::DebugUtilTest_TestSignalStackTrace_Test::TestBody() at ??:0
          @     0x7ff30b530f2b testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
          @     0x7ff30b522527 testing::Test::Run() at ??:0
          @     0x7ff30b5225fe testing::TestInfo::Run() at ??:0
          @     0x7ff30b522737 testing::TestCase::Run() at ??:0
          @     0x7ff30b526a37 testing::internal::UnitTestImpl::RunAllTests() at ??:0
          @     0x7ff30b530afb testing::internal::HandleExceptionsInMethodIfSupported<>() at ??:0
          @     0x7ff30b521c10 testing::UnitTest::Run() at ??:0
          @     0x7ff30df69f4c RUN_ALL_TESTS() at ??:0
          @     0x7ff30df698a2 main at ??:0
          @     0x7ff30a5bbcdd (unknown) at ??:0
          @     0x7ff30e3b8961 __tsan::cur_thread_placeholder at ??:0
      

        Attachments

          Activity

            People

            • Assignee:
              tlipcon Todd Lipcon
              Reporter:
              mpercy Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: