Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Running the newly added tests scenario TestScannerKeepAlivePeriodicallyCrossServers fails with SIGABRT when run as the following on macOS (but I guess it's not macOS-specific) in DEBUG build:
./bin/client-test --stress_cpu_threads=32 --gtest_filter='*TestScannerKeepAlivePeriodicallyCrossServers*'
The error message and the stacktrace is below:
F20231113 12:21:13.431455 41195482 thread_restrictions.cc:79] Check failed: LoadTLS()->wait_allowed Waiting is not allowed to be used on this thread to prevent server-wide latency aberrations and deadlocks. Thread 41195482 (name: "rpc reactor", category: "reactor")
*** Check failure stack trace: ***
Process 77090 stopped
* thread #335, name = 'rpc reactor-41195482', stop reason = signal SIGABRT
frame #0: 0x00007fff205b890e libsystem_kernel.dylib`__pthread_kill + 10
libsystem_kernel.dylib`__pthread_kill:
-> 0x7fff205b890e <+10>: jae 0x7fff205b8918 ; <+20>
0x7fff205b8910 <+12>: movq %rax, %rdi
0x7fff205b8913 <+15>: jmp 0x7fff205b2ab9 ; cerror_nocancel
0x7fff205b8918 <+20>: retq
Target 0: (client-test) stopped.
(lldb) bt
* thread #335, name = 'rpc reactor-41195482', stop reason = signal SIGABRT
* frame #0: 0x00007fff205b890e libsystem_kernel.dylib`__pthread_kill + 10
frame #1: 0x00007fff205e75bd libsystem_pthread.dylib`pthread_kill + 263
frame #2: 0x00007fff2053c406 libsystem_c.dylib`abort + 125
frame #3: 0x000000010f64ebd8 libglog.1.dylib`google::LogMessage::SendToLog() [inlined] google::LogMessage::Fail() at logging.cc:1946:3 [opt]
frame #4: 0x000000010f64ebd2 libglog.1.dylib`google::LogMessage::SendToLog(this=0x000070001a95e108) at logging.cc:1920:5 [opt]
frame #5: 0x000000010f64f47a libglog.1.dylib`google::LogMessage::Flush(this=0x000070001a95e108) at logging.cc:1777:5 [opt]
frame #6: 0x000000010f65428f libglog.1.dylib`google::LogMessageFatal::~LogMessageFatal(this=0x000070001a95e108) at logging.cc:2557:5 [opt]
frame #7: 0x000000010f650349 libglog.1.dylib`google::LogMessageFatal::~LogMessageFatal(this=<unavailable>) at logging.cc:2556:37 [opt]
frame #8: 0x000000010e545473 libkudu_util.dylib`kudu::ThreadRestrictions::AssertWaitAllowed() at thread_restrictions.cc:79:3
frame #9: 0x000000010013ebb9 client-test`kudu::CountDownLatch::Wait(this=0x000070001a95e2a0) const at countdown_latch.h:74:5
frame #10: 0x000000010a1749f5 libkrpc.dylib`kudu::Notification::WaitForNotification(this=0x000070001a95e2a0) const at notification.h:127:12
frame #11: 0x000000010a1748e9 libkrpc.dylib`kudu::rpc::Proxy::SyncRequest(this=0x000000011317e9b8, method="ScannerKeepAlive", req=0x000070001a95e428, resp=0x000070001a95e408, controller=0x000070001a95e458) at proxy.cc:259:8
frame #12: 0x000000010697220f libtserver_service_proto.dylib`kudu::tserver::TabletServerServiceProxy::ScannerKeepAlive(this=0x000000011317e9b8, req=0x000070001a95e428, resp=0x000070001a95e408, controller=0x000070001a95e458) at tserver_service.proxy.cc:98:10
frame #13: 0x000000010525c5b6 libkudu_client.dylib`kudu::client::KuduScanner::Data::KeepAlive(this=0x000000011290c700) at scanner-internal.cc:664:3
frame #14: 0x0000000105269e76 libkudu_client.dylib`kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(this=0x0000000112899858)::$_0::operator()() const at scanner-internal.cc:112:16
frame #15: 0x0000000105269e30 libkudu_client.dylib`decltype(__f=0x0000000112899858)::$_0&>(fp)()) std::__1::__invoke<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0&>(kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0&) at type_traits:3694:1
frame #16: 0x0000000105269dd1 libkudu_client.dylib`void std::__1::__invoke_void_return_wrapper<void, true>::__call<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(__args=0x0000000112899858)::$_0&>(kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0&) at __functional_base:348:9
frame #17: 0x0000000105269d9d libkudu_client.dylib`std::__1::__function::__alloc_func<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0, std::__1::allocator<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0>, void ()>::operator(this=0x0000000112899858)() at functional:1558:16
frame #18: 0x0000000105268ac9 libkudu_client.dylib`std::__1::__function::__func<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0, std::__1::allocator<kudu::client::KuduScanner::Data::StartKeepAlivePeriodically(unsigned long long, std::__1::shared_ptr<kudu::rpc::Messenger>)::$_0>, void ()>::operator(this=0x0000000112899850)() at functional:1732:12
frame #19: 0x00000001013ae082 libtserver_test_util.dylib`std::__1::__function::__value_func<void ()>::operator(this=0x0000000112899850)() const at functional:1885:16
frame #20: 0x00000001013adee5 libtserver_test_util.dylib`std::__1::function<void ()>::operator(this=0x0000000112899850)() const at functional:2560:12
frame #21: 0x000000010a16cd62 libkrpc.dylib`kudu::rpc::PeriodicTimer::Callback(this=0x0000000112899830, my_callback_generation=1) at periodic.cc:194:5
frame #22: 0x000000010a17159a libkrpc.dylib`kudu::rpc::PeriodicTimer::Callback(this=0x0000000111d62528, s=0x000070001a95e910)::$_0::operator()(kudu::Status const&) const at periodic.cc:214:14
frame #23: 0x000000010a171512 libkrpc.dylib`decltype(__f=0x0000000111d62528, __args=0x000070001a95e910)::$_0&>(fp)(std::__1::forward<kudu::Status const&>(fp0))) std::__1::__invoke<kudu::rpc::PeriodicTimer::Callback(long long)::$_0&, kudu::Status const&>(kudu::rpc::PeriodicTimer::Callback(long long)::$_0&, kudu::Status const&) at type_traits:3694:1
frame #24: 0x000000010a1714b2 libkrpc.dylib`void std::__1::__invoke_void_return_wrapper<void, true>::__call<kudu::rpc::PeriodicTimer::Callback(__args=0x0000000111d62528, __args=0x000070001a95e910)::$_0&, kudu::Status const&>(kudu::rpc::PeriodicTimer::Callback(long long)::$_0&, kudu::Status const&) at __functional_base:348:9
frame #25: 0x000000010a171462 libkrpc.dylib`std::__1::__function::__alloc_func<kudu::rpc::PeriodicTimer::Callback(long long)::$_0, std::__1::allocator<kudu::rpc::PeriodicTimer::Callback(long long)::$_0>, void (kudu::Status const&)>::operator(this=0x0000000111d62528, __arg=0x000070001a95e910)(kudu::Status const&) at functional:1558:16
frame #26: 0x000000010a16ff11 libkrpc.dylib`std::__1::__function::__func<kudu::rpc::PeriodicTimer::Callback(long long)::$_0, std::__1::allocator<kudu::rpc::PeriodicTimer::Callback(long long)::$_0>, void (kudu::Status const&)>::operator(this=0x0000000111d62520, __arg=0x000070001a95e910)(kudu::Status const&) at functional:1732:12
frame #27: 0x0000000101b256da libmaster.dylib`std::__1::__function::__value_func<void (kudu::Status const&)>::operator(this=0x0000000111d62520, __args=0x000070001a95e910)(kudu::Status const&) const at functional:1885:16
frame #28: 0x0000000101b17cbd libmaster.dylib`std::__1::function<void (kudu::Status const&)>::operator(this= Lambda in File periodic.cc at Line 208, __arg=0x000070001a95e910)(kudu::Status const&) const at functional:2560:12
frame #29: 0x000000010a188f89 libkrpc.dylib`kudu::rpc::DelayedTask::TimerHandler(this=0x0000000111d62500, (null)=0x0000000111d62560, revents=256) at reactor.cc:767:5
frame #30: 0x000000010a1965ce libkrpc.dylib`void ev::base<ev_timer, ev::timer>::method_thunk<kudu::rpc::DelayedTask, &(loop=0x00000001136f78c0, w=0x0000000111d62560, revents=256))>(ev_loop*, ev_timer*, int) at ev++.h:479:7
frame #31: 0x000000010f327d41 libev.4.dylib`ev_invoke_pending(loop=0x00000001136f78c0) at ev.c:3155:11
frame #32: 0x000000010a18027e libkrpc.dylib`kudu::rpc::ReactorThread::InvokePendingCb(loop=0x00000001136f78c0) at reactor.cc:204:3
frame #33: 0x000000010f3283ce libev.4.dylib`ev_run(loop=0x00000001136f78c0, flags=0) at ev.c:3555:7
frame #34: 0x000000010a186dbe libkrpc.dylib`ev::loop_ref::run(this=0x0000000113502a18, flags=0) at ev++.h:211:7
frame #35: 0x000000010a186ba8 libkrpc.dylib`kudu::rpc::ReactorThread::RunThread(this=0x0000000113502a10) at reactor.cc:505:9
frame #36: 0x000000010a190fd8 libkrpc.dylib`kudu::rpc::ReactorThread::Init(this=0x00000001133ab378)::$_0::operator()() const at reactor.cc:196:48
frame #37: 0x000000010a190f9d libkrpc.dylib`decltype(__f=0x00000001133ab378)::$_0&>(fp)()) std::__1::__invoke<kudu::rpc::ReactorThread::Init()::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) at type_traits:3694:1
frame #38: 0x000000010a190f4d libkrpc.dylib`void std::__1::__invoke_void_return_wrapper<void, true>::__call<kudu::rpc::ReactorThread::Init(__args=0x00000001133ab378)::$_0&>(kudu::rpc::ReactorThread::Init()::$_0&) at __functional_base:348:9
frame #39: 0x000000010a190f1d libkrpc.dylib`std::__1::__function::__alloc_func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator(this=0x00000001133ab378)() at functional:1558:16
frame #40: 0x000000010a18fab9 libkrpc.dylib`std::__1::__function::__func<kudu::rpc::ReactorThread::Init()::$_0, std::__1::allocator<kudu::rpc::ReactorThread::Init()::$_0>, void ()>::operator(this=0x00000001133ab370)() at functional:1732:12
frame #41: 0x00000001013ae082 libtserver_test_util.dylib`std::__1::__function::__value_func<void ()>::operator(this=0x00000001133ab370)() const at functional:1885:16
frame #42: 0x00000001013adee5 libtserver_test_util.dylib`std::__1::function<void ()>::operator(this= Lambda in File reactor.cc at Line 196)() const at functional:2560:12
frame #43: 0x000000010e4f92e7 libkudu_util.dylib`kudu::Thread::SuperviseThread(arg=0x00000001133ab320) at thread.cc:691:3
frame #44: 0x00007fff205e78fc libsystem_pthread.dylib`_pthread_start + 224
frame #45: 0x00007fff205e3443 libsystem_pthread.dylib`thread_start + 15
The version information is below:
$ ./bin/client-test --version kudu 1.18.0-SNAPSHOT revision 8644d88dae6a76c5df3595b8a5aeb13df4d6ab5c build type DEBUG built by ... at 13 Nov 2023 11:32:09 PST on ...