[KUDU-2408] Restarting masters and tservers on macOS always leads to fatal 'update clock beyond the max. error'

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 1.8.0
• Fix Version/s: 1.8.0
• Component/s: master, tserver
• Assignee: David Alves
• Reporter: Alexey Serbin
• Created: 13/Apr/18 17:56
• Updated: 11/May/18 16:00
• Resolved: 11/May/18 16:00

Description

NOTE: This issue is specific only for macOS and does not show up at Linux. As of now, macOS is a development-only platform.

Once master/tserser is started and gets some data under their data directories, shutting them down and then restarting always leads to the following fatal error (i.e. crash):

F0413 10:47:24.042886 10723328 tablet_bootstrap.cc:904] Check failed: _s.ok() Bad status: Invalid argument: Tried to update clock beyond the max. error.
*** Check failure stack trace: ***
    @        0x1121c5f90  google::LogMessage::SendToLog()
    @        0x1121c6631  google::LogMessage::Flush()
    @        0x1121ca1cf  google::LogMessageFatal::~LogMessageFatal()
    @        0x1121c6ea9  google::LogMessageFatal::~LogMessageFatal()
    @        0x10e357d94  kudu::tablet::TabletBootstrap::HandleReplicateMessage()
    @        0x10e35753b  kudu::tablet::TabletBootstrap::HandleEntry()
    @        0x10e354024  kudu::tablet::TabletBootstrap::PlaySegments()
    @        0x10e34fac6  kudu::tablet::TabletBootstrap::RunBootstrap()
    @        0x10e34d50c  kudu::tablet::TabletBootstrap::Bootstrap()
    @        0x10e34d0b7  kudu::tablet::BootstrapTablet()
    @        0x10db77d57  kudu::tserver::TSTabletManager::OpenTablet()
    @        0x10db8d1b9  boost::_mfi::mf2<>::operator()()
    @        0x10db8d10b  boost::_bi::list3<>::operator()<>()
    @        0x10db8d07a  boost::_bi::bind_t<>::operator()()
    @        0x10db8cde0  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @        0x110f83698  boost::function0<>::operator()()
    @        0x111ab90f9  kudu::FunctionRunnable::Run()
    @        0x111ab361c  kudu::ThreadPool::DispatchThread()
    @        0x111ace669  boost::_mfi::mf0<>::operator()()
    @        0x111ace5d0  boost::_bi::list1<>::operator()<>()
    @        0x111ace57a  boost::_bi::bind_t<>::operator()()
    @        0x111ace35d  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @        0x110f83698  boost::function0<>::operator()()
    @        0x111a9fae8  kudu::Thread::SuperviseThread()
    @     0x7fff8a89699d  _pthread_body
    @     0x7fff8a89691a  _pthread_start
    @     0x7fff8a894351  thread_start