[IMPALA-8764] Kudu data load failures due to "Clock considered unsynchronized"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 3.3.0
• Component/s: Infrastructure
• Labels: broken-build
• Epic Color: ghx-label-9
• Assignee: Unassigned
• Reporter: Sahil Takiar
• Created: 15/Jul/19 18:04
• Updated: 06/Aug/20 23:21
• Resolved: 06/Aug/20 23:21

Description

Dataload error:

03:08:38 03:08:38 Error executing impala SQL: Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql See: Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log

Digging through the mini-cluster logs, I see that the Kudu tservers crashed with this error:

F0715 02:58:43.202059   649 hybrid_clock.cc:339] Check failed: _s.ok() unable to get current time with error bound: Service unavailable: could not read system time source: Error reading clock. Clock considered unsynchronized
*** Check failure stack trace: ***
Wrote minidump to Impala/testdata/cluster/cdh6/node-3/var/log/kudu/ts/minidumps/kudu-tserver/395e6bb9-9b2f-468e-4d37d898-74b96d61.dmp
Wrote minidump to Impala/testdata/cluster/cdh6/node-3/var/log/kudu/ts/minidumps/kudu-tserver/395e6bb9-9b2f-468e-4d37d898-74b96d61.dmp
*** Aborted at 1563184723 (unix time) try "date -d @1563184723" if you are using GNU date ***
PC: @     0x7ff75ed631f7 __GI_raise
*** SIGABRT (@0x7d100000232) received by PID 562 (TID 0x7ff756c1e700) from PID 562; stack trace: ***
    @     0x7ff760b545e0 (unknown)
    @     0x7ff75ed631f7 __GI_raise
    @     0x7ff75ed648e8 __GI_abort
    @          0x1fb7309 kudu::AbortFailureFunction()
    @           0x9c054d google::LogMessage::Fail()
    @           0x9c240d google::LogMessage::SendToLog()
    @           0x9c0089 google::LogMessage::Flush()
    @           0x9c2eaf google::LogMessageFatal::~LogMessageFatal()
    @           0xc0c60e kudu::clock::HybridClock::WalltimeWithErrorOrDie()
    @           0xc0c67e kudu::clock::HybridClock::NowWithError()
    @           0xc0d4aa kudu::clock::HybridClock::NowForMetrics()
    @           0x9a29c0 kudu::FunctionGauge<>::WriteValue()
    @          0x1fb0dc0 kudu::Gauge::WriteAsJson()
    @          0x1fb3212 kudu::MetricEntity::WriteAsJson()
    @          0x1fb390e kudu::MetricRegistry::WriteAsJson()
    @           0xa856a3 kudu::server::DiagnosticsLog::LogMetrics()
    @           0xa8789a kudu::server::DiagnosticsLog::RunThread()
    @          0x1ff44d7 kudu::Thread::SuperviseThread()
    @     0x7ff760b4ce25 start_thread
    @     0x7ff75ee2634d __clone

Issue Links

• duplicates: IMPALA-9577 Use `system_unsync` time for Kudu test clusters
• is related to: IMPALA-8128 Build failure: Kudu functional data load - Kudu crashed
• is related to: IMPALA-5729 Kudu may crash in minicluster if clock becomes unsynchronized