Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8764

Kudu data load failures due to "Clock considered unsynchronized"

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: Impala 3.3.0
    • Fix Version/s: None
    • Component/s: Infrastructure
    • Labels:
    • Epic Color:
      ghx-label-9

      Description

      Dataload error:

      03:08:38 03:08:38 Error executing impala SQL: Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql See: Impala/logs/data_loading/sql/functional/create-functional-query-exhaustive-impala-generated-kudu-none-none.sql.log
      

      Digging through the mini-cluster logs, I see that the Kudu tservers crashed with this error:

      F0715 02:58:43.202059   649 hybrid_clock.cc:339] Check failed: _s.ok() unable to get current time with error bound: Service unavailable: could not read system time source: Error reading clock. Clock considered unsynchronized
      *** Check failure stack trace: ***
      Wrote minidump to Impala/testdata/cluster/cdh6/node-3/var/log/kudu/ts/minidumps/kudu-tserver/395e6bb9-9b2f-468e-4d37d898-74b96d61.dmp
      Wrote minidump to Impala/testdata/cluster/cdh6/node-3/var/log/kudu/ts/minidumps/kudu-tserver/395e6bb9-9b2f-468e-4d37d898-74b96d61.dmp
      *** Aborted at 1563184723 (unix time) try "date -d @1563184723" if you are using GNU date ***
      PC: @     0x7ff75ed631f7 __GI_raise
      *** SIGABRT (@0x7d100000232) received by PID 562 (TID 0x7ff756c1e700) from PID 562; stack trace: ***
          @     0x7ff760b545e0 (unknown)
          @     0x7ff75ed631f7 __GI_raise
          @     0x7ff75ed648e8 __GI_abort
          @          0x1fb7309 kudu::AbortFailureFunction()
          @           0x9c054d google::LogMessage::Fail()
          @           0x9c240d google::LogMessage::SendToLog()
          @           0x9c0089 google::LogMessage::Flush()
          @           0x9c2eaf google::LogMessageFatal::~LogMessageFatal()
          @           0xc0c60e kudu::clock::HybridClock::WalltimeWithErrorOrDie()
          @           0xc0c67e kudu::clock::HybridClock::NowWithError()
          @           0xc0d4aa kudu::clock::HybridClock::NowForMetrics()
          @           0x9a29c0 kudu::FunctionGauge<>::WriteValue()
          @          0x1fb0dc0 kudu::Gauge::WriteAsJson()
          @          0x1fb3212 kudu::MetricEntity::WriteAsJson()
          @          0x1fb390e kudu::MetricRegistry::WriteAsJson()
          @           0xa856a3 kudu::server::DiagnosticsLog::LogMetrics()
          @           0xa8789a kudu::server::DiagnosticsLog::RunThread()
          @          0x1ff44d7 kudu::Thread::SuperviseThread()
          @     0x7ff760b4ce25 start_thread
          @     0x7ff75ee2634d __clone
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stakiar Sahil Takiar
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: