Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1074

KuduClient::ListTables() does not retry

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: Feature Complete
    • Fix Version/s: n/a
    • Component/s: client
    • Labels:
      None

      Description

      I am running into an issue in the test environment where KuduClient::ListTables is returning me a ServiceUnavailable error.

      In the test environment, it looks like this: if an ExternalMiniCluster is started, and then TestWorkload::Setup() is immediately called, there is a race between the master completing startup and marking itself as leader (even with LocalConsensus) and the call by TestWorkload to check if the table it's about to create exists. If the Master is in the middle of starting up then we get an error that looks like this, because ListTables() doesn't retry up to some timeout.

      F0826 21:17:02.727133  3067 test_workload.cc:144] Check failed: _s.ok() Bad status: Service unavailable: Leader not yet ready to serve requests
      *** Check failure stack trace: ***
          @           0x5bd79d  google::LogMessage::Fail() at /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/thirdparty/glog-0.3.3/src/logging.cc:1458
          @           0x5c1c7d  google::LogMessage::SendToLog() at /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/thirdparty/glog-0.3.3/src/logging.cc:1412
          @           0x5bfaeb  google::LogMessage::Flush() at /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/thirdparty/glog-0.3.3/src/logging.cc:1282
          @           0x5bfe11  google::LogMessageFatal::~LogMessageFatal() at /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/thirdparty/glog-0.3.3/src/logging.cc:1985
          @           0x5ba747  kudu::TestWorkload::Setup() at /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/src/kudu/integration-tests/test_workload.cc:141
          @           0x5905a7  kudu::DeleteTableTest_TestDeleteTableWithConcurrentWrites_Test::TestBody() at /data1/jenkins-workspace/kudu-gerrit/BUILD_TYPE/RELEASE/label/kudu-gerrit-slaves/src/kudu/integration-tests/delete_table-test.cc:313
      

        Attachments

          Activity

            People

            • Assignee:
              mpercy Mike Percy
              Reporter:
              mpercy Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: