[KUDU-3266] Flakiness in dynamic_multi_master_test in VerifyClusterAfterMasterAddition() function

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.15.0
• Fix Version/s: 1.15.0
• Component/s: master, test
• Assignee: Bankim Bhavsar
• Reporter: Bankim Bhavsar
• Created: 18/Mar/21 20:14
• Updated: 22/Jul/21 15:37
• Resolved: 22/Jul/21 15:37

Description

ParameterizedRecoverMasterTest.TestRecoverDeadMasterSysCatalogCopy/1: /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/cluster_verifier.cc:119: Failure
Failed
Bad status: Not found: Unable to open table: the table does not exist: table_name: "table-1"
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/dynamic_multi_master-test.cc:603: Failure
Expected: cv.CheckRowCount(table_name, ClusterVerifier::EXACTLY, 0) doesn't generate new fatal failures in the current thread.
  Actual: it does.
2021-03-17T17:04:19Z chronyd exiting
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/master/dynamic_multi_master-test.cc:1099: Failure
Expected: VerifyClusterAfterMasterAddition(master_hps, orig_num_masters_) doesn't generate new fatal failures in the current thread.
  Actual: it does.

Although the same verification function is used by other tests for add master, this flakiness started showing up after introduction of the RecoverDeadMaster test.
https://github.com/apache/kudu/commit/4b4a8c0f2fdfd15524510821b27fc9c3b5d26b6b