[KUDU-2943] TsTabletManagerITest.TestTableStats flaky due to WAL/cmeta term disagreement

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.11.0
• Component/s: consensus, test
• Assignee: Unassigned
• Reporter: Adar Dembo
• Created: 17/Sep/19 20:40
• Updated: 24/Nov/21 19:51

Description

This new test failed in a strange (and worrying) way:

/home/jenkins-slave/workspace/kudu-master/1/src/kudu/integration-tests/ts_tablet_manager-itest.cc:753: Failure
Failed
Bad status: Corruption: Unable to start RaftConsensus: The last op in the WAL with id 3.4 has a term (3) that is greater than the latest recorded term, which is 2

From a brief dig through the code, looks like this means the current term as per the on-disk cmeta file is older than the term in the latest WAL op.

I can believe that this is somehow due to InternalMiniCluster exercising clean shutdown paths that aren't well tested or robust, but it'd be nice to determine that with certainty.

I've attached the full test log.

Issue Links

• relates to: KUDU-2942 A rare flaky test for the aggregated live row count
• relates to: KUDU-2944 TsTabletManagerITest.TestTableStats flaky due to data race