Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2943

TsTabletManagerITest.TestTableStats flaky due to WAL/cmeta term disagreement

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.11.0
    • Fix Version/s: None
    • Component/s: consensus, test
    • Labels:
      None

      Description

      This new test failed in a strange (and worrying) way:

      /home/jenkins-slave/workspace/kudu-master/1/src/kudu/integration-tests/ts_tablet_manager-itest.cc:753: Failure
      Failed
      Bad status: Corruption: Unable to start RaftConsensus: The last op in the WAL with id 3.4 has a term (3) that is greater than the latest recorded term, which is 2
      

      From a brief dig through the code, looks like this means the current term as per the on-disk cmeta file is older than the term in the latest WAL op.

      I can believe that this is somehow due to InternalMiniCluster exercising clean shutdown paths that aren't well tested or robust, but it'd be nice to determine that with certainty.

      I've attached the full test log.

        Attachments

        1. ts_tablet_manager-itest.txt
          721 kB
          Adar Dembo

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                adar Adar Dembo
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: