Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2943

TsTabletManagerITest.TestTableStats flaky due to WAL/cmeta term disagreement

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 1.11.0
    • None
    • consensus, test
    • None

    Description

      This new test failed in a strange (and worrying) way:

      /home/jenkins-slave/workspace/kudu-master/1/src/kudu/integration-tests/ts_tablet_manager-itest.cc:753: Failure
Failed
Bad status: Corruption: Unable to start RaftConsensus: The last op in the WAL with id 3.4 has a term (3) that is greater than the latest recorded term, which is 2

      From a brief dig through the code, looks like this means the current term as per the on-disk cmeta file is older than the term in the latest WAL op.

      I can believe that this is somehow due to InternalMiniCluster exercising clean shutdown paths that aren't well tested or robust, but it'd be nice to determine that with certainty.

      I've attached the full test log.

      Attachments

        1. ts_tablet_manager-itest.txt
          721 kB
          Adar Dembo

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. KUDU-2942 A rare flaky test for the aggregated live row count

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. KUDU-2944 TsTabletManagerITest.TestTableStats flaky due to data race

          • Major - Major loss of function.
          • Open

          Activity

            People

              Unassigned Unassigned
              adar Adar Dembo
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: