Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1779

Consensus "stuck" with all transaction trackers are at limit

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: consensus
    • Labels:

      Description

      In a stress cluster, I saw one tablet get "stuck" in the following state:

      • the transaction_tracker on all three replicas is "full" (no more can be submitted)
      • leader elections proceed just fine, but no leader is able to advance the commit index

      The issue seems to be that a replica will respond with 'CANNOT_PREPARE' when its transaction tracker is full. The leader then ignores this response, and doesn't advance the majority-replicated watermark. The transaction tracker stays full forever because the in-flight transactions can't get committed.

      Notes to follow.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tlipcon Todd Lipcon
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                1 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated: