Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2048

consensus: Only evict a replica is a majority is up to date

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.0
    • Fix Version/s: 1.6.0
    • Component/s: consensus, recovery
    • Labels:
      None

      Description

      In the context of replica eviction and 3-2-3 recovery, we currently have a "hacky" rule that states that evicting down to less than 2 replicas in a config is prohibited. However we don't currently check to see, when evicting, whether that would leave the config with less than a majority of caught-up voters.

      That means, for example, that if we have a config of 3 replicas

      { A, B, C }

      and B falls behind, so is currently undergoing a tablet copy, and C goes offline then the algorithm will evict C. However, since A is the only up-to-date replica, this leaves the config in a state where nothing can commit until B is done copying. Even worse, if B is killed or has an error, then we are left in a state that requires manual recovery.

      Consider adding an additional rule that states that to evict a node, we must have a majority of up-to-date replicas that are recently active. This will help prevent certain problem scenarios like the above from occurring.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tlipcon Todd Lipcon
                Reporter:
                mpercy Mike Percy
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: