Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3098

leadership change during tablet_copy process may lead to an isolate replica

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.10.1
    • Fix Version/s: None
    • Component/s: consensus, master
    • Labels:
      None

      Description

      Lately we found some tablets in a cluster with a very large "time_since_last_leader_heartbeat" metric, they are LEARNER/NON_VOTER and seems couldn't become VOTER for a long time.

      These replicas created during the rebalance/tablet_copy process. After beginning a new copy session from leader to the new added NON_VOTER peer, leadership changed, old leader aborted uncommited CHANGE_CONFIG_OP operation. Finally the tablet_copy session ended but new leader knew nothing about the new peer. 

      Master didn't delete this new added replica because it has a larger opid_index than the latest reported committed config. See the comments in CatalogManager::ProcessTabletReport

      // 5. Tombstone a replica that is no longer part of the Raft config (and
      // not already tombstoned or deleted outright).
      //
      // If the report includes a committed raft config, we only tombstone if
      // the opid_index is strictly less than the latest reported committed
      // config. This prevents us from spuriously deleting replicas that have
      // just been added to the committed config and are in the process of copying.
      

      Maybe we shouldn't use opid_index to determine if replicas are in the process of copying.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zhangyifan27 YifanZhang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: