[KUDU-3098] leadership change during tablet_copy process may lead to an isolate replica

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.10.1
• Component/s: consensus, master
• Assignee: Unassigned
• Reporter: YifanZhang
• Created: 31/Mar/20 11:32
• Updated: 08/Jul/21 05:41

Description

Lately we found some tablets in a cluster with a very large "time_since_last_leader_heartbeat" metric, they are LEARNER/NON_VOTER and seems couldn't become VOTER for a long time.

These replicas created during the rebalance/tablet_copy process. After beginning a new copy session from leader to the new added NON_VOTER peer, leadership changed, old leader aborted uncommited CHANGE_CONFIG_OP operation. Finally the tablet_copy session ended but new leader knew nothing about the new peer. 

Master didn't delete this new added replica because it has a larger opid_index than the latest reported committed config. See the comments in CatalogManager::ProcessTabletReport

// 5. Tombstone a replica that is no longer part of the Raft config (and
// not already tombstoned or deleted outright).
//
// If the report includes a committed raft config, we only tombstone if
// the opid_index is strictly less than the latest reported committed
// config. This prevents us from spuriously deleting replicas that have
// just been added to the committed config and are in the process of copying.

Maybe we shouldn't use opid_index to determine if replicas are in the process of copying.