Lately we found some tablets in a cluster with a very large "time_since_last_leader_heartbeat" metric, they are LEARNER/NON_VOTER and seems couldn't become VOTER for a long time.
These replicas created during the rebalance/tablet_copy process. After beginning a new copy session from leader to the new added NON_VOTER peer, leadership changed, old leader aborted uncommited CHANGE_CONFIG_OP operation. Finally the tablet_copy session ended but new leader knew nothing about the new peer.
Master didn't delete this new added replica because it has a larger opid_index than the latest reported committed config. See the comments in CatalogManager::ProcessTabletReport
Maybe we shouldn't use opid_index to determine if replicas are in the process of copying.