[KUDU-3325] When wal is deleted, fault recovery and load balancing are abnormal

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: consensus
• Code Review: https://gerrit.cloudera.org/#/c/17907/
• Assignee: Unassigned
• Reporter: yejiabao_h
• Created: 06/Oct/21 07:34
• Updated: 13/Oct/21 08:11

Description

1、using kudu leader step down to create multiple wal message

./kudu tablet leader_step_down $MASTER_IP   1299f5a939d2453c83104a6db0cae3e7

wal

cmeta

2、stop one of tserver to start tablet recovery，so that we can make opid_index flush to cmeta

wal

cmeta

3、stop all tservers，and delete tablet wal

4、start all tservers

we can see the index in wal starts counting from 1, but the opid_index recorded in cmeta is the value 20 which is before deleting wal
 

wal

 

cmeta

 

5、stop a tserver，trigger fault recovery

when the leader recovery a replica, and master request change raft config to add the new replica to new raft config, leader replica while ignored because the opindex is smaller than that in cmeta.
 

6、delete all wals

7、kudu cluster rebalance

./kudu cluster rebalance $MASTER_IP

rebalance is also failed when change raft config