[KUDU-1576] raft-config will stay in pending state a long time in node crash situation.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Fix Version/s: NA
• Component/s: consensus
• Assignee: Unassigned
• Reporter: zhangsong
• Created: 23/Aug/16 03:02
• Updated: 01/Jun/20 20:50
• Resolved: 01/Jun/20 20:50

Description

After experiencing two phsical nodes crash, i found one of my table is read-only. i did some search and found that both of two followers of a tablet is in down state. But from web-ui those down follower are still there. So i try to recovery the table with kudu-admin tool's change_config and it failed with below message:
Pending config: local: false peers { permanent_uuid: "515ab1adcbd64081b646a86133f5f60d" member_type: VOTER last_known_addr

{ host: "one_of_follower" port: 7052 }

} peers { permanent_uuid: "3a77ef5039f447d29db5a44c92279a7a" member_type: VOTER last_known_addr

{ host: "current_leader" port: 7052 }

}

it seems that after one of raft-config members is down, when current leader is trying to replicate the config, the "515ab1adcbd64081b646a86133f5f60d" crashed . In which case , the config just pend there, as the raft-config will never get accepted by majority.

It will be better that we can have some machanism to fix it , at least manually.

Issue Links

• is related to: KUDU-1449 tablet unavailable caused by follower can not upgrade to leader.