[RATIS-313] Raft client ignores the reinitilization exception when the raft server is not ready

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Assignee: Shashikant Banerjee
• Reporter: Shashikant Banerjee
• Created: 10/Sep/18 17:02
• Updated: 11/Sep/18 13:18
• Resolved: 11/Sep/18 13:18

Description

This was found in Ozone testing.

Three nodes in the pipeline.

group-2041ABBEE452:[bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9:172.27.12.96:9858, faa888b7-92bb-4e35-a38c-711bd1c28948:172.27.80.23:9858, ff544de8-96ea-4097-8cdc-460ac1c60db7:172.27.23.161:9858]

On two servers, the reinitialization request succeeds, 

2018-09-09 10:49:40,938 INFO org.apache.ratis.server.impl.RaftServerProxy: faa888b7-92bb-4e35-a38c-711bd1c28948: reinitializeAsync ReinitializeRequest(client-682DF1D0F737->faa888b7-92bb-4e35-a38c-711bd1c28948) in group-7347726F7570, cid=4, seq=0 RW, null, group-2041ABBEE452:[bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9:172.27.12.96:9858, faa888b7-92bb-4e35-a38c-711bd1c28948:172.27.80.23:9858, ff544de8-96ea-4097-8cdc-460ac1c60db7:172.27.23.161:9858

2018-09-09 10:49:40,209 INFO org.apache.ratis.server.impl.RaftServerProxy: bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9: reinitializeAsync ReinitializeRequest(client-DFE3ACF394F9->bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9) in group-7347726F7570, cid=3, seq=0 RW, null, group-2041ABBEE452:[bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9:172.27.12.96:9858, faa888b7-92bb-4e35-a38c-711bd1c28948:172.27.80.23:9858, ff544de8-96ea-4097-8cdc-460ac1c60db7:172.27.23.161:9858]

But around the same time, the third server is not ready

2018-09-09 10:49:41,414 WARN org.apache.ratis.grpc.server.RaftServerProtocolService: ff544de8-96ea-4097-8cdc-460ac1c60db7: Failed requestVote bfe9c5f2-da9b-4a8f-9013-7540cbbed1c9->ff544de8-96ea-4097-8cdc-460ac1c60db7#0: org.apache.ratis.protocol.ServerNotReadyException: Server ff544de8-96ea-4097-8cdc-460ac1c60db7 is not [RUNNING]: current state is STARTING

Though the reinitialization request never got processed on this server, the exception is ignored in RaftClientImpl. This needs to be addressed

Issue Links

• blocks: HDDS-420 putKey failing with KEY_ALLOCATION_ERROR