[RATIS-592] One node ratis writes fail forever after first NotLeaderException or LeaderNotReadyException

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.3.0
• Fix Version/s: 0.4.0
• Component/s: gRPC
• Assignee: Siddharth Wagle
• Reporter: Siddharth Wagle
• Created: 17/Jun/19 18:26
• Updated: 28/Jun/19 08:57
• Resolved: 28/Jun/19 08:37

Description

RATIS-571, modified the GrpcClientProtocolClient to not set the AsyncStreamObserver reference to null on an exception, however, the ReplyMap reference is set to null. This results in the client getting an AlredyClosedException on the stream on a retry for a NotLeader or a LeadrNotReady exception and never recovers. This is common in a unit test scenario where a request is sent immediately after the cluster is up.

There is nothing special here about one node Ratis however, the HDDS unit tests that fail are all one node Ratis and the most probable cause is that with client retrying a different node each time, increases the chance of success on a three-node ring.

Issue Links

• blocks: HDDS-1555 Disable install snapshot for ContainerStateMachine