RPC Client should implement a retry and failover proxy provider to failover between OM Ratis clients. The failover should occur in two scenarios:
- When the client is unable to connect to the OM (either because of network issues or because the OM is down). The client retry proxy provider should failover to next OM in the cluster.
- When OM Ratis Client receives a response from the Ratis server for its request, it also gets the LeaderId of server which processed this request (the current Leader OM nodeId). This information should be propagated back to the client. The Client failover Proxy provider should failover to the leader OM node. This helps avoid an extra hop from Follower OM Ratis Client to Leader OM Ratis server for every request.