[HDDS-3031] OM HA- Client requests get LeaderNotReadyException after OM restart

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Labels: OMHA OMHATest pull-request-available
• Assignee: Bharat Viswanadham
• Reporter: Bharat Viswanadham
• Created: 19/Feb/20 00:48
• Updated: 26/Feb/20 22:19
• Resolved: 20/Feb/20 20:04

Description

Scenario:

1.Set up OM HA cluster.

2. Perform some write operations.

3. Restart OM's.

4. Now try any write operation.

Below error will be thrown for 15 times, and finally, client request will fail.

 
2020-02-15 10:11:23,244 [qtp2025269734-19] INFO org.apache.hadoop.io.retry.RetryInvocationHandler: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.OMLeaderNotReadyException): om1@group-D0D586AF6951 is in LEADER state but not ready yet.
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer.processReply(OzoneManagerRatisServer.java:177)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerRatisServer.submitRequest(OzoneManagerRatisServer.java:136)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequestToRatis(OzoneManagerProtocolServerSideTranslatorPB.java:162)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:118)
        at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:72)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:97)
        at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
, while invoking $Proxy81.submitRequest over nodeId=om1,nodeAddress=om-ha-1.vpc.cloudera.com:9862 after 1 failover attempts. Trying to failover immediately.

 

Issue Links

• fixes: HDDS-3004 OM HA stability issues
• links to: GitHub Pull Request #564