[HDDS-1892] HA failover attempt log level should be set to DEBUG

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 1.0.0
• Component/s: OM HA, Ozone Manager
• Assignee: Hanisha Koneru
• Reporter: Siyao Meng
• Created: 01/Aug/19 23:47
• Updated: 04/Jun/20 20:17
• Resolved: 17/Apr/20 21:10

Description

Current HA failover log level is INFO:

$ ozone sh volume create /volume
2019-08-01 20:55:03 INFO RetryInvocationHandler:411 — com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.NotLeader Exception): OM om2 is not the leader. Could not determine the leader node.
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotleaderException(OzoneManagerProtocolServerSideTranslatorPB.java:183)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:172)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:94)
        at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
, while invoking $Proxy19.submitRequest over nodeld=om2,nodeAddress=om2:9862 after 1 failover attempts. Trying to failover immediately.
2019-08-01 20:55:03 INFO RpcClient:288 — Creating Volume: volume, with hadoop as owner.
VOLUME_ALREADY_EXISTS Volume already exists

$ ozone sh bucket create /volume/bucket
2019-08-01 20:55:23 INFO RetryInvocationHandler:411 — com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ozone.om.exceptions.Notleader Exception): OM om2 is not the leader. Could not determine the leader node.
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.createNotLeaderException(OzoneManagerProtocolServerSideTranslatorPB.java:183)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:172)
        at org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:94)
        at org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:524)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1025)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:876) at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:822)
        at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2682)
, while invoking $Proxy19.submitRequest over nodeId.om2,nodeAddress.om2:9862 after 1 failover attempts. Trying to failover immediately.
2019-08-01 20:55:24 INFO RpcClient:425 — Creating Bucket: volume/bucket, with Versioning false and Storage Type set to DISK and Encryption set to false

The client will print out every failover attempt.

Suppress it by setting the log level to DEBUG.

Issue Links

• is fixed by: HDDS-3406 Remove RetryInvocation INFO logging from ozone CLI output