[HDDS-10814] Log exception stack trace in addSCM only at debug level

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Implemented
• Fix Version/s: 2.0.0
• Component/s: SCM
• Labels: pull-request-available
• Assignee: Tejaskriya Madhan
• Reporter: Tejaskriya Madhan
• Created: 06/May/24 08:26
• Updated: 08/May/24 10:09
• Resolved: 08/May/24 10:09

Description

Failure in adding scm during setup is logged with a big stack trace even though the addSCM call is successful eventually due to the retry logic

ERROR [IPC Server handler 3 on 9863]-org.apache.hadoop.hdds.scm.ha.SCMRatisServerImpl: Failed to update Ratis configuration and add new peer. Cannot add new SCM: db781281-bbca-4c20-a5d9-3e33005b8b8c.
org.apache.ratis.protocol.exceptions.ReconfigurationInProgressException: Reconfiguration is already in progress: 1: peers:[db781281-bbca-4c20-a5d9-3e33005b8b8c|<host>:9894|admin:|client:|dataStream:|priority:0|startupRole:FOLLOWER]|listeners:[], old=null
    at org.apache.ratis.server.impl.RaftServerImpl.setConfigurationAsync(RaftServerImpl.java:1217)
    at org.apache.ratis.server.impl.RaftServerImpl.setConfiguration(RaftServerImpl.java:1189)
    at org.apache.ratis.server.impl.RaftServerProxy.setConfiguration(RaftServerProxy.java:455)
    at org.apache.hadoop.hdds.scm.ha.SCMRatisServerImpl.addSCM(SCMRatisServerImpl.java:325)
    at org.apache.hadoop.hdds.scm.ha.SCMHAManagerImpl.addSCM(SCMHAManagerImpl.java:420)
    at org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.addSCM(SCMBlockProtocolServer.java:320)
    at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.getAddSCMResponse(ScmBlockLocationProtocolServerSideTranslatorPB.java:253)
    at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.processMessage(ScmBlockLocationProtocolServerSideTranslatorPB.java:155)
    at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
    at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:113)
    at org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:14238)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2899) 

Log level can be reduced to warn.

Issue Links

• links to: GitHub Pull Request #6642