[HDDS-10098] SCM NetworkTopologyImpl.sortByDistanceCost throws IndexOutOfBoundsException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Unassigned
• Reporter: Wei-Chiu Chuang
• Created: 09/Jan/24 18:42
• Updated: 24/Apr/24 23:44
• Resolved: 09/Jan/24 19:05

Description

2024-01-09 08:57:25,710 WARN [IPC Server handler 19 on 9863]-org.apache.hadoop.hdds.scm.node.SCMNodeManager: Cannot find node for uuid 493970a9-5696-4eaf-8d30-49c2ee19798f
2024-01-09 08:57:25,710 WARN [IPC Server handler 19 on 9863]-org.apache.hadoop.ipc.Server: IPC Server handler 19 on 9863, call Call#5173 Retry#0 org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocol.send fro
m 10.140.17.6:44331
java.lang.IndexOutOfBoundsException: Index: 2, Size: 2
        at java.util.ArrayList.rangeCheck(ArrayList.java:657)
        at java.util.ArrayList.get(ArrayList.java:433)
        at org.apache.hadoop.hdds.scm.net.NetworkTopologyImpl.sortByDistanceCost(NetworkTopologyImpl.java:761)
        at org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer.sortDatanodes(SCMBlockProtocolServer.java:353)
        at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.sortDatanodes(ScmBlockLocationProtocolServerSideTranslatorPB.java:267)
        at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.processMessage(ScmBlockLocationProtocolServerSideTranslatorPB.java:158)
        at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
        at org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:113)
        at org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:14238)
        at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
        at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994)
        at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
        at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2899)

Issue Links

• duplicates: HDDS-9651 Restore random selection of datanode when reading RATIS objects
• is duplicated by: HDDS-10747 Key get for EC fails when DNs are stopped and SCM is restarted