[HDDS-11350] NullPointerException thrown on checking container balancer status

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Balancer
• Labels: pull-request-available
• Target Version/s: 2.0.0
• Epic Link: Fix the Container Balancer Status command
• Assignee: Siddhant Sangwan
• Reporter: Jyotirmoy Sinha
• Created: 21/Aug/24 14:22
• Updated: 28/Sep/24 16:38
• Resolved: 04/Sep/24 05:41

Description

Scenario - Run container balancer when there is no data in the cluster.

Configs -

"hdds.container.balancer.trigger.du.before.move.enable": "true",
"ozone.scm.container.size": "1GB",
"hdds.container.balancer.balancing.iteration.interval": "5m",
"hdds.container.balancer.size.moved.max.per.iteration": "2GB" 

Error stacktrace -

# /opt/cloudera/parcels/CDH/bin/ozone admin containerbalancer start  -t 1  -d 100
Container Balancer started successfully.

# /opt/cloudera/parcels/CDH/bin/ozone admin containerbalancer status
24/08/21 14:14:30 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
    at org.apache.hadoop.hdds.scm.container.balancer.ContainerBalancerTask.getCurrentIterationsStatistic(ContainerBalancerTask.java:353)
    at org.apache.hadoop.hdds.scm.container.balancer.ContainerBalancer.getBalancerStatusInfo(ContainerBalancer.java:191)
    at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getContainerBalancerStatusInfo(SCMClientProtocolServer.java:1213)
    at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getContainerBalancerStatusInfo(StorageContainerLocationProtocolServerSideTranslatorPB.java:1210)
    at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:608)
    at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
    at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:233)
    at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2899)
, while invoking $Proxy20.submitRequest over nodeId=node1,nodeAddress=ccycloud-1.quasar-ypdsqw.root.comops.site/10.140.49.132:9860 after 2 failover attempts. Trying to failover after sleeping for 2000ms.
24/08/21 14:14:34 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
    at org.apache.hadoop.hdds.scm.container.balancer.ContainerBalancerTask.getCurrentIterationsStatistic(ContainerBalancerTask.java:353)
    at org.apache.hadoop.hdds.scm.container.balancer.ContainerBalancer.getBalancerStatusInfo(ContainerBalancer.java:191)
    at org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer.getContainerBalancerStatusInfo(SCMClientProtocolServer.java:1213)
    at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.getContainerBalancerStatusInfo(StorageContainerLocationProtocolServerSideTranslatorPB.java:1210)
    at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.processRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:608)
    at org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:89)
    at org.apache.hadoop.hdds.scm.protocol.StorageContainerLocationProtocolServerSideTranslatorPB.submitRequest(StorageContainerLocationProtocolServerSideTranslatorPB.java:233)
    at org.apache.hadoop.hdds.protocol.proto.StorageContainerLocationProtocolProtos$StorageContainerLocationProtocolService$2.callBlockingMethod(StorageContainerLocationProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:533)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1070)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:994)
    at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:922)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2899)
, while invoking $Proxy20.submitRequest over nodeId=node1,nodeAddress=ccycloud-1.quasar-ypdsqw.root.comops.site/10.140.49.132:9860 after 4 failover attempts. Trying to failover after sleeping for 2000ms.
24/08/21 14:14:36 INFO retry.RetryInvocationHandler: com.google.protobuf.ServiceException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException 

Issue Links

• is caused by: HDDS-11120 Rich rebalancing status info
• links to: GitHub Pull Request #7134