[HDFS-16058] If the balancer cannot update the datanode's storage type in time, the balancer will throw NullPointerException

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: balancer & mover
• Assignee: Unassigned
• Reporter: lei w
• Created: 08/Jun/21 09:12
• Updated: 08/Jun/21 10:00

Description

The main logic of the balancer is to initialize the cluster information (number of datanodes, datanode's storage type and network topology etc)、calculate the source DataNode and target DataNode that need to move the block and then obtain the movable block through the getblocks(). Finally , move blocks. If a DataNode adds another type of storage and writes an EC block after the cluster information is initialized, and then returns the newly written EC block to balancer by getblocks(), it will cause balancer to throw NullPointerException .The main reason is that the balancer does not record the newly added storage type of the DataNode, which causes the current DataNode information not to be recorded when generate an object of type DBlockStriped . Finally, null is returned when the block of the current DataNode is obtained through the object of type DBlockStriped.

2021-04-23 19:38:21,233 WARN org.apache.hadoop.hdfs.server.balancer.Dispatcher: Dispatcher thread failed
java.lang.NullPointerException
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.chooseProxySource(Dispatcher.java:325)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.markMovedIfGoodBlock(Dispatcher.java:291)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.chooseBlockAndProxy(Dispatcher.java:271)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$PendingMove.access$2500(Dispatcher.java:235)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.chooseNextMove(Dispatcher.java:886)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.dispatchBlocks(Dispatcher.java:943)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$Source.access$3200(Dispatcher.java:751)
        at org.apache.hadoop.hdfs.server.balancer.Dispatcher$2.run(Dispatcher.java:1221)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)