[HDFS-16269] [Fix] Improve NNThroughputBenchmark#blockReport operation

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.2
• Fix Version/s: 3.4.0, 3.3.2
• Component/s: benchmarks, namenode
• Labels: pull-request-available
• Assignee: JiangHua Zhu
• Reporter: JiangHua Zhu
• Created: 12/Oct/21 07:00
• Updated: 02/Nov/21 18:07
• Resolved: 01/Nov/21 16:01

Description

When using NNThroughputBenchmark to verify the blockReport, you will get some exception information.
Commands used:
./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs xxxx -op blockReport -datanodes 3 -reports 1

The exception information:
21/10/12 14:35:18 INFO namenode.NNThroughputBenchmark: Starting benchmark: blockReport
21/10/12 14:35:19 INFO namenode.NNThroughputBenchmark: Creating 10 files with 10 blocks each.
21/10/12 14:35:19 ERROR namenode.NNThroughputBenchmark: java.lang.ArrayIndexOutOfBoundsException: 50009
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 50009
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.addBlocks(NNThroughputBenchmark.java:1161)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$BlockReportStats.generateInputs(NNThroughputBenchmark.java:1143)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)

Checked some code and found that the problem appeared here.
private ExtendedBlock addBlocks(String fileName, String clientName)
throws IOException {
for(DatanodeInfo dnInfo: loc.getLocations())

{ int dnIdx = dnInfo.getXferPort()-1; datanodes[dnIdx].addBlock(loc.getBlock().getLocalBlock()); }

}
It can be seen from this that what dnInfo.getXferPort() gets is a port information and should not be used as an index of an array.

Issue Links

• duplicates: HDFS-14240 blockReport test in NNThroughputBenchmark throws ArrayIndexOutOfBoundsException
• links to: GitHub Pull Request #3544