[HDFS-16389] Improve NNThroughputBenchmark test mkdirs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.9.2
• Fix Version/s: 3.4.0, 3.2.4, 3.3.5
• Component/s: benchmarks, namenode
• Labels: pull-request-available
• Assignee: JiangHua Zhu
• Reporter: JiangHua Zhu
• Created: 20/Dec/21 05:42
• Updated: 19/Apr/22 03:14
• Resolved: 18/Apr/22 03:22

Description

When using the NNThroughputBenchmark test to create a large number of directories, some abnormal information will be prompted.
Here is the command:
./bin/hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -fs hdfs://xxxx -op mkdirs -threads 30 -dirs 5000000

There are some exceptions here, such as:
21/12/20 10:25:00 INFO namenode.NNThroughputBenchmark: Starting benchmark: mkdirs
21/12/20 10:25:01 INFO namenode.NNThroughputBenchmark: Generate 5000000 inputs for mkdirs
21/12/20 10:25:08 ERROR namenode.NNThroughputBenchmark: java.lang.ArrayIndexOutOfBoundsException: 20
at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextDirName(FileNameGenerator.java:65)
at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextFileName(FileNameGenerator.java:73)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$MkdirsStats.generateInputs(NNThroughputBenchmark.java:668)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 20
at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextDirName(FileNameGenerator.java:65)
at org.apache.hadoop.hdfs.server.namenode.FileNameGenerator.getNextFileName(FileNameGenerator.java:73)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$MkdirsStats.generateInputs(NNThroughputBenchmark.java:668)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:257)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1528)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.runBenchmark(NNThroughputBenchmark.java:1430)
at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1550)

These messages appear because some parameters are incorrectly set, such as dirsPerDir or filesPerDir.
When we see this log, this will make us have some questions.

Issue Links

• links to: GitHub Pull Request #3819