[HDFS-9421] NNThroughputBenchmark replication test NPE with -namenode option

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
• Component/s: benchmarks
• Hadoop Flags: Reviewed
• Assignee: Mingliang Liu
• Reporter: Xiaoyu Yao
• Created: 12/Nov/15 23:49
• Updated: 03/Jul/17 19:54
• Resolved: 17/Nov/15 00:46

Description

Hit the following NPE when reviewing fix for HDFS-9387 with manual tests as NNThroughputBenchmark currently does not have JUnit tests.

HW11217:centos6.4 xyao$ hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op replication -namenode hdfs://HW11217.local:9000
15/11/12 14:52:03 INFO namenode.NNThroughputBenchmark: Starting benchmark: replication
15/11/12 14:52:03 ERROR namenode.NNThroughputBenchmark: java.lang.NullPointerException
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$ReplicationStats.generateInputs(NNThroughputBenchmark.java:1312)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:280)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1509)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1534)
...

However, the root cause is different from HDFS-9387.
From ReplicationStats#generateInputs, nameNode is uninitialized before use, which causes the NPE.

      final FSNamesystem namesystem = nameNode.getNamesystem();

From NNThroughputBenchmark#run, nameNode is only initialized when -namenode option is not specified. The fix is to initialize it properly in the else block when -namenode option is specified OR we should block this if it is not supported.

 if (namenodeUri == null) {
        nameNode = NameNode.createNameNode(argv, config);
        NamenodeProtocols nnProtos = nameNode.getRpcServer();
        nameNodeProto = nnProtos;
        clientProto = nnProtos;
        dataNodeProto = nnProtos;
        refreshUserMappingsProto = nnProtos;
        bpid = nameNode.getNamesystem().getBlockPoolId();
      } else {
        FileSystem.setDefaultUri(getConf(), namenodeUri);
        DistributedFileSystem dfs = (DistributedFileSystem)
            FileSystem.get(getConf());
        final URI nnUri = new URI(namenodeUri);
        nameNodeProto = DFSTestUtil.getNamenodeProtocolProxy(config, nnUri,
            UserGroupInformation.getCurrentUser());
 

Issue Links

• relates to: HDFS-9387 Fix namenodeUri parameter parsing in NNThroughputBenchmark
• relates to: HDFS-7847 Modify NNThroughputBenchmark to be able to operate on a remote NameNode