Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha1
    • Component/s: benchmarks
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Hit the following NPE when reviewing fix for HDFS-9387 with manual tests as NNThroughputBenchmark currently does not have JUnit tests.

      HW11217:centos6.4 xyao$ hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -op replication -namenode hdfs://HW11217.local:9000
      15/11/12 14:52:03 INFO namenode.NNThroughputBenchmark: Starting benchmark: replication
      15/11/12 14:52:03 ERROR namenode.NNThroughputBenchmark: java.lang.NullPointerException
      	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$ReplicationStats.generateInputs(NNThroughputBenchmark.java:1312)
      	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark$OperationStatsBase.benchmark(NNThroughputBenchmark.java:280)
      	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.run(NNThroughputBenchmark.java:1509)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
      	at org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark.main(NNThroughputBenchmark.java:1534)
      ...
      

      However, the root cause is different from HDFS-9387.
      From ReplicationStats#generateInputs, nameNode is uninitialized before use, which causes the NPE.

            final FSNamesystem namesystem = nameNode.getNamesystem();
      

      From NNThroughputBenchmark#run, nameNode is only initialized when -namenode option is not specified. The fix is to initialize it properly in the else block when -namenode option is specified OR we should block this if it is not supported.

       if (namenodeUri == null) {
              nameNode = NameNode.createNameNode(argv, config);
              NamenodeProtocols nnProtos = nameNode.getRpcServer();
              nameNodeProto = nnProtos;
              clientProto = nnProtos;
              dataNodeProto = nnProtos;
              refreshUserMappingsProto = nnProtos;
              bpid = nameNode.getNamesystem().getBlockPoolId();
            } else {
              FileSystem.setDefaultUri(getConf(), namenodeUri);
              DistributedFileSystem dfs = (DistributedFileSystem)
                  FileSystem.get(getConf());
              final URI nnUri = new URI(namenodeUri);
              nameNodeProto = DFSTestUtil.getNamenodeProtocolProxy(config, nnUri,
                  UserGroupInformation.getCurrentUser());
       
      

        Attachments

        1. HDFS-9241.000.patch
          3 kB
          Mingliang Liu
        2. HDFS-9421-branch-2.7.patch
          4 kB
          Konstantin Shvachko

          Issue Links

            Activity

              People

              • Assignee:
                liuml07 Mingliang Liu
                Reporter:
                xyao Xiaoyu Yao
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: