Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16345

Potential NPE when instantiating FairCallQueue metrics

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.4, 3.3.0, 3.2.1, 3.1.3
    • Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
    • Component/s: ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Ahmed Hussein initially reported the following NullPointerException issue on HADOOP-16266:

      2019-05-31 14:32:10,417 ERROR [main] impl.MetricsSourceAdapter (MetricsSourceAdapter.java:getMetrics(202)) - Error getting metrics from source ipc.8020.FairCallQueue
      java.lang.NullPointerException
      at org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.getQueueSizes(FairCallQueue.java:382)
      at org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.getMetrics(FairCallQueue.java:410)
      at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200)
      at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:183)
      at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:156)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
      at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:100)
      at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:73)
      at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:222)
      at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:101)
      at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
      at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:233)
      at org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.<init>(FairCallQueue.java:361)
      at org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.getInstance(FairCallQueue.java:368)
      at org.apache.hadoop.ipc.FairCallQueue.<init>(FairCallQueue.java:114)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
      at org.apache.hadoop.ipc.CallQueueManager.createCallQueueInstance(CallQueueManager.java:149)
      at org.apache.hadoop.ipc.CallQueueManager.<init>(CallQueueManager.java:79)
      at org.apache.hadoop.ipc.Server.<init>(Server.java:3059)
      at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:1039)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:427)
      at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:347)
      at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:848)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.<init>(NameNodeRpcServer.java:467)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:803)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:709)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:960)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:933)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1699)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1766)
      2019-05-31 14:32:10,418 INFO [main] ipc.CallQueueManager (CallQueueManager.java:<init>(84)) - Using callQueue: class org.apache.hadoop.ipc.FairCallQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DecayRpcScheduler, ipcBackoff: true.
      

      It would appera that when HADOOP-15481 added support for FCQ stats to the metrics2 system, it opens up the potential for the MetricsProxy to be used before the delegate has been set, causing an NPE.

        Attachments

        1. HADOOP-16345.000.patch
          2 kB
          Erik Krogen

          Issue Links

            Activity

              People

              • Assignee:
                xkrogen Erik Krogen
                Reporter:
                xkrogen Erik Krogen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: