[HADOOP-16345] Potential NPE when instantiating FairCallQueue metrics

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.4, 3.3.0, 3.2.1, 3.1.3
• Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 3.1.3
• Component/s: ipc
• Hadoop Flags: Reviewed
• Assignee: Erik Krogen
• Reporter: Erik Krogen
• Created: 03/Jun/19 21:02
• Updated: 07/Jun/19 21:33
• Resolved: 07/Jun/19 21:33

Description

ahussein initially reported the following NullPointerException issue on HADOOP-16266:

2019-05-31 14:32:10,417 ERROR [main] impl.MetricsSourceAdapter (MetricsSourceAdapter.java:getMetrics(202)) - Error getting metrics from source ipc.8020.FairCallQueue
java.lang.NullPointerException
at org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.getQueueSizes(FairCallQueue.java:382)
at org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.getMetrics(FairCallQueue.java:410)
at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:200)
at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:183)
at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:156)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:100)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:73)
at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:222)
at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:101)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:268)
at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:233)
at org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.<init>(FairCallQueue.java:361)
at org.apache.hadoop.ipc.FairCallQueue$MetricsProxy.getInstance(FairCallQueue.java:368)
at org.apache.hadoop.ipc.FairCallQueue.<init>(FairCallQueue.java:114)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.ipc.CallQueueManager.createCallQueueInstance(CallQueueManager.java:149)
at org.apache.hadoop.ipc.CallQueueManager.<init>(CallQueueManager.java:79)
at org.apache.hadoop.ipc.Server.<init>(Server.java:3059)
at org.apache.hadoop.ipc.RPC$Server.<init>(RPC.java:1039)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server.<init>(ProtobufRpcEngine.java:427)
at org.apache.hadoop.ipc.ProtobufRpcEngine.getServer(ProtobufRpcEngine.java:347)
at org.apache.hadoop.ipc.RPC$Builder.build(RPC.java:848)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.<init>(NameNodeRpcServer.java:467)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createRpcServer(NameNode.java:803)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:709)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:960)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:933)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1699)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1766)
2019-05-31 14:32:10,418 INFO [main] ipc.CallQueueManager (CallQueueManager.java:<init>(84)) - Using callQueue: class org.apache.hadoop.ipc.FairCallQueue, queueCapacity: 1000, scheduler: class org.apache.hadoop.ipc.DecayRpcScheduler, ipcBackoff: true.

It would appera that when HADOOP-15481 added support for FCQ stats to the metrics2 system, it opens up the potential for the MetricsProxy to be used before the delegate has been set, causing an NPE.

Issue Links

• is broken by: HADOOP-15481 Emit FairCallQueue stats as metrics