[HBASE-21800] RegionServer aborted due to NPE from MetaTableMetrics coprocessor

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha-1, 2.2.0, 1.4.10, 2.3.0, 2.1.4
• Component/s: Coprocessors, meta, metrics, Operability
• Labels: Meta
• Assignee: Sakthi
• Reporter: Sakthi
• Created: 29/Jan/19 07:42
• Updated: 07/Oct/19 18:26
• Resolved: 25/May/19 22:15

Description

I was just playing around the code, trying to capture "Top k" table metrics from MetaMetrics, when I bumped into this issue. Though currently we are not capturing "Top K" table metrics, but we can encounter this issue because of the "Top k Clients" that is implemented using the LossyAlgo.

 

RegionServer gets aborted due to a NPE from MetaTableMetrics coprocessor. The log looks somewhat like this:

2019-01-28 23:31:10,311 ERROR [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020] coprocessor.CoprocessorHost: The coprocessor org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw java.lang.NullPointerException
java.lang.NullPointerException
	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.markMeterIfPresent(MetaTableMetrics.java:123)
	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.tableMetricRegisterAndMark2(MetaTableMetrics.java:233)
	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.preGetOp(MetaTableMetrics.java:82)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:840)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:837)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:551)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:625)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:837)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2608)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2547)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
2019-01-28 23:31:10,314 ERROR [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020] regionserver.HRegionServer: ***** ABORTING region server 10.0.0.24,16020,1548747043814: The coprocessor org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw java.lang.NullPointerException *****
java.lang.NullPointerException
	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.markMeterIfPresent(MetaTableMetrics.java:123)
	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.tableMetricRegisterAndMark2(MetaTableMetrics.java:233)
	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.preGetOp(MetaTableMetrics.java:82)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:840)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:837)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:551)
	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:625)
	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:837)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2608)
	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2547)
	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 

Issue Links

• is related to: HBASE-19722 Meta query statistics metrics source