Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21800

RegionServer aborted due to NPE from MetaTableMetrics coprocessor

    XMLWordPrintableJSON

    Details

      Description

      I was just playing around the code, trying to capture "Top k" table metrics from MetaMetrics, when I bumped into this issue. Though currently we are not capturing "Top K" table metrics, but we can encounter this issue because of the "Top k Clients" that is implemented using the LossyAlgo.

       

      RegionServer gets aborted due to a NPE from MetaTableMetrics coprocessor. The log looks somewhat like this:

      2019-01-28 23:31:10,311 ERROR [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020] coprocessor.CoprocessorHost: The coprocessor org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw java.lang.NullPointerException
      java.lang.NullPointerException
      	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.markMeterIfPresent(MetaTableMetrics.java:123)
      	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.tableMetricRegisterAndMark2(MetaTableMetrics.java:233)
      	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.preGetOp(MetaTableMetrics.java:82)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:840)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:837)
      	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:551)
      	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:625)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:837)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2608)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2547)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
      2019-01-28 23:31:10,314 ERROR [RpcServer.priority.FPBQ.Fifo.handler=19,queue=1,port=16020] regionserver.HRegionServer: ***** ABORTING region server 10.0.0.24,16020,1548747043814: The coprocessor org.apache.hadoop.hbase.coprocessor.MetaTableMetrics threw java.lang.NullPointerException *****
      java.lang.NullPointerException
      	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.markMeterIfPresent(MetaTableMetrics.java:123)
      	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.tableMetricRegisterAndMark2(MetaTableMetrics.java:233)
      	at org.apache.hadoop.hbase.coprocessor.MetaTableMetrics$ExampleRegionObserverMeta.preGetOp(MetaTableMetrics.java:82)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:840)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$19.call(RegionCoprocessorHost.java:837)
      	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost$ObserverOperationWithoutResult.callObserver(CoprocessorHost.java:551)
      	at org.apache.hadoop.hbase.coprocessor.CoprocessorHost.execOperation(CoprocessorHost.java:625)
      	at org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:837)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2608)
      	at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2547)
      	at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:41998)
      	at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
      	at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
      	at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304) 

        Attachments

        1. hbase-21800.branch-1.001.patch
          12 kB
          Sakthi
        2. hbase-21800.branch-1.002.patch
          12 kB
          Sakthi
        3. hbase-21800.branch-1.003.patch
          12 kB
          Sakthi
        4. hbase-21800.branch-1.004.patch
          11 kB
          Sakthi
        5. hbase-21800.master.001.patch
          13 kB
          Sakthi
        6. hbase-21800.master.002.patch
          13 kB
          Sakthi
        7. hbase-21800.master.003.patch
          13 kB
          Sakthi

          Issue Links

            Activity

              People

              • Assignee:
                sakthi Sakthi
                Reporter:
                sakthi Sakthi
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: