[HBASE-20531] RS may throw NPE when close meta regions in shutdown procedure.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-1
• Fix Version/s: 3.0.0-alpha-1
• Labels: quota
• Hadoop Flags: Reviewed
• Assignee: Zheng Hu
• Reporter: Zheng Hu
• Created: 04/May/18 10:34
• Updated: 22/Apr/19 16:13
• Resolved: 06/May/18 03:43

Description

See also : https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16463322&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16463322

The NPE stack is as following:

2018-05-03 21:05:58,075 ERROR [RS_CLOSE_REGION-regionserver/instance-2:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server instance-2.c.gcp-hbase.internal,42063,1525381545380: Unrecoverable exception while closing region tes
t,,1525381436038.66de217a470764f3b37d8faebfd8e8c8., still finishing close *****
java.io.IOException: java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1637)
        at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1466)
        at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
        at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
        at org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3709)
        at org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2718)
        at org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2649)
        at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:929)
        at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1615)
        at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1612)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        ... 3 more

In HRegionServer#run(), we have the following:

@Override
public void run() {
......
    // Stop the quota manager
    if (rsQuotaManager != null) {
      rsQuotaManager.stop();
    }
    if (rsSpaceQuotaManager != null) {
      rsSpaceQuotaManager.stop();
      rsSpaceQuotaManager = null;
    }
......
    // Closing the compactSplit thread before closing meta regions
    if (!this.killed && containsMetaTableRegions()) {
      if (!abortRequested || this.fsOk) {
        if (this.compactSplitThread != null) {
          this.compactSplitThread.join();
          this.compactSplitThread = null;
        }
        closeMetaTableRegions(abortRequested);
      }
    }
......
}

We stop the rsSpaceQuotaManager firstly, and then close the meta region, but when close meta region, we need to use rsSpaceQuotaManager to reportFileArchivalForQuotas() , just as the stack trace said ...

Issue Links

• is duplicated by: HBASE-20599 NPE in HRegionServer.reportFileArchivalForQuotas