Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-20531

RS may throw NPE when close meta regions in shutdown procedure.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 3.0.0
    • Component/s: None
    • Labels:
    • Hadoop Flags:
      Reviewed

      Description

      See also : https://issues.apache.org/jira/browse/HBASE-20475?focusedCommentId=16463322&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16463322

      The NPE stack is as following:

      2018-05-03 21:05:58,075 ERROR [RS_CLOSE_REGION-regionserver/instance-2:0-1] helpers.MarkerIgnoringBase(159): ***** ABORTING region server instance-2.c.gcp-hbase.internal,42063,1525381545380: Unrecoverable exception while closing region tes
      t,,1525381436038.66de217a470764f3b37d8faebfd8e8c8., still finishing close *****
      java.io.IOException: java.lang.NullPointerException
              at org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1637)
              at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1466)
              at org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:104)
              at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NullPointerException
              at org.apache.hadoop.hbase.regionserver.HRegionServer.reportFileArchivalForQuotas(HRegionServer.java:3709)
              at org.apache.hadoop.hbase.regionserver.HStore.reportArchivedFilesForQuota(HStore.java:2718)
              at org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2649)
              at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:929)
              at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1615)
              at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1612)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              ... 3 more
      

      In HRegionServer#run(), we have the following:

      @Override
      public void run() {
      ......
          // Stop the quota manager
          if (rsQuotaManager != null) {
            rsQuotaManager.stop();
          }
          if (rsSpaceQuotaManager != null) {
            rsSpaceQuotaManager.stop();
            rsSpaceQuotaManager = null;
          }
      ......
          // Closing the compactSplit thread before closing meta regions
          if (!this.killed && containsMetaTableRegions()) {
            if (!abortRequested || this.fsOk) {
              if (this.compactSplitThread != null) {
                this.compactSplitThread.join();
                this.compactSplitThread = null;
              }
              closeMetaTableRegions(abortRequested);
            }
          }
      ......
      }
      

      We stop the rsSpaceQuotaManager firstly, and then close the meta region, but when close meta region, we need to use rsSpaceQuotaManager to reportFileArchivalForQuotas() , just as the stack trace said ...

        Attachments

        1. HBASE-20531.v1.patch
          2 kB
          Zheng Hu

          Issue Links

            Activity

              People

              • Assignee:
                openinx Zheng Hu
                Reporter:
                openinx Zheng Hu
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: