Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-21349

Cluster is going down but CatalogJanitor and Normalizer try to run and fail noisely

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.1.0
    • 3.0.0-alpha-1, 2.2.0, 2.1.1, 2.0.3
    • None
    • None
    • Reviewed

    Description

      Shutting down can take a while. Meantime catalog janitor and or normalizer (etc?) try to run and when they can't, they fail noisely. Looks bad:

      2018-10-19 21:23:24,962 INFO org.apache.hadoop.hbase.master.ServerManager: Cluster shutdown set; vc1205.halxg.cloudera.com,22101,1539991730711 expired; onlineServers=51
      2018-10-19 21:25:54,502 WARN org.apache.hadoop.hbase.master.CatalogJanitor: Failed scan of catalog table
      java.io.IOException: connection is closed
              at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:267)
              at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:763)
              at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734)
              at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:684)
              at org.apache.hadoop.hbase.MetaTableAccessor.scanMetaForTableRegions(MetaTableAccessor.java:679)
              at org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:185)
              at org.apache.hadoop.hbase.master.CatalogJanitor.getMergedRegionsAndSplitParents(CatalogJanitor.java:137)
              at org.apache.hadoop.hbase.master.CatalogJanitor.scan(CatalogJanitor.java:243)
              at org.apache.hadoop.hbase.master.CatalogJanitor.chore(CatalogJanitor.java:116)
              at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      2018-10-19 21:25:54,507 ERROR org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore: Failed to normalize regions.
      java.io.IOException: connection is closed
              at org.apache.hadoop.hbase.MetaTableAccessor.getMetaHTable(MetaTableAccessor.java:267)
              at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:763)
              at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:734)
              at org.apache.hadoop.hbase.MetaTableAccessor.scanMeta(MetaTableAccessor.java:690)
              at org.apache.hadoop.hbase.MetaTableAccessor.fullScanTables(MetaTableAccessor.java:240)
              at org.apache.hadoop.hbase.master.TableStateManager.getTablesInStates(TableStateManager.java:189)
              at org.apache.hadoop.hbase.master.HMaster.normalizeRegions(HMaster.java:1718)
              at org.apache.hadoop.hbase.master.normalizer.RegionNormalizerChore.chore(RegionNormalizerChore.java:48)
              at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:186)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
              at org.apache.hadoop.hbase.JitterScheduledThreadPoolExecutorImpl$JitteredRunnableScheduledFuture.run(JitterScheduledThreadPoolExecutorImpl.java:111)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      
      

      Attachments

        1. HBASE-21349.master.002.patch
          3 kB
          Xu Cang
        2. HBASE-21349.master.002.patch
          3 kB
          Michael Stack
        3. HBASE-21349.master.002.patch
          3 kB
          Xu Cang
        4. HBASE-22349.master.001.patch
          3 kB
          Xu Cang

        Activity

          People

            xucang Xu Cang
            stack Michael Stack
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: